[bash] why my variable is not updated?

Does anyone know why the below script is not working? Why is not the variable tot_files updated?

location=$1
cd "$location"

tot_files=0

(
    echo ""
    # recursively gets the total number of files
    tot_files=$(for t in files ; do echo `find . -type ${t:0:1} | wc -l` $t | cut -f1 -d" "; done 2> /dev/null) & 
) | wait ${!} | yad --progress --pulsate --progress-text="" --center --title="title" --text="Please wait... Reading files..." --no-buttons --no-escape --width=300 --height=200

echo "tot_files: " $tot_files

I would like to show a yad (zenity alternative) window that says to the user to wait until all the files in $1 have been counted.
That's why I used wait ${!} but I am not sure I have used in the right way.

Everything between the parentheses marked in red above is performed in a subshell. Nothing set inside that subshell will be visible in the shell that invoked that subshell (including changes to variables tot_files and ! . And, since there is no wait inside the subshell, the subshell will return to the main shell before tot_files is updated in the subshell. And, the main shell has no idea that there is anything left running in the background. And, when it completes, the updated value assigned to tot_files will silently be ignored.

Furthermore, wait should never be in the middle of a pipeline; it doesn't read anything from its standard input; and even if it did, nothing from the first stage of your pipeline writes anything to its standard output.

I'm afraid I have no experience with zenity nor yad so I can only tell you what is wrong with your script; not how to fix it to make yad work.

2 Likes

Because you updated it in a subshell. In fact it is updated - but only in the subshell and with leaving the subshell the value is lost too.

These two are identical and they:

( command )

(
    command
)

tell the shell to open a new shell and execute everything within the braces in this new shell and then close it.

The next problem is:

tot_files=$(<...>) &

This is not working at several levels.

First: what do you want to assign to the variable tot_files ? Obviously the stdout of whatever is inside the subshell ( "$(...) ). It is like this:

var=$( echo "blabla" )

The shell opens a subshell and executes echo "blabla" in this subshell. So you get some output at <stdout>, in this case a string. This output is then assigned to the variable.

The next thing is you put that assignment into background:

tot_files=<something> &

But an assignment is not a separate process, it is done within the shell internally. x=5 does not generate a process you could put into background (which is in fact just another way of saying it runs asynchronously to the main process). I am not sure about that (perhaps Don Cragun knows more about these fringe situations) but i suppose that the "&" is simply ignored in this case.

The next problem is:

tot_files=<...> & | wait ${!} | <...>

A pipeline takes the <stdout> (the "normal" output) of a process and feeds it into the input (<stdin>) of another process. The output of

var=value

is <nothing>. Try it out on the commandline. When you pour that nothing into another command, what will happen, hmm? Furthermore, the wait command does not even process any input, and neither does it have any output. So, even if there would be any output from the assignment operation it would be lost by now - for the same reasons as above. So, whatever you pipe the output of wait into, there will be nothing to process because there will be no output.

Now, after this lengthy explanation of what will not work, let us finish on some positive note:

In principle you do that the following way: first display the message for the user that he should wait, then you start the lengthy process. Finally you update (or maybe redraw, depending on which possibilities are offered by your environment) the screen with a finishing message and the result of the operation:

echo "Starting calculations, hold on...."
result=<...lengthy calculation here...>
echo "Finished. Result is: $result"

As an addendum i suggest to go over the counting command itself too (to me that looks not ideal, whatever it is supposed to do), but since i do not know exactly what you want to achieve i will not suggest an alternative yet. You are welcome to post your goals and we can do that too.

I hope this helps.

bakunin

1 Like

Actually, this is pretty straight forward. But, an "&" is never simply ignored in the shell command language. The "&" causes the previous list (where "list" in this case is a simple command that is an assignment statement) to be run asynchronously in a subshell. So, we not only have a subshell due to the parentheses, we have a sub-subshell to run the assignment statement in the background. It is not clear to me that the standards mandate whether the command substitution that is part of this command is run in the background or whether the shell can run the command substitution in the foreground and then run the assignment of the results of that command substitution to the specified variable in the background.

In either case result of the variable assignment is visible only in the sub-subshell; it is not visible in the environment of the shell that started the subshell and it is not visible in the subshell that started the asynchronous list.

I hope this helps.

  • Don
2 Likes

First off, thank you for clearing that up. It actually never occurred to me to put an assignment (or anything else not being an external executable, for that matter) in background so i couldn't draw on experience. One thing is still not completely clear to me and i hope you can clarify. You say:

I take it, the mechanism of

$( list; ... )

is to open a subshell, execute list there and then close the subshell. Now, any command (and, as i have just learned, even an assignment) put into background is executed in its own subprocess (and as the assignment is a shell command, a subshell, yes?). Therefore:

var=$( list; ... ) &

should, as i understood it, open a subshell (running asynchronously), execute

var=<something>

there and open another sub-subshell in there to calculate this <something> because <something> is actually $( ... ) . So the assignment happens at the first subshell and the list inside $( list; ... ) happens at the sub-subshell. Have i gotten that right or am i a hopeless case?

Anyway, i have at least tested a bit and i found:

# x=5
# echo $x
5
# y=5 &
[1]     6887
[1] +  Done                    y=5 &
# echo $y
# 

So, putting an assignment into background does not only make no sense it even (for practical purposes) nullifies its effect because the variable gets assigned in a subshell. At least this much i take away from your explanation and this is true regardless of in which way the subshells and sub-subshells are arranged. Thank you again.

bakunin

1 Like

Thank you so much for the info provided... After your explanation I was able to make it work :b:

tot_files_found="$(echo $RANDOM)_find.$CURRENT_USER.$(echo $RANDOM)$(echo $RANDOM)" 
while true; do
    if [ -f "$tot_files_found" ]; then      # File exists
        if [ -s "$tot_files_found" ]; then  # File exists and not empty; (find . has written the file; operation ended)
            break
        fi     
    else
        # recursively gets the total number of files     
        find . -type f | wc -l > $tot_files_found &    
    fi
done | yad --image="$MD5SUM_ICONS/binary.png" --progress --pulsate --progress-text="" --center --title="Getting things ready..." --text="Reading files..." --auto-kill --auto-close --no-buttons --no-escape --fixed 
 

Of course if anyone has a better way to do this, I am all hears.
Thanks again.

Still not clear why you (a) pipe (b) nothing into yad . Does it need meaningful data on its stdin?
a) piping is (not illegal but) not an apt way to put execution of commands into a certain order. If not reading from stdin, the second command may be finished before the first does anything meaningful.

b) No info will be supplied by your script; anything found will be output to $tot_files_found .

Hello,
What I am trying to achieve is the following:
As long as find . process is up and running, I want to show a message to the user and that is why I use

| yad --image="$MD5SUM_ICONS/binary.png" --progress --pulsate --progress-text="" --center --title="Getting things ready..." --text="Reading files..." --auto-kill --auto-close --no-buttons --no-escape --fixed

This is a yad's window running waiting for find . process to terminate yad.png - Google Drive

I use $tot_files_found to store the result coming from find . and to break from the while loop

That's all. I added wait ${!} to prevent multiple instances of find to be executed.

I am sure there are much better ways to do the same, but I am new in bash :frowning:

Right now, everything seems to work but I would like to find a better way to accomplish what I want to do.

tot_files_found="$(echo $RANDOM)_find.$CURRENT_USER.$(echo $RANDOM)$(echo $RANDOM)" 
while true; do
    if [ -f "$tot_files_found" ]; then      # File exists
        if [ -s "$tot_files_found" ]; then  # File exists and not empty; (find . has written the file; operation ended)
            break
        fi     
    else
        # recursively gets the total number of files     
        find . -type f | wc -l > $tot_files_found & 
        wait ${!}
    fi
done | yad --image="$MD5SUM_ICONS/binary.png" --progress --pulsate --progress-text="" --center --title="Getting things ready..." --text="Reading files..." --auto-kill --auto-close --no-buttons --no-escape --fixed

The entire purpose of running a job asynchronously is to allow you to run other jobs at the same time. If you don't want to run any other jobs at the same time while your find command is running; don't run it asynchronously.

The two commands:

        find . -type f | wc -l > $tot_files_found & 
        wait ${!}

produce exactly the same output that the single command:

        find . -type f | wc -l > $tot_files_found

does, but takes fewer keystrokes and makes less busywork for the shell. The first two commands force the shell to create a subshell to run find asynchronously; the last command is run in the current shell execution environment without the need to create a subshell environment. The first two commands slow down your system for you and other users on your system.

Note also that every time this script runs, it creates a new temporary file and (from what you have shown us) never uses anything written into it and never removes it. And the name you are using for that temporary file (with three random numbers included in the name and with two of those random numbers adjacent to each other with no separator) increases the chance that two different invocations of your script will accidentally come up with the same name. If that happens, find will never be run for this invocation of your script.

And, there is no need to create three subshells to include those three random numbers in the name. The command:

tot_files_found=${RANDOM}_find.$CURRENT_USER.$RANDOM$RANDOM 

produces exactly the same temporary filename as your current code does but avoids creating three unneeded subshells to set the name.

A more common way to do this (since there can never be two shells running with the same process ID) is to use:

tot_files_found=find.$CURRENT_USER.$$

to lessen the chance of a collision with your various multiple adjacent uses of $RANDOM . Of course, we might also comment that as far as we can tell from what you have shown us, $CURRENT_USER will expand to an empty string because it has never been set to anything else. And, since we have no idea how, or if, CURRENT_USER has been set, any expansion of tot_files_found when field splitting might occur should be included in a double-quoted string.

From everything that you have said about what you are trying to do, I do not understand why you have a loop at all? The following would seem to be a better fit for your stated goals (which I still do not understand):

tot_files_found=find.$CURRENT_USER.$$ 

# recursively gets the total number of files     
find . -type f | wc -l > "$tot_files_found" |
    yad --image="$MD5SUM_ICONS/binary.png" --progress --pulsate \
        --progress-text="" --center --title="Getting things ready..." \
        --text="Reading files..." --auto-kill --auto-close --no-buttons \
        --no-escape --fixed

But, as has been said before, having yad as the second process in a pipeline when the first process in that pipeline will NEVER write anything into that pipe is not a "normal" way to do things. Whatever output yad produces has nothing to do with whether or not find is making any progress; all it can tell us is when find has terminated. And that could be done more simply with:

tot_files_found=find.$CURRENT_USER.$$ 

# recursively gets the total number of files     
echo 'find is running'
find . -type f | wc -l > "$tot_files_found"
echo 'find is done'

We should probably also include a trap to remove the temporary file created by this script, but I haven't yet figured out if there might be some reason why you would want to actually know how many files were found instead of just wasting time figuring out how many there are and then exiting without ever looking at the results.

1 Like

To show the yad picture while find is running, did you consider

yad ... &
find ...
kill %1 (yad's jobspec, hopefully)
1 Like

To expand on what my colleagues RudiC and Don Cragun has already said:

I might not be (see above) the biggest expert on all things subshell but i know a thing or two about programming. Make no mistake, writing a shell script of 5 lines is essentially a programming task and it is never too early to start on healthy habits.

Programming is mostly about organizing your thoughts and the best way to organize them is to compartmentalize big problems into several smaller problems again and again until there are no problems any more. It is like moving a big heap: you take a shovel and dig one small shovel full after the other until there is no heap any more. If you try to move the whole pile you are likely to fail but with a single shovel full it is easy to succeed. One shovel - success, and another one - success again and before you notice you have the pile moved.

So, let us start with - as exactly as possible - clarify your goals: you want to get a count of (a certain type of) files within a certain subtree. Yes? (I don't mind if you respond "NO!" here, because this is what this stage is about - clarifying your goals. If i haven't understood them correctly just say so.)

You also want - because the operation might take some time - display a window telling the user to wait for completion which you want to update to tell when the counting is finished (probably, but that is my assumption, along with presenting the result). Yes?

So, first for the counting - note that we set aside everything else here! Again, shovel for shovel and we concern ourselves here with only this shovel, not the pile and not another shovel. We can use find for this but we can do that in a single instance because find can traverse the filesystem (or select parts of it) on its own. To start several instances of find and then compile their results is simply not necessary. In addition to what RudiC and Don Cragun already said you find a more thorough introduction to find here. See how far that gets you and ask if you still have questions.

There is another question to this: how is the result of this operation propagated to whereever you will need it. I will postpone that for the moment because the rest of the operation is not clear yet.

Second is the business with the user notification. I told you above that the basic logic of this is to:

  • present a user the initial window with a "please wait..."-message
  • run the find -operation
  • replace the "please wait...."-message with something else (i.e. a "done" or "result is:..."-message)

This is basically the same as what RudiC (and myself) earlier have already suggested. For this you do not need any pipes, background processes or anything else of that sort as Don Cragun has already explained.

So, if you want help for the further commencement of this, you are welcome to more clearly describe your goals and we can take it from there.

I hope this helps.

bakunin

2 Likes

In the end, thanks to you guys, I did the following:

tot_files_found=0
yad --image="$MD5SUM_ICONS/binary.png" --progress --pulsate --progress-text="" --center --title="Getting things ready..." --text="Reading files..." --auto-kill --auto-close --no-buttons --no-escape --width=300 --height=$HEIGHT &
tot_files_found=$(find . -type f -perm -a=r | wc -l)
kill %1

and since I am not using a sub-shell any-longer, I am able to store the value returned by find into a regular variable instead of a file :slight_smile:
On top of that, I changed all the $(echo $RANDOM) with .$CURRENT_USER.$$ in my other scripts too.

One last thing though... Is possible to kill/terminate the find process as soon as I close the yad window?

Bye and thank you so much for all your help and detailed explanations.

Here are two tips: first, if you operate with variables inside larger expressions like .$CURRENT_USER.$$ it is a good idea to use the long form of notation: .${CURRENT_USER} . The difference is that the curly braces make sure where the variables name ends and the rest of the string begins so that this rest of the string is not mistaken as a part of the variables name. For example: you want to add a slash to a variable so $var/ is fine, but add an underscore instead and $var_ would be interpreted as a variable named "var_". ${var}_ will always work, though.

Another tip is: clean up behind you. If you create a temporary file make sure that it is removed when the program creating it exits. You can do that with "traps", which is a theme for another thread but you might want to read up the shells documentation for it.

Yes - but that would be really advanced stuff. You would need to go back to sending the find/icode] into background, then the yad -window in another background and check in the main program if either of these subprocesses are finished, then kill the other. The logic is tricky and i suggest to put that on the backburner until you have some experience with shell programming. Get some shell-programming done and you will be able to do it in short time.

I hope this helps.

bakunin