Difficulty with CAT redirection in script

I have not been able to append the contents of many files into one file. I have executed the CAT command shown below separately substituting an actual path and file name for the array variable to verify that I have the syntax correct. The bottom line - nothing is happening with CAT. I am running the script with sudo. Please excuse all the debug statements - just verifying that the variables contain valid values. I am running this on Ubuntu 14.04.

#!/bin/bash
echo ""
echo "==============================================================================="
echo "This script copies all of the LocalConfig.pri files contents into another file."
echo "==============================================================================="
echo ""
cd /
echo "Changed directory to ..."
pwd

# Assign 'locate' results to a variable.
output=$(locate -b "LocalConfig.pri")

# Parse the $output variable into an array. Each line is a full path.
while read -r line; do outputArray+=("$line"); done <<<"$output"

# Loop through the array.
for idx in "${!outputArray[@]}"; do
    # Print array element index and contents.
    printf ' Output number %d is %s' "$idx" "${outputArray[idx]}"
    printf '\n'
    # Cat each file into one 'master' file.
    echo -e "\tconcatenating\n\t\t ${outputArray[idx]} \n\tinto \n\t\t./LocalConfigMaster.pri ..."
    cat "${outputArray[idx]}" >> ./LocalConfigMaster.pri
done

Since you have all of this debugging information, why don't you share it with us so we can see what is going on? (Did you consider using set -xv instead of adding all of those debugging statements?)

Are you seeing any errors like ./LocalConfigMaster.pri: Permission denied ?

You could also avoid using arrays and use a for loop to set your file like this:

idx=0
locate -b "LocalConfig.pri" | while read file
do
    # Print element number and contents.
    printf 'Output number %d is %s\n' $((++idx)) "$file"
    # Cat each file into one 'master' file.
    echo -e "\tconcatenating\n\t\t $file \n\tinto \n\t\t./LocalConfigMaster.pri ..."
    cat "$file" >> ./LocalConfigMaster.pri
done

Don't you mean to use echo instead of cat? Or is the array element a filename? with a complete path....

Another point: You may want to declare outputArray as an array:

outputArray=()
# or
declare -a outputArray

I assume this is bash 3.2 or higher. Arrays were somewhat experimental in v3.1 bash.
What output do you get/see in stdout? Please post a sample.

The debug output is nothing but full path file names followed by my target file name which is

. There are no errors and there is no output generated by the cat command. (from the command line) If I substitute anyone of these full path files names for the array variable, the cat command works just fine. These full path file names look something like this

. Again, works great from the command line, does nothing inside this script of mine. I am unfamiliar with

but not for long. :slight_smile: I am fairly new to the bash world. Thanks for your help and patience Don, it is much appreciated.

---------- Post updated at 09:51 PM ---------- Previous update was at 09:45 PM ----------

Thanks Jim. The samples I read suggested cat. The array is working, I can verify that - I am getting the expected results, it's just this cat command that is doing nothing. I provided a small one-line sample output in my reply to Don. Thank you for the suggestions though. I'm a little green when it comes to bash. And I do not know what version it is but I'll find out.

---------- Post updated at 09:53 PM ---------- Previous update was at 09:51 PM ----------

---------- Post updated at 09:55 PM ---------- Previous update was at 09:53 PM ----------

At first I did see the permission denied error but that went away after invoking the script with sudo. And I appreciate the tips on tightening up this code. Thanks.

In general it is usually a good idea to just post the information instead of telling us about it. The reason is: being a newcomer (as per your own admission) you might lack the knowledge of what is vital information and what is not. Show the scripts code (you did that), then run it and copy its output from start to end (ideally along with the return code it produced) here, enclosed in CODEtags, like this:

# ./myscript.sh
bla
foo
error: unable to make the flurbishes grommicking

# echo $?
2

Here is one more such suggestion: are you sure you want to add always to the file in question? Wouldn't you rather want to start an empty file for every run of the script and then fill the contents of all the other files to this one?

If so, supposing outputarray is an array variable containing path names AND there are no special characters (blanks, tabs, ...) in the file names you can do:

cat ${outputarray[*]} > /some/file

${outputarray[*]} will expand to a list of all the array elements and cat takes an (open-ended) list of filenames to process as arguments. There are some restrictions on this and if the array has several hundreds of entries this method might break eventually (the exact point depending on your systems configuration), but for a handful (anything less than hundred for sure) of filenames this works well.

I hope this helps.

bakunin

[code]

1 Like

Note that the command:

cat ${outputarray[*]} > /some/file

may fail if any of the filenames in the array contain whitespace characters. To protect against this case, the following is usually safer:

cat "${outputarray[@]}" > /some/file
1 Like

Not sure if I read your script correctly, but if its purpose is to concatenate all LocalConfig.pri files into one master file file and generate a contents file in parallel, would this simplified command do:

cat $(locate -b LocalConfig.pri | tee contentsfile) > ./LocalConfigMaster.pri

The index number could, if need be, calculated from the contentsfile's lines...

Great tip and I am already doing that. The locate command retrieves 108 file names, none of which contain any whitespace in their name. The problem is definitely perplexing.

---------- Post updated at 09:44 AM ---------- Previous update was at 09:05 AM ----------

You almost have it. No contents file. My contents file is the LocalConfigMaster.pri file (it will contain the contents of all the LocalConfig.pri files).

Well, read: Table of Contents; file containing all the file names/-paths found which will be lost when just cat ting the files.

[quote=bakunin;302990251]
In general it is usually a good idea to just post the information instead of telling us about it. The reason is: being a newcomer (as per your own admission) you might lack the knowledge of what is vital information and what is not. I hear you but this debug output is very simplistic. I am new to bash but not to software development/script writing. But I will generate and post some output below. The output is filenames - that's it (no errors, no mysterious codes, just names of files, 108 of them) - and I'm throwing them up against cat and cat is not doing anything with them. Show the scripts code (you did that), then run it and copy its output from start to end (ideally along with the return code it produced) here, enclosed in CODEtags, like this:

# ./myscript.sh
bla
foo
error: unable to make the flurbishes grommicking

# echo $?
2

Here is one more such suggestion: are you sure you want to add always to the file in question? Wouldn't you rather want to start an empty file for every run of the script and then fill the contents of all the other files to this one? Good suggestion but at this moment I am not concerned about it. Once I understand how I am not using cat correctly then I may come back to tweak the process.

If so, supposing outputarray is an array variable containing path names AND there are no special characters (blanks, tabs, ...) in the file names you can do:

cat ${outputarray[*]} > /some/file

${outputarray[*]} will expand to a list of all the array elements and cat takes an (open-ended) list of filenames to process as arguments. There are some restrictions on this and if the array has several hundreds of entries this method might break eventually (the exact point depending on your systems configuration), but for a handful (anything less than hundred for sure) of filenames this works well.

I hope this helps. I definitely appreciate your input and help!

bakunin

Here is the code ...

#!/bin/bash
echo ""
echo "==============================================================================="
echo "This script copies all of the LocalConfig.pri files contents into another file."
echo "==============================================================================="
echo ""
cd /
echo "Changed directory to ..."
pwd

# Assign 'locate' results to a variable.
output=$(locate -b "LocalConfig.pri")

# Parse the $output variable into an array. Each line is a full path.
while read -r line; do outputArray+=("$line"); done <<<"$output"

# Loop through the array.
for idx in "${!outputArray[@]}"; do
    # Print array element index and contents.
    printf ' Output number %d is %s' "$idx" "${outputArray[idx]}"
    printf '\n'
    # Cat each file into one 'master' file.
    echo -e "\tconcatenating\n\t\t ${outputArray[idx]} \n\tinto \n\t\t./LocalConfigMaster.pri ..."
    cat "${outputArray[idx]}" >> ./LocalConfigMaster.pri
done

I highlighted the code above that is not doing anything as best as I can tell.

Here is the output (I abbreviated it because there are 108 files returned from the locate command)...

my.name@host:~/bash_scripts$ sudo ./CopyLocalConfig.sh
[sudo] password for my.name: 

===============================================================================
This script copies all of the LocalConfig.pri files contents into another file.
===============================================================================

Changed directory to ...
/
 Output number 0 is /fouo/project-release-develop/project/LocalConfig.pri
	concatenating
		 /fouo/project-release-develop/project/LocalConfig.pri 
	into 
		./LocalConfigMaster.pri ...

The output highlighted above in bold, repeats for every occurrence of LocalConfig.pri (107 more times).

The script terminates without any errors. I execute ls -la and there is no LocalConfigMaster.pri file.

Just to be sure we cover the obvious case, what output do you get from executing the command:

ls -l /LocalConfigMaster.pri

instead of the command:

ls -la

?

Just because your script changes directory to the system's root directory does not mean that the directory of your interactive shell has moved to that location after you run your script...

1 Like
-rw-r--r--    1 root root 16226292 Jan 24 10:01 LocalConfigMaster.pri

Oh my God, I am such an idiot! LOL! cat is working fine. Talk about a stupid mistake. LocalConfigMaster.pri is getting written to the directory that I had the script cd to. I kept looking for it in the directory that I executed the script from. I can't believe how many hours I blew on this. Thank you for pointing out what should've been obvious to me Don!

I'm glad we worked it out. Sometimes the obvious possibility just seems to unlikely to ask about. :cool:

The 2nd set of eyes is a valuable programming tool. :slight_smile: Thanks again for you help Don. Hopefully I won't pull too many more bonehead moves like that again. :slight_smile:

Don't worry, it happens to the best of us.

As a general rule: do not change directories inside a script at all (for exactly this reason, to avoid this confusion). Whenever you work on files make sure you always use absolute pathes like this:

# cat wrong.sh

cd /some/where
command > ./output.file

# cat correct.sh

outdir="/some/where"
command > "${outdir}/output.file"

The same goes for all other filenames. This way your script will work regardless of where you started it. If you work on several files using a variable for the directory part ensures they all land in the same place. The absolute worst you can do, though, is to use relative pathes:

# cat worst-of-worst.sh

command1 > outfile1
cd ..
command2 > outfile2

If you ever find that in anyones code: have them promise never to write any shell script again. This is a surefire recipe for disaster because the script will (maybe) work if you call it from one directory and fail if you call it from another.

In general the script you call inherits the environment from its calling process - the command shell you used to call it. "Envrionent" means not only the values for variables (all that have been "export"ed before) like PATH, TZ (timezone), LANG, etc.. but also the current directory and similar things. It is good practice to make your script independent from this environment by setting it to a certain state except for the few variables where you explicitly want this effect to take place. This includes (but is not limited to) making it independent of the current path it was called from.

I hope this helps.

bakunin

1 Like

Absolutely! Great tips! Thank you.