Linux Compatibility

Hi all.

I'm in the process of migrating existing script on UNIX server to the LINUX platform.

One of the script that have issues is this one:

cd /home/edwh_test/S13018/EDWH-DMT03/stgdata/RPT/

# GIANT #
rm -f INPUT_GIANT.csv
filename=INPUT_GIANT_*csv
GIANT_MONYYYY=$(echo $filename | awk '{print substr($0,length($0)-10,7)}')
# cp INPUT_GIANT_*.csv INPUT_GIANT.csv
tr '\r' '\n' < INPUT_GIANT_*.csv | grep -v '^$' > INPUT_GIANT.csv

After further investigation, this is due to the

filename=INPUT_GIANT_*csv

portion which mysteriously working when executed manually but not effective when run the whole script.

Error returned was:

debug_GST.sh: line 8: INPUT_GIANT_*csv: No such file or directory

So is there anyway to fix this since I need to put asterisk because I would not know what would be the actual file month.

Thank you.

Change:

filename=INPUT_GIANT_*csv

to:

filename=$(printf '%s\n' INPUT_GIANT_*csv)

to fix your immediate problem. However, this will not work if the filename matching pattern INPUT_GIANT_*csv doesn't match any files in the current directory or if it matches more than one file.

Change:

GIANT_MONYYYY=$(echo $filename | awk '{print substr($0,length($0)-10,7)}')

to:

GIANT_MONYYYY=${filename%???}
GIANT_MONYYYY=${GIANT_MONYYYY#???????}

to make your script run faster. Of course since you haven't shown us an actual filename, this is just a guess; but it seems to be a reasonable guess based on the assumption that that line in your current script works on a UNIX system. (Or, if GIANT_MONYYYY isn't referenced later in your script, just get rid of that line.)

Thanks Don.

Actually I've already fixed it by

for file in INPUT_GIANT_*csv; do filename=$file; done

Should be OK right? By right, there should be only one INPUT_GIANT_*csv at a time, it's just that the file month and year would be different.

By the way, does anyone knows why is that the

filename=INPUT_GIANT_*csv

works by manual typing in the terminal but failed with nohup sh?

Thank you.

I would be very surprised if the command:

filename=INPUT_GIANT_*csv

ever did what you think it did. If you type in that command manually at a terminal while sitting in a directory where one file matching that pattern exists and immediately follow that command with the command:

printf 'filename: %s\n' "$filename"

I think you'll find that it prints:

filename: INPUT_GIANT_*csv

while the command:

printf 'filename: %s\n' $filename

(with the double quotes removed), prints:

filename: INPUT_GIANT_XXXcsv

where XXX is the string matched by the asterisk in your filename matching pattern.

You haven't shown us what your script does with the variables file , filename , and GIANT_MONYYYY other than the one place that $filename is expanded while setting the variable GIANT_MONYYYY . Without seeing how those variables are used in the rest of your script, it is hard to guess at what might be wrong.

With what you have shown us, it is hard to guess at why your script would work on a UNIX system (other than a Solaris/SunOS UNIX system) if you were using a Korn shell or a bash shell on both your UNIX system and on your Linux system.

What UNIX operating system were you using and what Linux system are you using now?

What shell were you using on the UNIX operating system and what shell are you using on your Linux system now?

Please show us your complete script as it was when it worked on the UNIX system you were using and please show us the complete script as it now appears on your Linux system.

Note that the error message you are getting is saying that when you run the command:

tr '\r' '\n' < INPUT_GIANT_*.csv | grep -v '^$' > INPUT_GIANT.csv

the pattern INPUT_GIANT_*.csv is not matching any existing file. Note also that the pattern used here is not the same pattern that was assigned to the variable filename . (One ends in .csv while the other one ends in csv (without the <period> character.) There is nothing in the command that is failing that refers to any variable. So, why do you think an expansion of the variable filename is your problem? (Or, did you not show us the first 8 lines of the script named debug_GST.sh ?)

What output do you get from the command:

ls -l /home/edwh_test/S13018/EDWH-DMT03/stgdata/RPT/INPUT_GIANT*

at a time when debug_GST.sh fails with the diagnostic you showed us in post #1 in this thread?

Thanks again Don.

ls -l /home/edwh_test/S13018/EDWH-DMT03/stgdata/RPT/INPUT_GIANT*

yields

-rw-r--r-- 1 edwh_test users 23082 Aug  5 16:05 /home/edwh_test/S13018/EDWH-DMT03/stgdata/RPT/INPUT_GIANT_JUN2016.csv

So that's the file that expected to match.

UNIX:

HP-UX system1 B.11.31 U ia64 0189138652 unlimited-user license

LINUX:

Linux ex01db03.tm.com.my 2.6.39-400.264.1.el6uek.x86_64 #1 SMP Wed Aug 26 16:42:25 PDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Manual typing to the terminal works as expected:

So

filename=INPUT_GIANT_JUN2016.csv
GIANT_MONYYYY=JUN2016

I know it's a problem with that

filename=INPUT_GIANT_*csv

because it is inside the log temp.rst saying

debug_GST.sh: line 8: INPUT_GIANT_*csv: No such file or directory

.

After fixing the script, the log is empty

nohup sh debug_GST.sh > temp.rst &

Thanks a lot.

Please copy and paste text rather than posting screenshots. (Especially screenshots with light purple text on a slightly darker purple background!)

You did not run the command:

printf 'filename: %s\n' "$filename"

(with the double quotes around $filename ) as I requested. If you had run that command as I requested, you would have seen that the variable filename contains the filename matching pattern you assigned to that variable (i.e., INPUT_GIANT_*csv ); not the filename matching that pattern INPUT_GIANT_JUN2016.csv .

You also did not show us your complete script. If you had shown us the original script that produced the diagnostic you showed us in post #1 in this thread:

debug_GST.sh: line 8: INPUT_GIANT_*csv: No such file or directory

and line 8 in that script did expand $filename , I would bet that it did so within double quotes like the printf command I asked you to run above using "$filename" .

And, although I did have an off-by-one error since I didn't know what your filenames looked like. Your script will run faster if you change the lines:

GIANT_MONYYYY=$(echo $filename | awk '{print substr($0,length($0)-10,7)}')
# cp INPUT_GIANT_*.csv INPUT_GIANT.csv
tr '\r' '\n' < INPUT_GIANT_*.csv | grep -v '^$' > INPUT_GIANT.csv

in the script you showed us in post #1 to:

GIANT_MONYYYY=${filename%????}
GIANT_MONYYYY=${GIANT_MONYYYY#???????}
# cp INPUT_GIANT_*.csv INPUT_GIANT.csv
tr -d '\r' < $filename > INPUT_GIANT.csv

since it eliminates a subshell invocation and the invocation of two unneeded external utilities ( awk and grep ).

What would happen if someone created a file called INPUT_GIANT_123 rm -rf * .csv ?

I fear a simple use of filename=INPUT_GIANT_*.csv might evaluate this as:-Set the variable filename to INPUT_GIANT_123 and then run rm -rf * .csv which might not be what you actually want to do. This might not be malicious and could be an error typing on the command line.

You might well be safer to sanitise the input before using it openly.

Robin

Hi Robin,
In POSIX-conforming shells (or any shell based on Bourne shell syntax that I know of), field splitting and pathname expansions do not occur when assigning a string to a variable. So, in your example, the assignment:

filename=INPUT_GIANT_*.csv

shouldn't cause any problem. But if the file's name had been:

INPUT_GIANT_123;rm -rf * .csv

then in some contexts, expanding $filename unquoted could result in attempting to remove all files in the file hierarchy rooted in the current directory (except hidden files in the current directory) and then attempting to remove the hidden file named .csv from the current directory and, if it was a directory, all files in the file hierarchy rooted in that directory.

Thanks Don,

Maybe I'm being paranoid, but I feel sure there are things going to bite me

I just know that I've been caught out before when trusting filenames on disk, standard messages in log files etc. so I'm just always careful when reading input to try not to assume that it's all good. If this discussion makes people consider the bad things that can happen, then I'm all for it. The worst example I found was the evil eval being used to execute files, something like:

#
#
# Setup and code for creating a set of 'build-scripts' that are then
# run in parallel for no good reason that I could ever work out
#
# then it went:.....

for file in *.build-script
do
   eval $file > $file-out &
done
wait
exit 0

How may ways was this so wrong? Well, it's all gone now but it highlights what a mess we can make if we're not careful

Having inherited it, it ran in the overnight batch for a few months silently doing the job, but it went VERY badly wrong and it was effectively DR time as we lost /etc making login a little difficult ....... :rolleyes:

Hi Robin,
You're not being paranoid. Assigning a pathname matching pattern to a variable won't expand the pattern during the assignment (so that isn't inherently dangerous). But, expanding that variable unquoted is a completely different matter. And expanding it with eval (quoted or unquoted) gives you MUCH more to be concerned about.