I need help on following linux bash script. When I linux commands for loop or while loop on individual file it runs great. but now I want the script to run on N number of files so it gives me ambiguous redirect error on line 12 and syntax error on line 22 : (pls help );
#!/bin/bash
# MCE Repricing ASPRODUCTSUMMARY
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS;
for files in $(ls *.MCESOLUTION); do echo $files $(mv -v $files 'UNPROCESSED/ASPRODUCTSUMMARY/'${files%}.CSV) ; done ;
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY;
####ls -1 /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY > ListofMCECSVFilesASPRODUCTSUMMARY.txt;
FOLDERDIR="/u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY";
cd $FOLDERDIR;
for i in "ls -l | grep ^- | awk '{print $9}'"
do
grep -i "^..............ASPRODUCTSUMMARY"< "$i" > "$i"; done;
for file in '$(ls -l $FOLDERDIR)'; do
caseid=$(echo $file | cut -c1-10);
studytype=$(echo $file | cut -c12-15);
prefix="$caseid,$studytype,";
while read -r line
do
echo "${prefix}$line" done; < "$file" > temp.CSV;
mv -v temp.CSV ${file};
done;
exit 0;
It would be helpful to post the error messages, maybe embedded in the script's output when run with options -v and/or -x set. For now, I see the following noteworthy:
for i in "ls -l | grep ^- | awk '{print $9}'"
won't fly. For command substitution use backticks or the $(...) construct instead of double quotes. BTW, there may be better ways than this to eliminate special files.
The
will destroy every file in your directory that does not have the grep pattern in it and reduce those who have to the lines containing the pattern. If filenames contain special chars, e.g. space, the ambiguous redirect error may result (not sure though).
The second error you mention will probably be syntax error: unexpected end of file
and is the result of a missing ; in line 19 just in front of the done
After converting from "..." to $(...) or `...`, the unquoted ^- will also be a problem in many shells. At least in the Bourne shell and Korn shell ^ is a synonym for | .
Since the all redirections occur in the shell before grep is called, that command will always either result in an empty file or a redirection error.
First of all thank you so much for helping me RudiC and Don Cragon,
I changed the " " to $(), and quoted '^-':
Before :
for i in "ls -l | grep ^- | awk '{print $9}'"
Now : for i in
"ls -l | grep '^-' | awk '{print $9}'"
and put the ; before done on line 19th, I am a beginner for linux so don't know much about it but this script will sol the purpose I need to do with N number of files.
I exactly want to delete unwanted lines not having word "ASPRODUCTSUMARY" from a specific position 15th from any CSV file coming in as a result of
for i in "ls -l | grep '^-' | awk '{print $9}'"
Now I am getting error on line 12 after the file name as correctly being passed. The output of the script is as follows :
[madhu@devinfaav POPS]$ bash MCEASPRODUCTSUMMARY.sh
: command not found.sh: line 3:
: command not found.sh: line 4:
1400009823_RTBM_20120614_1715_0_0_0_002.MCESOLUTION `1400009823_RTBM_20120614_1715_0_0_0_002.MCESOLUTION' -> `UNPROCESSED/ASPRODUCTSUMMARY/1400009823_RTBM_20120614_1715_0_0_0_002.MCESOLUTION.CSV'
1400009823_RTBM_20120614_1715_0_0_0_00.MCESOLUTION `1400009823_RTBM_20120614_1715_0_0_0_00.MCESOLUTION' -> `UNPROCESSED/ASPRODUCTSUMMARY/1400009823_RTBM_20120614_1715_0_0_0_00.MCESOLUTION.CSV'
: command not found.sh: line 5:
: command not found.sh: line 6:
: command not found.sh: line 8:
: command not found.sh: line 9:
: command not found.sh: line 11:
: command not found.sh: line 11:
: command not found.sh: line 11:
: command not found.sh: line 11:
: No such file or directorye 14: 1400009823_RTBM_20120614_1715_0_0_0_00.MCESOLUTION.CSV
: command not found.sh: line 12:
: command not found.sh: line 13:
MCEASPRODUCTSUMMARY.sh: line 26: syntax error near unexpected token `done'
'CEASPRODUCTSUMMARY.sh: line 26: `echo "${prefix}$line"; done< "$file" > temp.CSV;
UPDATED SCRIPT:
#!/bin/bash
# MCE Repricing ASPRODUCTSUMMARY
# Moves file in desired folder and changes them the CSV file types
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS;
for files in $(ls *.MCESOLUTION); do echo $files $(mv -v $files 'UNPROCESSED/ASPRODUCTSUMMARY/'${files%}.CSV) ; done ;
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY;
# below will delete unnecessary lines from the file in current directory
FOLDERDIR="/u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY";
cd $FOLDERDIR;
for i in $(ls -l | grep '^-' | awk '{print $9}')
do
grep -i "^.............ASPRODUCTSUMMARY"< "$i" > "$i"; done;
# to add case id and studytype value in front of each line of each file(N files)
# for example caseid='1400009823' and studytype ='RTBM' which are
# extracted from filename and then appended in front of each line of a line
# and then repeat for N number of CSV files in current directory
for file in '$(ls -l $FOLDERDIR)'; do
caseid=$(echo $file | cut -c1-10);
studytype=$(echo $file | cut -c12-15);
prefix="$caseid,$studytype,";
while read -r line
do
echo "${prefix}$line"; done< "$file" > temp.CSV;
mv -v temp.CSV ${file};
done;
exit 0;
As far as I can see, the script is intended to analyse the names of files found in a directory and produce a list of those files (prefixed by two comma-separated fields derived from the filename) in a file called temp.CSV in the same directory. This however does not tie up with the description in your previous post which implies some data edit.
Before you end up trashing every file in your directory with "trial-and-error" scripting, please post sample filenames and a full description of the process complete with matching sample output.
Also, please use code tags for code and data samples.
It is never necessary or desirable to end a line of a Bourne Shell script with a semi-colon (except for the one exception of certain syntax of the "find" command). This is unix script language not Oracle or Perl.
Please use the CODE and /CODE tags around your scripts. (Highlight the code and then press the symbol in the Message: menu that has the work CODE in a rectangular box with a hand pointing into the box.) When you don't, HTML processing can significantly change the meaning of what appears to be your code. Please either edit you last posting to include these tags, or repost your code (with tags) in a new message.
Note, however, that you still have the command line:
seems strange. The mv -v writes the source and target files to stdout, and when combined with the echo $files in front of that, you're writing the source file twice on these output lines. Then the ${files%} is missing the word that identifies the pattern that is to be removed from the end of the expansion of the files variable.
Hi Don, Yes, that true I need to delete all lines from the file in current directory not having word "ASPRODUCTSUMMARY" on 15th position.
Also the thing you mentioned of the code line :
No. It will delete all lines no matter what is on them; not just lines not having the text you're looking for.
The use of any utility, for example x , will get rid of all data in file when you issue the command:
x < file > file
This will be done by the shell before x even starts running.
Line 20 in your script:
for file in '$(ls -l $FOLDERDIR)'; do
sets $file to be a line of output from ls -l when it seems highly unlikely that you want an expansion of $file to be anything other than the file's name. Also, since line 11 of your script is:
cd $FOLDERDIR;
and you haven't changed directory between these two lines, I think you just want line 20 to be:
for file in '$(ls)'; do
You probably got lots of syntax errors from using $file instead of "$file" on lines 21 and 22. Even when $(ls) instead of $(ls -l), if any of your filenames contain characters that have meaning to the shell you're using, you need to use double-qoutes around $file. You should get into the habit of surrounding the expansion of any variable that may contain characters that the script itself did not produce. You never know when a user will try to hack your script by including characters that are special to the shell!
Hi Don, I truly appreciate your help.
Here is the script I changed and have placed "$file" everywhere it was.
Script is working fine upto line 13 (where grep command is redirecting to newtemp.csv file (with a bug that it will do it for the first file and second file remains untouched after it was moved to the ASPRODUCTSUMMARY subfolder.
Also giving syntax error at line 15 where there is
for file in $(ls -l)
do
.
I changed everything as you suggested. I am attaching some sample files that this script is processing.
#!/bin/bash
# MCE Repricing ASPRODUCTSUMMARY
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS;
for files in $(ls *.MCESOLUTION);do $(cp -v $files 'UNPROCESSED/ASPRODUCTSUMMARY/'${files%.MCESOLUTION}.CSV); done;
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY;
####ls -1 /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY > ListofMCECSVFilesASPRODUCTSUMMARY.txt
FOLDERDIR='/u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY';
cd $FOLDERDIR;
for i in $(ls -l | grep '^-' | awk '{print $9}')
do
$(grep -i "^..............ASPRODUCTSUMMARY"< "$i" > "newtemp.csv");$(mv -v newtemp.csv "$i"); done;
for file in $(ls -l)
do
caseid=$(echo "$file" | cut -c1-10)
studytype=$(echo "$file" | cut -c12-15)
prefix="$caseid,$studytype,"
while read -r line
do echo "${prefix}$line" ;done< "$file" > "temp.CSV"; $(mv -v temp.CSV "$i");
done;
exit 0;
below is the latest script output error :
Error on line 12 newtemp.csv is rather processing correctly for 1 out of 2 files, and didn't process the second file and left it as it is. then gives error
Please provide the basic information I requested in Post #5.
Bottom line: Your script is a disaster. It needs a total rewrite. Only certain syntax issues are stopping the script destroying all files in the directory (as others have advised).
Please stop running trial-and-error on live data, and publish the specification for the process.
Thank you for your concern and help man. I agree the script must be a disaster for sure.
The Specifications are :
1) Will copy the data to a subfolder and rename them as .CSV files. Which now successfully being achieved by the following code:
#!/bin/bash
# copying MCE Repricing ASPRODUCTSUMMARY
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS;
for files in $(ls *.MCESOLUTION);do $(cp -v $files 'UNPROCESSED/ASPRODUCTSUMMARY/'${files%.MCESOLUTION}.CSV); done;
exit 0;
2) Delete all the lines from each file which are not having "ASPRODUCTSUMMARY" word in the file in the subdirectory files were copied(that's why the original files provided were copied so that we don't destroy the entire data but have our own version of file copied and deleting all unwanted lines). This is also being successfully achieved by the following code:
#!/bin/bash
# MCE Repricing ASPRODUCTSUMMARY ONLY DATA
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY;
####ls -1 /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY > ListofMCECSVFilesASPRODUCTSUMMARY.txt
FOLDERDIR='/u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY';
cd $FOLDERDIR;
for i in $(ls -l | grep '^-' | awk '{print $9}')
do
$(grep -i "^..............ASPRODUCTSUMMARY"< "$i" > newfile);
$(mv -v "newfile" "$i"); done;
exit 0;
3) In each file modified with the lines required. Want append each line of each file with CASEID and STUDYTYPE which are extracted from the filename itself. For example if the filename is 1400009823_RTBM_20120614_1715_0_0_0_002.CSV so the caseid ='1400009823' and studytype='RTBM'. So this should repeat for each file and append in front of each line. This is the challenge (:wall:) right now.
The code for this which is giving syntax error :
When running while loop for individual file one by one it runs perfectly fine, but gives syntax error when trying to run for multiply files using "for loop".
#!/bin/bash
# MCE Repricing ASPRODUCTSUMMARY Append Casid n studytype in each line
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY;
####ls -1 /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY > ListofMCECSVFilesASPRODUCTSUMMARY.txt
FOLDERDIR='/u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY';
cd $FOLDERDIR;
for i in '$(ls -l)'
do
caseid=$(echo "$i" | cut -c1-10)
studytype=$(echo "$i" | cut -c12-15)
prefix="$caseid,$studytype,"
while read -r line
do echo "${prefix}$line" ;done< "$i" > "temp.CSV"; $(mv -v "temp.CSV" "$i");
done;
exit 0;
I did try putting
for i in "$(ls -l)"
on line 10 in single quote, double quote. ; before do, nothing helps resolving the syntax error.
Thank you
Methyle BASIC Specifications are :
1) Will copy the data to a subfolder and rename them as .CSV files.
2) Delete all the lines from each file which are not having "ASPRODUCTSUMMARY" word in the file in the subdirectory files were copied
3) append each line of each file with CASEID and STUDYTYPE which are extracted from the filename itself. For example if the filename is 1400009823_RTBM_20120614_1715_0_0_0_002.CSV so the caseid ='1400009823' and studytype='RTBM'. So this should repeat for each file and append in front of each line
You didn't make all of the changes I suggested and you made some changes I didn't suggest. Let's analyze the commands being run inside the for loop. (I've removed the semicolons and split the commands onto separate lines to make it easier to read.) With the reformatting, you have:
Since you have $(...) around both of these commands, the shell executes the grep (which succeeded the first time) and then tried to execute the output produced by the grep. I assume this gave you the first command not found error.
Then the mv command executes producing output that looks something like
and then tries to execute that string (which I assume gave you a
error message.) Remove the $( and the matching ) from both of these commands and you should make some progress.
Then you have a for loop starting with:
for file in $(ls -l)
this executes the loop nine times for every file found in the directory. The first time through the loop, $file will be something like -rw-r--r--; the second time through the loop, $file will be the number of links to the file; ...; and the ninth time through the loop for each file, $file will be set to the name of a file. I had suggested that you change your original '$(ls -l $FOLDERDIR)' to '$(ls)' . I made a mistake leaving in the single quotes (and you corrected that. Thank you. But leaving $(ls -l) instead of $(ls) gives you a big problem.
Then in the final while loop you have another mv -v command inside $(...) which I assume is giving you several more command not found errors. Get rid of the $( and the matching ) again here.
Please also reformat this. It has been noted (and you agreed) that this code is a disaster. It looks like you are trying to minimize the number of lines of code in your script instead of trying to make it easy to understand. This may make sense when you have a simple program that can be written as a single line; but in a case like this it just makes your script unreadable. Don't you agree that:
for i in $(ls -l | grep '^-' | awk '{print $9}')
do
grep -i "^..............ASPRODUCTSUMMARY" < "$i" > "newtemp.csv" mv -v newtemp.csv "$i"
done
for file in $(ls)
do
caseid=$(echo "$file" | cut -c1-10)
studytype=$(echo "$file" | cut -c12-15)
prefix="$caseid,$studytype,"
while read -r line
do echo "$prefix$line"
done < "$file" > temp.CSV
mv -v temp.CSV "$i"
done
exit 0
The code above is an exaggeration of the formatting you're using, but not by much.
There are some problems that will arise even with this if any of the file names encountered while running ls contain any spaces, tabs, or newlines. I will assume for now that these directories are always being updated programmatically and that other users don't have permission to create any files in these directories.
Hi Don, Thanks for explaining how linux OS will behave on my previously provided coding. That's why it was giving so many errors.
Can you help me with the last peace, now I was able to run upto
for i in $(ls -l | grep '^-' | awk '{print $9}')
do
grep -i "^..............ASPRODUCTSUMMARY" < "$i" > "newtemp.csv" mv -v newtemp.csv "$i"
done/
I ran the following code you reformatted:
#!/bin/bash
for i in $(ls -l | grep '^-' | awk '{print $9}')
do
grep -i "^..............ASPRODUCTSUMMARY" < "$i" > "newtemp.csv" mv -v newtemp.csv "$i"
done
for file in $(ls)
do
caseid=$(echo "$file" | cut -c1-10)
studytype=$(echo "$file" | cut -c12-15)
prefix="$caseid,$studytype,"
while read -r line
do echo "$prefix$line"
done < "$file" > temp.CSV
mv -v temp.CSV "$i"
done
exit 0
The problem is at line 8
for file in $(ls)
do
. The code you provided above is giving syntax error on this line 8.
What I did is, I have separated the tasks, that is I made a separate script to copy files. I made separate script to delete unwanted lines from the copied files. and the finally separate script to append
and
in each line of each file. but this last script is not working when run on multiple files. Please help. How will you code such requirement.
The code that I reformatted was only the last part of your script. I thought I had talked you through the changes that needed to be made to the first part.
As I mentioned in-line in the quote from your message, I made a cut and paste error. One line in the reformatted code I posted should have been:
but was a single line in what I pasted. I imagine that this is what is causing many of your problems. I sincerely apologize for causing this confusion.
I have no idea what code you are now using. After splitting the code into three scripts I have no idea whether the variables you are using in the second and third scripts were set in the first or second scripts and are, therefore, unset or incorrectly set in the later scripts. Please post each script that you now have as separate code segments (with the CODE and /CODE tags) and put the names of each script in the description of what each code segment is. Also, please include an example of a line of output from the
so we have some idea of what might appear in the values of $prefix.
I have meeting later this morning and am going to have to actually go to bed soon. If you can give me a complete, clear listing of the code you're now using and describe the differences between what you're getting and what you want I may be able to go through this again this evening. (It is now after 1:30 AM PDT where I'm located.) :wall:
Hi Don,
I have posted the separate script now I am using in POST#11.
1) code is coping files only :
#!/bin/bash
# copying MCE Repricing ASPRODUCTSUMMARY
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS;
for files in $(ls *.MCESOLUTION);do $(cp -v $files 'UNPROCESSED/ASPRODUCTSUMMARY/'${files%.MCESOLUTION}.CSV); done;
exit 0;
2) second script is deleting unwanted lines from the files copied in above script #1
#!/bin/bash
# MCE Repricing ASPRODUCTSUMMARY ONLY DATA
cd /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY;
####ls -1 /u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY > ListofMCECSVFilesASPRODUCTSUMMARY.txt
FOLDERDIR='/u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY';
cd $FOLDERDIR;
for i in $(ls -l | grep '^-' | awk '{print $9}')
do
$(grep -i "^..............ASPRODUCTSUMMARY"< "$i" > newfile);
$(mv -v "newfile" "$i"); done;
exit 0;
ABOVE two script s are working perfectly fine and doing they are are supposed to.
3) I need help in the last script below, which should extract the caseid and studytype from filename itself and then append these two values(separated by ,) in front of each line of each file provided by the
[list]
of
for
loop. The code for this I have is:
#!/bin/bash
#!/bin/bash
# MCE Repricing ASPRODUCTSUMMARY Append Casid n studytype in each line
FOLDERDIR='/u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY';
cd $FOLDERDIR;
for f in $(ls *.CSV); do caseid="$(echo "$f" | cut -c1-10)";
studytype="$(echo "$f" | cut -c12-15)";
prefix="$caseid,$studytype,";
echo $prefix;
while read line
do echo "${prefix}$line" done < "$f" > "newtempfile";
mv -v "newtempfile" "$f";
done;
exit 0;
Getting syntax error
---------- Post updated at 04:02 PM ---------- Previous update was at 09:39 AM ----------
Thank you all who tried to help me.
the following coding resolved the problem and I could achieve what I needed to:
#!/bin/bash
# MCE Repricing ASPRODUCTSUMMARY Append Casid n studytype in each line
FOLDERDIR='/u02/sppdw/dw/infa_shared/SrcFiles/POPS/UNPROCESSED/ASPRODUCTSUMMARY';
cd $FOLDERDIR;
for f in $(ls *.CSV); do caseid="$(echo "$f" | cut -c1-10)"
studytype="$(echo "$f" | cut -c12-15),"
prefix="${caseid//[[:space:]]/},${studytype//[[:space:]]/}";
while read line; do echo -n "${prefix}$line" ;done< "$f" > newtempfile;
$(mv -v "newtempfile" "$f");
done;
exit 0;
putting
//[[:space:]]/
was also useful to eliminate line characters otherwise it was adding a new line each time.
Going to close this post as resolved.