running a looping script for all files in directory

I have three different linux command scripts that I run for 20+ files in one directory.

it goes like this
FIRST SCRIPT:
grep 'something' -w file > newfile1
.
.
.
grep 'something -w file > newfile20

then I take all these 'newfileN' and run this:
awk 'BEGIN { format="%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n" } { printf(format,$1,$15,$4,$4,$4,$4,$2,$3,$4,$4)}' newfile1 > new.newfile1
.
.
.
awk 'BEGIN { format="%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n" } { printf(format,$1,$15,$4,$4,$4,$4,$2,$3,$4,$4)}' newfile20 > new.newfile20

then finally I take these 'new.newfileN' and do this:
sed -e 's/[+-]/U0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/+/F/1' -e 's/-/R/2' -e 's/[+-]/../2' -e 's/chr//1' new.newfile1 > new.new.newfile1
.
.
.
sed -e 's/[+-]/U0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/+/F/1' -e 's/-/R/2' -e 's/[+-]/../2' -e 's/chr//1' new.newfile20 > new.new.newfile20

So, is there a way I can make this all into one script so instead of copying and pasting each scripted line 20+ times.

thanks

Try this perhaps (untested):

for f in $(seq -f file%g 1 20)
do
    grep 'something' -w $f > new$f
    awk -v OFS='\t' '{ print $1,$15,$4,$4,$4,$4,$2,$3,$4,$4 }' new$f > new.new$f
    sed -e 's/[+-]/U0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/+/F/1' -e 's/-/R/2' -e 's/[+-]/../2' -e 's/chr//1' new.new$f > new.new.new$f
done

If you don't need all of the temporary files in between:

for f in $(seq -f file%g 1 20)
do
    grep 'something' -w $f |
    awk -v OFS='\t' '{ print $1,$15,$4,$4,$4,$4,$2,$3,$4,$4 }' |
    sed -e 's/[+-]/U0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/+/F/1' -e 's/-/R/2' -e 's/[+-]/../2' -e 's/chr//1' > new.$f
done

Thank you, I haven't tried but I have question before I do. Sorry if this is an obvious question, but how do I execute this? Do I copy it into a file and then call it from the command line?

Okay, I think I figured that out. i created a file.sh (with the proper heading and chmod it to 755), then ran it
> sh file.sh

then I got a return error

seq: command not found

looks like my mac doesn't have seq. can i use jot? if so, how would i set it up?

thanks a bunch!

Why did you say "linux command scripts" if you are using a Mac??

Change the first line to for f in file1 file2 file3 ... up to file20 and it should work. Or else use a counter...

You shouldn't need to run it with sh. Once you have made it executable, either use ./file.sh (if you are in the same directory), or use the full path, e.g. /some/directory/file.sh.

Sorry about the confusion. I'm still trying to get a handle on all this. I'm connected to my mac at work via ssh (using my PC from home).

I did what you suggested and it worked like a charm.

but is there a way to forgo writing 20 file names in the first line and instead call them all like you initially had it?

Yes, there is:

i=1
while (( i<=20 ))
do
    grep 'something' -w file$i |
    awk -v OFS='\t' '{ print $1,$15,$4,$4,$4,$4,$2,$3,$4,$4 }' |
    sed -e 's/[+-]/U0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/+/F/1' -e 's/-/R/2' -e 's/[+-]/../2' -e 's/chr//1' > new.file$i
    (( i=i+1 ))
done

Thank you Annihilannic. It works

Hi, I'm getting a new error and I can't seem to understand why.

'/convert.sh: line 2: syntax error near unexpected token `do
'/convert.sh: line 2: `do

for f in file1 file2 file3
do
    awk -v OFS='\t' '{ print $1,$15,$4,$4,$4,$4,$2,$3,$4,$4 }' $f |
    sed -e 's/[+-]/U0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/[+-]/0/2' -e 's/+/F/1' -e 's/-/R/2' -e 's/[+-]/../2' -e 's/chr//1' > $f.new
done

Not sure what's going on?

use code like:-
t=`ls -l |wc -l`
total=`echo "$t -1"|bc`
while [ $total -le 1 ];
do

steps to do;
$total=`"echo $total -1"|bc`
done

thanks for replying.

I got this error when I copied your code

I copied only

t=`ls -l |wc -l`
total=`echo "$t -1"|bc`
while [ $total -le 1 ];
do

steps to do;
$total=`"echo $total -1"|bc`
done

The "steps to do" was meant as a "fill in the blank" comment, I guess.

The use of ls -l if you don't actually use the long format for anything is obviously Useless. Fixing that also removes the need to remove the "total" line from the sum.

But anyway, if you don't use the count for anything, either, just loop over the file names.

for file in *; do
  # stuff
done

Tangentially, I guess RahulJoshi really meant -ge rather than -lt

Okay, so is the error I'm getting in post #10 above because of a formatting issue? I'm not sure I understand what is going on. It was working before then it's not???

Is there a way for me to trouble shoot this?

I'm not sure either... is that all of your code that you have posted in #10, or is there more in the script? Are the filenames exactly as you have posted (i.e. file1 file2 file2?

yes, that's the exact code, nothing more. I even tried changing the quotes from ' to `. Same error. And all the files are right in the same directory...

What shell are you in? And how exactly are you executing this script?

Try adding a shebang line as the first line of the script to enforce ksh (using the correct path to ksh on your system of course):

#!/usr/bin/ksh

I typed 'echo $SHELL' and it returned:

/bin/bash

I tried using the shebang line but it didn't work. it stated:
-bash: ./loop.sh: /usr/bin/ksh^M: bad interpreter: No such file or directory

I created a new sh script with this

echo "Hi there.";

and after I chmod to 755 and typed ./file.sh it worked with no error. this is telling me that there's a format issue with the original one, but I can't find it....

Ah-ha!

The ^M holds the key... you must be creating these scripts under Windows and uploading them to Unix, right? Windows and Unix use different text file formats. Windows ends lines with Carriage Return and Line Feed characters (CR/LF), whereas Unix just uses a Line Feed (LF). You need to convert the script file(s) to Unix format using dos2unix windowsfile > unixfile or tr -d '\r' < windowsfile > unixfile.

You may find life simpler if you learn how to use a Unix editor and edit them directly on the system where you are running them.

oh that makes sense! i learned my lesson! emacs is now my new friend