Apply command to all files in folder

Hi all!
I have this command

grep -E '^\To: |^\Date: |^\Subject: ' fileA.txt > fileA_1.txt && grep -v '^\To: |^\Date: |^\Subject: ' fileA.txt >> fileA_1.txt && rm fileA.txt && sed -i -e 's/\(Date: \|Subject: \|To: \)//g' fileA_1.txt

How do I apply it to all the files in the folder (each file has a different name, one is fileA.txt, but the rest of the files are different (another one is "175.txt", another one is "published3.txt"...)

In the end I should get fileA_1.txt, 175_1.txt, published3_1.txt ...
Thanks

*I am on Ubuntu 16.04.3 LTS

Hi guilliber,

Sometimes it helps to explain what you are trying to do, textually, than just what you have tried code-wise, otherwise (often wrong) assumptions can be made about your intent and lead to more questions than answers.

To perform an operation on multiple files implies iteration, which requires a loop to accomplish.

Well I have several text files with different names. The content of these files is diverse, but they all usually include a field with sender of an email, subject of email, date...
But then there are more lines that are less important.
My purpose is to move the lines I am interested in to the top of each file, and leave the rest below. Basically remove some cruft from each of the files using grep/sed...
I don't know how to accomplish this with all the files in the folder, as they have different names. Plus the resulting files of the process should have the initial name plus some kind of suffix (Example: original name 175.txt, after process 175_1.txt)
As you can see my command includes several steps, and maybe more steps in the future (file 175.txt > 175_1.txt > 175_2.txt...)

Thanks for the help!

Scott is correct: here is a start using your existing grep and sed logic. Which I think has some problems. Tinker with it, do not uncomment the lines that remove files, so you can run it till you get the sed the way you want.

I don't see how you determine the files to process
but let's assume you write the files you want to process to a text file.

ls *.txt > /tmp/list  # create a list of files
while read fname 
do

shortname=${fname%.txt}  # remove trailing .txt string
grep -E '^\To: |^\Date: |^\Subject: ' $fname > ${shortname}_1.txt && 
grep -v '^\To: |^\Date: |^\Subject: ' $fname >> ${shortname}_1.txt && 
sed -i -e 's/\(Date: \|Subject: \|To: \)//g' ${shortname}_1.txt

done < /tmp/list

# moved the rm out here for safety.  You cannot test code that removes files like that
# assume this worked correctly so: uncomment if true
# rm $(cat /tmp/list)
# rm /tmp/list

PS: stringing together in looong lines gains you very little and prevents making changes like I showed. You still need a loop.

Plus this code is not always going to work like you described - or so it seems to me.

OK, all seems to work perfectly! Thanks a lot!

I'm surprised that the -E option to grep should be necessary for the first but not for the second grep ?

Instead of invoking grep twice and sed once for each file to be processed, one might also consider just using ed :

for fname in *.txt
do	if [ "$fname" != "${fname%_1.txt}" ]
	then	printf '%s has already been processed.\n' "$fname"
		continue
	fi
	newfname=${fname%.txt}_1.txt
	printf 'Creating "%s" from "%s"...\n' "$newfname" "$fname"
	ed -s "$fname" <<-EOF && echo rm "$fname"
		H
		g/^Subject: /.m0
		g//.m0\\
		    s///
		g/^Date: /.m0
		g//.m0\\
		    s///
		g/^To: /.m0
		g//.m0\\
		    s///
		w $newfname
		q
	EOF
done

Note that this will skip files with names ending with _1.txt and that it will put all To: lines first in the output, followed by all Date: lines, followed by all Subject: lines (each of the above with the keywords removed), followed by the remaining lines in the original file. If you know beforehand that there will never be more than one of the header lines of each of the three header line types, this code could be further simplified.

Note that the above code uses tabs (not spaces) to indent lines. If you convert those tabs to spaces, the shell won't find the end of the here-document.

Note also that this code uses:

echo rm "$fname"

which just prints the command to remove the old files. If, after testing, you find that the code does what you want; remove the echo to actually remove the old files.

Hi.

This can be thought of as an ordering problem. Usually we think of situations like this in terms of character sorts.

However, we could also specify a set of strings to be used as the ordering sequence. In this case:

To:
Date:
Subject:

So we place these strings into a sort-order file, and sort the data files. Program msort can handle these kinds of tasks. Here is an example with 2 simple data files:

#!/usr/bin/env bash

# @(#) s1       Demonstrate sort, ordering by character strings, msort.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C dixf

pl " Input sort-order file sort-order:"
head sort-order

pl " Input files data*:"
head data*

pl " Results:"
for j in data*
do
  msort -q -j -l -n 1,1 -s sort-order $j
  pe
done

pl " Some detail about msort:"
dixf msort

exit 0

producing:

$ ./s1 

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.9 (jessie) 
bash GNU bash 4.3.30
dixf (local) 1.54

-----
 Input sort-order file sort-order:
To:
Date:
Subject:

-----
 Input files data*:
==> data1 <==
Subject:        Email test 1
From:   drl
To:     drl
Date:   Today 05:36
Hi ... cheers, drl

==> data2 <==
To:     drl
Subject:        Email test 1
Hi ... cheers, drl1
From:   drl
Date:   Today 06:36

-----
 Results:
To:     drl
Date:   Today 05:36
Subject:        Email test 1
From:   drl
Hi ... cheers, drl

To:     drl
Date:   Today 06:36
Subject:        Email test 1
From:   drl
Hi ... cheers, drl1


-----
 Some detail about msort:
msort   sort records in complex ways (man)
Path    : /usr/bin/msort
Version : 8.53
Type    : ELF64-bitLSBexecutable,x86-64,version1(SYSV ...)
Help    : probably available with -h,--help
Repo    : Debian 8.9 (jessie) 
Home    : http://www.billposer.org/Software/msort.html (pm)
Home    : http://billposer.org/Software/msort.html (doc)

The msort program is slower than the usual system sorting code, sort , but has a number of additional features that make its use very attractive.

As this demonstration illustrates, a single call to msort handles this task. The creation (and maintenance) of the sort-order file can be done easily with a text editor.

Best wishes ... cheers, drl