A few things jump out at me straight off:
for toclean in `ls $FILE/*`
do
basetoclean=`basename $toclean .txt` # note 1 below
sed '1,/============/d' $toclean > $toclean.txt # note 2 below
sed -e '1d' -e '2d' -e '/^$/,/Output/d' $toclean.txt > $toclean.log #note 3
done
1) I don't understand your need to take the basename of the original file. Not to mention that you dont use the variable. Also, given your examples both indicate that the files have the suffix .log
it seems removing .txt seems wrong.
2) If $toclean.txt
is really just a temp file, then it is better to create a temporary file in /tmp than to use some form of the original filename. For example:
sed 's/foo/bar/' $toclean >/tmp/cleanscript_$$.tmp1
The name makes some previsions to avoid collisions if more than one instance of the script is running concurrently, and all temp files can be easily cleaned up with one rm command at the end.
3) It appears you are trying to overlay the original file, however since the sufix wasn't stripped from the original name you'll end up with something line path/file.log.log
Further, it's not a good idea to overlay the file directly (in my opnion). Disk fill, and other things causes commands to incorrectly generate, or fail to generate, output, so until you are sure that the command was successful, I'd write the second output to another temp file and check the return code. If the return code indicates success, then move the tmp file to the original file.
Finally remove any temp files that you created. Assuming all were created with the example name above:
rm -f /tmp/cleanscript_$$.*
Hope this helps you move forward.