This question has been asked many times, but my problem is slightly different.
In my shell script i am connecting to oracle database and loading the results to .dat file. This .dat file is later used to create to .xls file[to be sent to the users] Some times the size of .dat file becomes more than 120000 bytes. In these cases, the sed and other commands used in the script fails with cannot execute [Argument list too long] error.
I tried splitting the .dat file to smaller files, but i am not even able to split the file and it again throws me the same [Argument list too long] error.
The problem is not just with sed, but all commands used in the shell script.
My piece of code :
cat tgt_type_${tmp_file_suffix}i.dat | while read line
do
export FETCH=`sqlplus -silent ${USERNAME}/${PASS}@${DATABASE} <<EOF
whenever sqlerror exit 1
whenever oserror exit 1
set pagesize 2000
set linesize 2000
set underline off
set feedback off
set heading off
set trim on
spool missing_$line_${tmp_file_suffix}i.dat
/*Oracle sql query comes here */;
spool off
EOF`
fi
sed /^$/d *.dat
cat missing_$line_${tmp_file_suffix}i.dat | while read line
do
#
#coding
#coding
#
done
done
What does ls -l *.dat return? The same error? If so, that's the number of files, not their size - look at using man find ("posix") and possibly man xargs ("posix"). E.g. something like:
find . -type f -name "*.dat" | xargs sed /^$/d
Incindetally, sed /^$/d *.dat won't do anything to the actual files (unless you've aliased sed). Is that your intention?
Because of the kernel ARG_MAX value is limited and asterisk character expands the all matching files,directory,links or any other forms.
Therefore your bash(shell) command argument buffer's ( commandline length ) is fill and and then overflow.
You can use
for loop
while loop
or any scriptable methods
split the arguments to meaning values
ls -1 tgt_type_${tmp_file_*}.dat
xargs (to split to the arguments with find is one good method )
ls (use the folder hierarchy with asteriks )
ls -1 yourfolder/
find (use the folder hierarchy without asteriks )
find /yourfolder -name '*'
For example :
while read myfile ; do sed '/^$/d' $myfile ; done | find /yourdolfer -name '*.dat'|xargs -n1
# (use the while loop and read the arguments one by one (-n1) )
# (use the spaces instead of newlines with xargs and use getdents to get the file entries with find)
and consider the @CarloM 's notice..
your sed command's default output is your stdout (usually your screen tty,pts,console,...) ,
therefore you must redirect to another file for save the changes or use the '-i' parameter , if you have gnu sed for save to your file same time.
Hi ygemici, I am trying to implement your suggestions and i have made couple of changes to my script to use while loop and read from file.
But i am stuck in the below section:
$1F is the first field from the first line of the input file. Similarly $2F is the the second field from the first line of input file. This should loop for all the lines in the input file. Totally there are 13 fields in the input file. That is the reason i am using 13 awk commands!
Basically the fields in the input value are seperated by ";". Based on the number of records in the input file, there should be corresponing entries in the output file.
Please help on this issue.
