Unix commands failing inside the shell script

Shell Programming and Scripting
1

When my script deals with large input files like 22Gb or 18 GB the basic commands like sort or join fails when run from inside the shell scripts. Can there be any specific reason for this?

For e.g.
sort -u -t "," -k1,1 a.csv > a.csv.uniq"
sort -u -t "," -k1,1 b.csv > b.csv.uniq"

The first sort fails in the midway and begins the sort on the b.csv without throwing any error.

When run from the command line they run perfectly..they are giving me a problem only when they are run from a shell script. Does the HUGE file size has anything to do with this??

2

thats because there is not enough space in
/var/tmp area as sort by default sort uses /var/tmp area
mention directory where you have more than 22GB area with sort command

sort -T directorypath -u -t "," -k1,1 a.csv > a.csv.uniq
3

i changed the code to

sort -T /home/esha/tmp/ -u -t "," -k1,1 a.csv > a.csv.uniq
sort -T /home/esha/tmp/ -u -t "," -k1,1 b.csv > b.csv.uniq

and I have abt 120GB free. But still the problem is occurs...

4

Try it with more steps per file, an example:

awk '/^[a-hA-H]/' file | sort .... > file1
awk '/^[i-pI-P]/' file | sort .... > file2
awk '/^[q-zq-Z]/' file | sort .... > file3

cat file1 file2 file3 > csv.uniq