Split large file to smaller fastly

hi ,

I have a requirement

input file:

1	1111111111111	108
1	1111111111111	109
1	1111111111111	109
1	1111111111111	110
1	1111111111111	111
1	1111111111111	111
1	1111111111111	111
1	1111111111111	112
1	1111111111111	112
1	1111111111111	112

The output should be,

108.txt:
======
1	1111111111111	108

109.txt:
======
1	1111111111111	109
1	1111111111111	109

110.txt:
======
1	1111111111111	110

111.txt:
======
1	1111111111111	111
1	1111111111111	111
1	1111111111111	111

112.txt:
======
1	1111111111111	112
1	1111111111111	112
1	1111111111111	112

the input file will be a huge file. based on the column i have split it by groups and filename should based the coulmn only.
if i use awk
it throwing an error:

awk: too many output files 10
 record number 11

Please give a command which will split the huge file to multiple files in less time.

$ awk '{f=sprintf("%s.txt",$NF); if(f in F){ print >>f }else { print >f ; F[f]} close(f) }' file

try

awk '{print > $3".txt"}' filename

if you are not closing you will get too many Too many open files error with large input file.

then like this .. will it?

awk '{print > $3".txt"; close($3".txt")}' file

This will overwrite file everytime, did you try ?

please give commands other than awk,

awk commands showing the error

awk: too many output files 10
record number 11

os :

SunOS sasbsd27c1 5.10 Generic_150400-10 sun4u sparc SUNW,SPARC-Enterprise

Which command did that? You were given actually 2 commands and - without having tried it - as much as i know about awk the one Ashkay Hedge gave seems to be OK.

While you are at it: some information about your system (like: which awk you use or, alternatively, which OS you use so we can deduce which awk version is involved and which other utilities can be expected as available) would not exactly hurt your cause either.

Further, I'd like to know if this is homework. The file contents look either very simplified - and, given how you come across, i wonder if you would be able to adapt a working solution to also work with the original file contents - or they are as they are, but then they seem so meaningless that they can only be dummy contents, like in a homework/classwork example.

bakunin

which OS you are using ? if sunos / solaris use nawk

SunOS sasbsd27c1 5.10 Generic_150400-10 sun4u sparc SUNW,SPARC-Enterprise

Hi.

Similar to others, and run on Solaris specifically:

#!/usr/bin/env bash

# @(#) s1       Demonstrate sifting, collecting lines to files, awk.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C awk

FILE=${1-data1}

# Remove debris.
rm -f [0-9]*.txt

pl " Input data file $FILE:"
cat $FILE

pl " Results:"
awk '
BEGIN          { outfile = lastfile = "" }
NR == 1        { lastfile = $3; outfile = $3".txt" ; print > outfile ; next }
$3 == lastfile { print > outfile; next }
               { close (outfile) ; lastfile = $3 ; outfile = $3".txt" ; print > outfile }
' $FILE
wc -l [0-9]*.txt

sample=109.txt
pl " Sample output: file $sample:"
cat $sample

exit 0

producing:

$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: SunOS, 5.10, i86pc
Distribution        : Solaris 10 10/08 s10x_u6wos_07b X86
bash GNU bash 3.00.16
awk - ( local: /usr/xpg4/bin/awk, Oct 10 2007 )

-----
 Input data file data1:
1       1111111111111   108
1       1111111111111   109
1       1111111111111   109
1       1111111111111   110
1       1111111111111   111
1       1111111111111   111
1       1111111111111   111
1       1111111111111   112
1       1111111111111   112
1       1111111111111   112

-----
 Results:
       1 108.txt
       2 109.txt
       1 110.txt
       3 111.txt
       3 112.txt
      10 total

-----
 Sample output: file 109.txt:
1       1111111111111   109
1       1111111111111   109

Best wishes ... cheers, drl

Hi All,

Thank you very much for your response.

all the scripts is not working in my machine.I have attached the file i am using and the error i am getting.

the script is able to create only 10 files after that it is throwing error.

Please test with the file and give a new command please.

Did you try post #2 ???

yes Akshay. that also throwed error

Please rerun and post output.. change awk to /usr/xpg4/bin/awk or /usr/xpg6/bin/awk or nawk

Thank u Akshay. it worked..

Thank u everyone :):)

Glad to know that your problem solved, please use solved tag.

awk '{f=sprintf("%s.txt",$NF); if(f in F){ print >>f }else { print >f ; F[f]} close(f) }' file

how it is determining it is 3rd column.

how to use in comma delimited file and how to specify the column name

Use this

awk -F, '{f=sprintf("%s.txt",$3); if(f in F){ print >>f }else { print >f ; F[f]} close(f) }' file

OR like this

awk  '{f=sprintf("%s.txt",$3); if(f in F){ print >>f }else { print >f ; F[f]} close(f) }' FS=',' file

OR like this

awk -vFS=',' '{f=sprintf("%s.txt",$3); if(f in F){ print >>f }else { print >f ; F[f]} close(f) }'  file

"NF" means "number of fields", therefore "$NF" is the content of the field number "number-of-fields", which is the last field always. Because your lines have 3 fields it is is field number 3.

I hope this helps.

bakunin