Split file based on field

Shell Programming and Scripting
1

Hi

I have a large file 2.6 million records and I am trying to split the file based on last column.

I am doing

awk -F"|" '{ print > $NF }' filename1

After around 1000 splits it gives me a error
awk: can't open file 3332332423
input record number 1068, file filename1
source line number 1

For different input files it gives error in different line numbers. I expect to create around 50,000 files. Is there any limitation?

Adarsh

2

What do you mean split the file based on the last column? You want to split the file into many files?

3

Yes

I have records like

a|b|b|c|c|dsda|ada|123
a|b1|b|c1|c|dsda|ada|223
a|b1|b|c1|c|dsda|ada|223
a|b|b|c|c|dsda|ada|423
a|b|b|c|c|dsda|ada|1234
a|b|b|c|c|dsda|ada|123

I require

123
a|b|b|c|c|dsda|ada|123
a|b|b|c|c|dsda|ada|123

223
a|b1|b|c1|c|dsda|ada|223
a|b1|b|c1|c|dsda|ada|223

1234
a|b|b|c|c|dsda|ada|1234

423
a|b|b|c|c|dsda|ada|423

Actually awk -F"|" '{ print > $NF }' filename1 is doing the job. My input file has 2.6 million records. It created some 1000 files and then giving a strange error

Please advice me

Adarsh

4 
awk -F"|" '{ print > $NF ; close($NF)}' filename1
1 Like
5

Thanks vgersh99
Its working

Was it something related to file not closing?

thanks again

6

if you don't close the output stream you run the risk of running over the MAXIMUM number of the opened streams which varies depending on your version of 'awk'.

7

thx again