I have a 1 million record file and in there i have 580 bad records meaning like they were spread on to 2 lines and is making my process fail as it is expecting single line records. How can i correct this records spread into 2 lines into single line or how i can writer these records into a new file and delete from the original flle so that i can worry about the bad records later. Please let me know.
dsravan,
Without a sample of the data we cannot help you.
Please, display a sample with the "good" and the "bad" records with no more than 10 or 15 records total.
how i can display a reocrd namely 20000 in a million reocrd file. I am getting the error temp file too large when trying to open in vi. Can i know the sed command to display that record.
Try using "less" or "more" commands, or give a sample of the first 10 as:
head -10 file_name
Iused had and tail commands and got the file
with naked eye it seems the file is ok. But the reocrd that is having problem is the 3rd one from above.
dsravan,
You said:
" have 580 bad records meaning like they were spread on to 2 lines"
The 3rd record in your sample is not spread on to 2 lines.
Could you please clarify?
Thanks.
sed -n 20000p
my process throws out erro syaing that record doesn't have enough number of vartext records in that file. I am not able to see with naked eye whats wrong with that line which is the bad record.
Don't know what to tell you. Maybe your count is off a bit, so try:
sed -n 19995,20005p
Here is a awk one-liner to find records without 12 fields..
$ cat data
321916335|CITY OF JASPER-R GSM AL WSCA CRU||||||4994||||
321916337|CITY OF JASPER-N GSM AL WSCA IRU||||||4994||||
321916341|NEW GENERATION CONSTRUCTION-N CBE CRU||||||4994|||||
321916343|NEW GENERATION CONSTRUCTION-N CBE IRU||||||4994||||
309514681|TRACFONE|TRACFONE|||||4994||||
$ awk -F'|' 'NF != 12 {print "record ", NR, " is ", $0 }' < data
record 3 is 321916341|NEW GENERATION CONSTRUCTION-N CBE CRU||||||4994|||||
$
I added an extra field to a record for testing.
here is the file contenets:
see that one record has span into two lines. I want to make them on to singlr line. It's a 1 millino file and i can't do that manually. please let me know how i can starighten them out.
assuming all the records/lines in your file are supposed to have the same number of fields, what IS this number? In other words, what is the number of '|' separated fields that your 'application' is expecting?
Get the line numbers for the bad two records... let's say 12001 and 12002...
sed -n 1,12000 > piece1 < oldfile
sed -n 12001,12002 > piece2 < oldfile
sed -n 12003,1000000 > piece3 < oldfile
vi piece2
cat piece1 piece2 piece3 > newfile
vgresh,
Each record has 11 pipes in it for sure. And there are 12 data values in each record.
to join 'broken' lines....
nawk -f ds.awk myBigFile.txt > myBigFile_new.txt
ds.awk
BEGIN {
FS=OFS="|"
FLD_max=12
}
# NF - number of fields in the current record
# if the number of fields in mthe current record is less than maximum number
# of fields - execute the action: if "rec" is NOT empty - concatenate it with
# the current record [rec OFS $0$0]; if "rec" IS empty - assign the current
# record/line to it [$0]. Increment the "running" number of fields [fld] by the
# number of the fields in the current record [NF]. Proceed to the next record
# [next].
NF < FLD_max {rec=(rec != "") ? rec OFS $0 : $0; fld+=NF;next }
# if the running number of fields [fld] is greater tan the MAX: output the
# "rec" and initialize "rec" to be empty and the running number of fields to be "0"
fld >= FLD_max { print rec; rec=""; fld=0 }
# if NONE of the above actions could be executed - "print" the current
# record/line: "1" is a shortname for "print $0"
1
Vgresh,
thank you very much.
can you please explain this.
Thanks,
Arun
vgresh,
I ran the script but still there are still records starting with "|" in the beginning which means it is spannig to 2 records.
Please help
explanation in the code.
post the offending records. I ran your first sample and it works fine.
vgresh,
The code is not working as desired becuase i am still having bad records spaning to 2 lines. Pleae help.