Implement in one line sed or awk having no delimiter and file size is huge

Shell Programming and Scripting
1

I have file which contains around 5000 lines.
The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters.

I want to delete the lines
a> if it starts with 1 and if 576th postion is a digit i,e 0-9
or
b> if it starts with 0 or 9(i,e header and footer)

I want to redirect all the lines to another file which does not meet above criteria.

Following is the file named A.DAT where i have placed only one line otherwise it would be huge

018012011---->header
120110116 000001000000000001200000.00USD0000000001.2000135058492000094000000.00PL 000.150000B 000000000000.00 0105.00000.000
0002011011820110121MODE0 D4028 27385 JP31368000044028 6466866 FJT140280 COMNIVBIHARA SANGYO Y50
BPY ABC000000000000.00Eabcdend VICVBERTTN0093.000
053.000000.00 20110118 05:57:31 000.00000010185450 TFLOW NBD NNF CHASNEAB MODFAN CHABD CDBV as agdnt (NE180 ESCROW)
UTCHBS
ABC.BADKS NMRSJPJT AD-SD5213
NIMIPA
ABC.BCDSS MHCBJPJ2 0000380
MIZUHO 000.150000000001200000.0000000100
0000 VICKRRTT NNNN NNN000000000000.00N0000N 00 000000000000
.00 0000000000.
0000000000000001000000 000.000000000.000000000.000000000000000000.00000000000000.00000000000000.00 000000000000.00 000000000
0.000000000000000000.00000000000000.000000000000.000000000000000000.00000000000000.00000000000000.00000000000000.00000000000000.0020110121000000000000.00Y0000
01136432.33 NY10185450 N0900080.00
0000000001.20000000000000000.0000000T000000000000.000000010000
00.00000001000000.00000001000000.00ABC 0100.00 0000000000
94567--->footer

I am using the following but it is giving error "sed: command garbled"

sed -e 's#^1\(.\{575\}\)[0-9]\(.*\)$##' -e's/^[9]//' A.DAT >> B.DAT

But if am running this command for a less no position like 57 instead of 575 ,it is working fine.

Can you pls someone tell me what is the issue here.And what would be the appropriate command using sed or awk.

2

Script looks fine to me - Perhaps it's a limitation of your sed?

How about using awk:

awk '!/^[90]/ && substr($0,576,1) !~ "[0-9]"' A.DAT >> B.DAT
3

In awk...

awk '!/^[09]/&&!(/^1/&&substr($0,576,1)~/[0-9]/)' A.DAT > B.DAT

---------- Post updated at 11:54 AM ---------- Previous update was at 11:49 AM ----------

In perl...

perl -n -e 'print $_ if !/^[09]/ && !(/^1/ && substr($_, 576, 1) =~ /[0-9]/)' A.DAT > B.DAT
2 Likes
4

Opps, I forgot the Starts with '1' criteria, Ygor's awk solution trumps mine.

5

Thanks a lot guys...

awk '!/^[90]/ && substr($0,576,1) !~ "[0-9]"' A.DAT >> B.DAT

it worked.

but whn i tried with

awk '!/^[09]/&&!(/^1/&&substr($0,576,1)~/[0-9]/)' A.DAT > B.DAT

it was giving error like the too long line..

Anyway,i got the solution..Thanks...........

Cheers!!!

---------- Post updated at 08:38 PM ---------- Previous update was at 08:16 PM ----------

oh!!! yes...how do i implement the criteria of starts with 1?
Ygor's command is giving error "too long line".

Ygor/Chubler,

Any other way?????

---------- Post updated at 08:48 PM ---------- Previous update was at 08:38 PM ----------

Hey Guys,

This worked for me.

nawk '!/^[90]/ && /^1/ && substr($0,576,1) !~ "[0-9]"' A.DAT > B.DAT

I think it also do the required operation on my file.

whats say?