Merging of rows

ssachins · January 5, 2009, 5:16am

Hi guys,

Wish you all a very Happy New Year!!!.

Thanks in advance.

Type|FxDiscreteDNT|TradeID|SSA32553H|SystemID|GGT04481|SourceSystem|RMS|SourceSystemBook|OT|Book|OT|BookBranchParent|Go|Version|8|LatestVersionYN|Y|Counterparty|DOB|Status|KnockedOut|IsLiveYN|N|DBLegalEntity|London|LastUpdated|JAN 2009 00:26:35|Action|KnockedOut|CounterpartySourcesystem|RMS|TradeDate|JUL 2006|ProductID|KKI99106|StructureID|DDS32553|DealCaptureSystem|RMS|CustomerSourceSystem|RMS|Customer|ESL|PricingSite|Option|UnderCurrency1|JPY|AccCurrency1|US|UpBarrier1|1.5|LowBarrier1|1|UnderCurrency2|JPY|AccCurrency2|CH|UpBarrier2|1.64|LowBarrier2|1.49|UnderCurrency3|EUR|AccCurrency3|JPY|UpBarrier3|175|LowBarrier3|105|LastFixed

The outout should be
The below record should be in one line.

I tried using below mentioned script but AWK can't handle > 20,000 bytes. Also, using String as 'LINE_BREAK' for identification rows got merged.

FileName=LINE_TEST.TXT
FileName1=O_LINE_TEST.TXT
OUTPUT_FILE=OUTPUT_WITHOUT_LINE_BREAK.tmp
if [[ -e $FileName ]]; then
sed "s/%/percentage/g" $FileName > $FileName1
## To remove '%' as 'awk' can not handle it.

/usr/xpg4/bin/awk 'NR==1\{printf $0;next\}
!/^TradeType/ \{printf "LINE_BREAK" $0;next\}
\{printf "\\n" $0\}
END\{print ""\}
' $FileName1 &gt; $OUTPUT_FILE

fi

radoulov · January 5, 2009, 6:19am

Try with nawk:

nawk '$1=$1' FS='\n' RS= ORS='\n\n' infile

ssachins · January 5, 2009, 6:45am

I am getting the below mentioned error :

13460 Segmentation Fault(coredump). Is it related Memory ?

The Actual File size is : 95532469 bytes.

radoulov · January 5, 2009, 6:48am

Try with Perl:

perl -00ple'tr.\n. .' infile

ssachins · January 5, 2009, 6:53am

How to identfy which lines were merged while processing the file ?

radoulov · January 5, 2009, 7:06am

Just append some string to identify them:

perl -00ple'tr.\n. . and $_.=" __FIXED__"' infile

ssachins · January 5, 2009, 7:25am

Hi Radoulov,

It's working !!!

Thank you very much for quick reply. Appriciated !!!

One last request

Could you please confirm how this script works ?

rujuta_rahalkar · January 5, 2009, 7:28am

Have a doubt for the below code --

nawk '$1=$1' FS='\n' RS= ORS='\n\n' infile

In this code there is no print statement. Even then we get the output after running this script. How is that possible ?
Also did not get the meaning of the code '$1=$1' . What does it do exactly ?

radoulov · January 5, 2009, 7:36am

I believe you meant explain.
It just transliterates the newlines to spaces inside the paragraphs:
the 0 switch with the 0 argument (-00) cause Perl to slurp files in paragraph mode
(one or more lines separated by one or more blank lines),
tr transliterates the \n to a space.

radoulov · January 5, 2009, 7:44am

rujuta_rahalkar:

Have a doubt for the below code --
nawk '$1=$1' FS='\n' RS= ORS='\n\n' infile
In this code there is no print statement. Even then we get the output after running this script. How is that possible ?

It's because in AWK the default action is print the current record:

% print OK|awk 42
OK

Quoting Ed Morton:

ssachins · January 5, 2009, 7:53am

frankly speaking, I never worked in 'perl', hence unable to understand the code.
Is it possible for you to explain the code step by step?

Thanks.

radoulov · January 5, 2009, 9:00am

Well,
in this case I believe you need something different.
You could start here.

ssachins · January 5, 2009, 10:37am

Hi,
I forgot to ask, How this script will check whether the next line starts with word 'Type|' and if it is then should not merge with first row.

*When I ran the script, its concating both the lines mentioned below which should not happen. (these lines should not merge)

radoulov · January 5, 2009, 10:39am

OK,
please post a bigger sample of your real data that includes both cases.

ssachins · January 5, 2009, 11:00am

Hi,

Please find the sample data. The file contains 3 rows. First row breaks into many lines, row 2nd and 3rd are without any line breaks. So, when the script runs, row starts with word 'Type|', should not merge with previous row but if not then should merge with previous line.

Row 1

Each cashflow : notional 0.0500 million

The Output Should be

Row 1

Row 2

radoulov · January 5, 2009, 11:34am

Try this:

perl -nle'
  print $x and undef $x 
    if $x and /^Type/ or eof;
   ($x .= $_) =~ tr/\n/ /s;
  ' infile

ssachins · January 5, 2009, 12:16pm

How to identify which rows got Merged ?

It's not working if I modify the script as below

($x .= $_) =~ tr/\n/LINE_BREAK/s;

radoulov · January 5, 2009, 2:21pm

Actually tr is not needed because of the -l switch.

Try this:

perl -nle'
  print $x, $f > 1 ? "_FIXED_" : "" 
    and ($x, $f) = undef 
    if $x and /^Type/ or eof;
  $f++ if $x .= " ". $_ and !/^$/
  ' infile

You may add another linefeed if you need an extra blank line between the records.

ssachins · January 6, 2009, 4:02am

Hi,
if I execute the script, there will be one space(:() at the begaining of every line, the word '_FIXED_' comes only at the end of file but not where the line was break (_FIXED_) .

Row 1

frostmourn · January 6, 2009, 4:51am

sed '/^Type/{:a;N;/\nType/!{s/\n/ /;ba};s/\n/&&/}'