Hi all,
I'm looking for some tips on an ideal method of parsing a huge fixed length flat file (~500gb) into a delimited text file. We have to do this because our data warehouse platform only accepts delimited file loads. In the past, we've done this with SAS (only on smaller ~40GB files) by importing into a SAS dataset using an input statement then dumping to a tab delimited text file using a simple proc export. I want to make this process more efficient and get SAS out of the process. I know this can all be done with Perl, Python, Java, etc. but I don't have any experience w/ those tools. Any suggestions or thoughts would be much appreciated.
One other item I forgot to mention is that the file contains 5 different file layouts which is identified by the first 2 bytes of the row (each row is 276 bytes wide). I provided a piece of my SAS code that shows 2 of the layouts. Thanks in advance.
SAS Code snippet:
INFILE MYFILE LRECL=276TRUNCOVER;
INPUT @1 RECTYPE $CHAR2. @;
SELECT (RECTYPE);
WHEN('CO') DO;
INPUT
@ 16 a $6.
@ 22 b $1.
@ 23 c $7.
@ 30 d $6.
@ 36 e $2.
@ 38 f $6.
@ 44 g $1.
@ 45 h $38.
@263 x $14.;
OUTPUT CO_DATA;
END;
WHEN('HD') DO;
INPUT
@ 16 aa $6.
@ 22 bb $2.
@ 24 cc $2.
@263 x $14.;
OUTPUT HD_DATA;
END;