Creating Header & Trailer for bulk volume data file

Shell Programming and Scripting
1

Hi all,

I have a requirement to create a Header &Trailer for a file which is having 20 millions of records.

If I use the following method, i think it will take more time.

cat "Header"> file1.txt
cat Data_File>>file1.txt
cat "Trailer">>file1.txt

since second CAT command has to read all the data(20M records) from file and has append into file1.txt.

Is there any better optimized method to create Header&Trailer through UNIX script for this kind of BULK volume data files.

-- Thanks
--Raam C.

2

In general, calling cat with just one argument is a useless use of cat. Instead of running cat 3 times, just do:

cat header datafile footer > file1.txt

That's a bit more optimized than your way.

Beyond that, no; there's no UNIX system call to insert data before other data in a file, and I don't know a system that does. Nor is there a faster form of read() than the general purpose one, why would anyone ever use the "slow" one?

If whatever's using these datafiles can accept input on stdin, you could generate them at runtime to avoid the extra reads and wasted storage:

cat header datafile footer | command
3

my data file is having around 20 millions of records, it is a fixed width format file.
Now I want to create Header and trailer like this...

Hcurrentdate"                  "
data"                             "
data"                             "
T0000004"                      "

Header will start with letter 'H' followed by currentdate, followed by 20 spaces.
Then data records will come with same length
Trailer record will start with 'T', number of records in file(including header&trailer) in 10 length filed, followed by 20 spaces.

Please suggest me how to implement all this logic in script?

Thanks in advanvce...

--Raamc

4

try this..

awk 'BEGIN{print H'`date`'<put spaces here>}{1}END{print TNR<put spaces here>}' filename > newfile
5

vidyadhar,

This is not working, giving the error as below

awk: illegal statement
 input record number 1, file 108Learn.sh
 source line number 1

Any other ideas?