Multiple headers in a file

Shell Programming and Scripting
1

Hi ,

I have a .txt file in which I have multiple headers, the header record starts with $ symbol...like the first column name is $Account.

I have to keep the header in the first line and delete all the remaining headers which are in the file.

I tried using sort adc.txt | uniq -u , but my file is really big and I don't want to so this..., is there any other way we can delete the header based on the first column i.e $ symbol from all the places except the first place...the first header record should exist for loading my data in to datastge.

Regards,
Deepti

2

Please show us input and desired output - example files

3

Assuming your .txt file looks like the following:

$Account AAA BBB
$HEADER1 CCC EEE
$HEADER2 DDD GGG
THIS_IS_NOT_HEADER FIELD2

awk 'NR==1 || $1 !~ /^\$/' data.txt

The script above will output:

$Account AAA BBB
THIS_IS_NOT_HEADER FIELD2
4

Hey ,

Below is my sample source data.

$source|ABC|XYZ|DEF|.......|N
10records
$source|ABC|XYZ|DEF|.......|N
1000 records
$source|ABC|XYZ|DEF|.......|N
10000 records
$source|ABC|XYZ|DEF|.......|N
max is like 2 lakh records
$source|ABC|XYZ|DEF|.......|N

I want the out put as

$source|ABC|XYZ|DEF|.......|N
10records
1000 records
10000 records
max is like 2 lakh records

Regards,
Deepti

5

The script I provided just did what you desire.

6

The script is working from the command line..,

I tried to automate it for all the .txt files in the directory...I am removing the multiple headers from the file and assigning the same file name again with the below script(i.e i am removing the headers from files a,b,c and assigning the same name again to the new files)..I am getting an error like error in line 1...can you pls tell me where I am doing it wrong

while read FILENAME
do
awk 'NR==1 || $1 !~ /^\$/' < "$FILENAME" >  > "${FILENAME}.txt

"

7

If you got an error message, you should have post it here when asking questions, that would help people helping you.

I tested the following code, which did what you desire:

#!/bin/bash

while read FILENAME; do
    awk 'NR==1 || $1 !~ /^\$/'  $FILENAME >> out.txt
done < filenames.txt
8 
for file_name in `ls f*.txt`
do
awk 'NR==1 || $1 !~ /^\$/' $file_name > "${file_name}".output
done