BASH script to read external file to perform text replacements?

Shell Programming and Scripting
1

Hi all,

I have a moderate size (300 lines) BASH Shell script that performs various tasks on different source reports (CSV files). One of the tasks that it performs, is to use SED to replace 'non-conforming' titles with conformant ones. For example "How to format a RAW Report" needs to become "how_to_format_a_raw_report".

Right now, I include the SED commands within the BASH script. However, new titles are added almost daily, and we don't want any Joe (no offense to all the Joes out there..) going into the BASH script to add new SED commands.

I do have a delimited file that contains just two columns; SourceTitle:::OutputTitle (with ::: as the delimiter).

What would be the best way to incorporate a command (say SED, or AWK, or any other common-use command that doesn't need to be installed separately) to read each line of the delimited text file to get its information?

This is what the delimited text file looks like (titles.text):

 SourceTitle:::OutputTitle
This is a non-conforming title:::this_title_now_conforms
This_title_no_so_bad:::this_title_is_not_so_bad
How to format reports:::how_to_format_reports
Formatting source reports:::formatting_source_reports

This is what the actual source report the BASH script works upon (source.csv):

 Date,Title,Views,Source
21-05-16,ProdA,This is a non-conforming title,487,YouTube
26-05-16,ProdA,This_title_no_so_bad,38,YouTube
14-06-16,ProdB,How to format reports,83,YouTube
22-06-16,ProdC,Formatting source reports,129,Internal

I can't unfortunately, include the BASH script here, but suffice it to say, if I had an idea of how to include a command that would step out and read the titles.text file and make the changes to source.csv, I'd be set.

:slight_smile:

Rich

P.S. At first glance the titles look like you can simply apply commands such as tr, sed, etc. to perform tasks such as 'replace whitespace with underscore', 'change all to lowercase', etc. but it isn't the case. Each title is evaluated on an individual basis to determine what the output will be. This is how the titles.text is generated.

P.P.S. The environment is Cygwin & Solaris using BASH, and AIX 7.1 using KSH. The Cygwin/Solaris environment is a must; the AIX is gravy.. :wink:

2

How about

awk 'NR==FNR {T[$1] = $NF; next} $3 in T {$3 = T[$3]} 1' FS=":" file1 FS="," OFS="," file2
Date,Title,Views,Source
21-05-16,ProdA,this_title_now_conforms,487,YouTube
26-05-16,ProdA,this_title_is_not_so_bad,38,YouTube
14-06-16,ProdB,how_to_format_reports,83,YouTube
22-06-16,ProdC,formatting_source_reports,129,Internal

I'm inclined to bet that your ~300 lines bash can be boiled down to 20 - 30 lines awk .

1 Like
3

hi RudiC,
Thanks for the response/solution! I'm just trying to incorporate it now, and I'm wondering if you can explain how the awk command knows to replace the first field in the mapping.txt with the second field in my CSV file. I don't quite follow the logic.

:slight_smile:

R

4

It fills translation table T with correct headings ($NF from file1) indexed by non-conforming ones ($1). When done with file1 (titles.txt), it switches the field separator to comma, and works on file2. If a line in $3 contains a non-conforming header that is found in T, it is replaced by the correct one.

1 Like