Hi there,
I'm a newbie in unix and am fishing for options related to how raw input data files are handled. The scenario, as I'm sure y'all must be very familiar with, is this : we receive upwards of 50 data files in ASCII format from various source systems - now each file has its own structure (columns, datatypes etc) as well as certain "impurities" e.g. leading/trailing whitespaces, junk characters (produced during conversion from mainframe data to ASCII) etc...there is a need to 'sanitize' these files i.e. strip them of whitespaces, junk characters etc - how do we do this.....
Ideally, we would like to have a common shell script that parses each input file and produces a clean version (is this possible? will I need to have multiple shell scripts , one for each file?)
Can you please provide feedback based on your experience...
Thanks