Invalid Characters in the file.

Shell Programming and Scripting
1

I am working on AIX. We ftp files to a database. The flat files are having thousands of records and each record is having some 50 to 60 characters(there are fields having certain character length). In addition to some valid ascii characters some invalid characters like �, �, �, � or pipes creep in which datawarehouse rejects to load in.
Example: AcuM-^?a 051706 ;
above is a field in the record which is having special characters like -,^ and ? , which should not have been there.

The record separator is a new line and there is no field seperator.

How can I remove these invalid or special characters to creep in the records?
Please help me to find the logic in the shell sripting..

2

After removing the invalid characters I want to add the space at the same place.
Means, I have to replace the invalid character with a space.

Thanks in advance...

3

Hi Kanu,

Please try the below.
Note: The change will be made to the input file directly.

perl -pi -e 's/-/ /g'  <file >
perl -pi -e 's/^/ /g'  <file >

Once you have the list of the strange characters, you can proceed like this.

==========
Thanks,
Karthikeyan.

4

Hi Karthik,

Thanks for the reply.
I do not have any particular list of invalid characters, the characters come up randomly. For example : ~, �, �, �, � etc. Also I am not aware with perl, so clueless.
Can you plz help me in some other ways...!

-Kanu

5

I guess your query was answered in your earlier post on the same topic Invalid Characters in the file.

Please avoid duplicating the post

Thanks
Nagarajan G

6

I will close this duplicate thread. Please read our rules. Double posting is not allowed.