Removing special chars from file and maintain field separator

UNIX for Advanced & Expert Users
1

Running SunOs 5.6. Solaris.

I've been able to remove all special characters from a fixed length file which appear in the first column but as a result all subsequent columns have shifted to the left by the amount of characters deleted.
It is a space separated file. Line 1 in input file is corrupt. Line 2 is what it should look like. The string 000022000362700 starts at position 49 on both lines. The problem I'm having is that after removing the 3 special chars the field moves to position 46.

Line 1
        GAVISCON LIQUID PEPPERMINT �OT        000022000362700   159588000007979400  50001584182        0006S020000

Line 2
        GAVISCON LIQUID PEPPERMINT OT        000022000362700   159588000007979400  50001584182        0006S020000

The command I'm using is as follows :

cat file.txt | grep '[^ -~]' | sed 's/[^ -~]//g'

This produces following output:

        GAVISCON LIQUID PEPPERMINT OT        000022000362700   159588000007979400  50001584182        0006S020000

By removing the special characters every field to the right of the changed field has moved to the left changing the field start positions.

It appears that the line of text is not showing the correct character spacing but all fields from the first numeric field onwards should have the same starting position.

I've been searching for a while now and cannot find any solution for this issue.

How should I proceed?

2

Please use code tags for data as well!

How about

sed -n 's/[^ - ~]/ /pg' file.txt
1 Like
3 
cat file.txt | grep '[^ -~]' | sed 's/[^ -~]//g' | sed -n 's/[^ -~]/ /pg'

sed: command garbled: s/[^ -~]/ /pg
4

try: s/[^ ~-]/ /pg instead.
Any particular reason why you're using cat ?
And why you're not using code tags?

5

Apologies. Realised about the code tags just now.

grep '[^ -~]' file.txt | sed 's/[^ -~]//g'

GAVISCON LIQUID PEPPERMINT OT 000022000362700 159588000007979400 50001584182 0006S020000

Getting garbled message now.

grep '[^ -~]' file.txt | sed 's/[^ -~]//g' | sed s/[^ ~-]/ /pg

sed: command garbled: s/[^

I was wondering shouldn't we be using awk for column manipulation to move characters from position 46 to 49 ?

---------- Post updated at 03:02 PM ---------- Previous update was at 03:00 PM ----------

grep '[^ -~]' file.txt

GAVISCON LIQUID PEPPERMINT �OT 000022000362700 159588000007979400 50001584182 0006S020000

6

Replace this

cat file.txt | grep '[^ -~]' | sed 's/[^ -~]//g'

by

sed -n 's/[^ -~]/ /pg' file.txt

Make sure the leading single quote is there. Drop the -n option and the p flag for a test, if the error msgs remains. Or swap the p and the g flags.

1 Like
7

Thank you for all your help. Finally it worked !!!

Solution

sed 's/[^ -~]/ /g' file.txt