# Extended ASCII Characters keep on getting reintroduced to text files

**URL:** <https://community.unix.com/t/extended-ascii-characters-keep-on-getting-reintroduced-to-text-files/361333>\
**Category:** Shell Programming and Scripting\
**Created:** [July 9, 2016, 12:57pm UTC](https://community.unix.com/t/extended-ascii-characters-keep-on-getting-reintroduced-to-text-files/361333 "2016-07-09T12:57:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![lewk](https://community.unix.com/letter_avatar/lewk/32/5_5575768a8748004e209b776fc1b2916d.png) [@lewk](https://community.unix.com/u/lewk)\
**Post date:** [July 9, 2016, 12:57pm UTC](https://community.unix.com/t/extended-ascii-characters-keep-on-getting-reintroduced-to-text-files/361333/1 "2016-07-09T12:57:57Z")

</div>

I am working with a log file that I am trying to clean up by removing non-English ASCII characters. I am using Bash via Cygwin on Windows.

Before I start I set:

```nohighlight
export LC_ALL=C 

```

I clean it up by removing all non-English ASCII characters with the following command;

```nohighlight
grep -v $'[^\t\r -~]' filename_01.csv > filename_02.csv

```

I then check whether there is any non-English ASCII characters left with the following command and it returns nothing, indicating that there is no non-English ASCII characters left.

```nohighlight
perl -ane '{ if(m/[[:^ascii:]]/) { print } }' filename_02.csv

```

I then deleted the first line with the following command;

```nohighlight
tail -n +2 filename_02.csv > filename_03.csv

```

When I check filename\_03.csv again for non-English characters it returns quite a few lines with non-English ASCII characters :mad: Why is this happening, what am I doing wrong? It somehow got reintroduced when I ran the tail command, how is this possible?

```nohighlight
perl -ane '{ if(m/[[:^ascii:]]/) { print } }' filename_03.csv

```

Example of the characters that got introduced back into my text file after I ran the tail command that I initially cleaned.

```nohighlight
Gateway1,Gateway2 4dU4'E"morel

```

---

<div class="post-metadata">

**Author:** ![RudiC](https://community.unix.com/letter_avatar/rudic/32/5_5575768a8748004e209b776fc1b2916d.png) [@RudiC](https://community.unix.com/u/RudiC)\
**Post date:** [July 9, 2016, 3:55pm UTC](https://community.unix.com/t/extended-ascii-characters-keep-on-getting-reintroduced-to-text-files/361333/2 "2016-07-09T15:55:30Z")

</div>

I think you'll get better response if you post representative samples of your log file indicating which characters you want to remove, or a sample of the result file.

---

<div class="post-metadata">

**Author:** ![Scrutinizer](https://community.unix.com/user_avatar/community.unix.com/scrutinizer/32/1216_2.png) [@Scrutinizer](https://community.unix.com/u/Scrutinizer)\
**Post date:** [July 10, 2016, 9:18am UTC](https://community.unix.com/t/extended-ascii-characters-keep-on-getting-reintroduced-to-text-files/361333/3 "2016-07-10T09:18:15Z")

</div>

One thing I noticed is that this:

```nohighlight
[[:^ascii:]]

```

should be:

```nohighlight
[^[:ascii:]]

```

Also, this :

```nohighlight
grep -v $'[^\t\r -~]' filename_01.csv > filename_02.csv

```

does not just remove non-ascii characters, it discards entire lines that contains one of those characters that are not `[\t\r -~]`

---

<div class="post-metadata">

**Author:** ![wisecracker](https://community.unix.com/user_avatar/community.unix.com/wisecracker/32/2800_2.png) [@wisecracker](https://community.unix.com/u/wisecracker)\
**Post date:** [July 10, 2016, 1:44pm UTC](https://community.unix.com/t/extended-ascii-characters-keep-on-getting-reintroduced-to-text-files/361333/4 "2016-07-10T13:44:41Z")

</div>

Using the very limited info, done longhand.  
Each line in the file contains a single ` ' ` and ` " ` .  
Not sure if this a just a very small part of the string but here goes.

```nohighlight
#�/bin/bash
# nonascii.sh
# Macbbok Pro, circa August 2012, OSX 10.7.5, deafult bash terminal.
> /tmp/nonascii.dat
> /tmp/newascii.txt
# Create 3 lines of data as per the very limited info.
printf "%s\n" "Gateway1,Gateway2 4dU4"\'"E\"morel" >> /tmp/nonascii.dat
printf "%s\n" "Gateway1,Gateway2 4dU4"\'"E\"morel" >> /tmp/nonascii.dat
printf "%s\n" "Gateway1,Gateway2 4dU4"\'"E\"morel" >> /tmp/nonascii.dat
cat /tmp/nonascii.dat
# Now remove non-ascii characters, Backspace character, (127 decimal), also removed here.
length=$( wc -c < /tmp/nonascii.dat )
deci_string=( $( od -tu1 -An < /tmp/nonascii.dat ) )
for n in $( seq 0 1 $((length-1)) )
do
	if ["${deci_string[$n]}" -le "126" ]
	then
		printf '\x'$( printf "%x" "${deci_string[$n]}" )
	fi
done > /tmp/newascii.txt
# Prove extended characters have gone.
cat /tmp/newascii.txt

```

Results:-

```nohighlight
Last login: Sun Jul 10 18:34:52 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> ./nonascii.sh
Gateway1,Gateway2 4dU4'E"morel
Gateway1,Gateway2 4dU4'E"morel
Gateway1,Gateway2 4dU4'E"morel
Gateway1,Gateway2 4dU4'E"morel
Gateway1,Gateway2 4dU4'E"morel
Gateway1,Gateway2 4dU4'E"morel
AMIGA:barrywalker~/Desktop/Code/Shell> _

```

---

<div class="post-metadata">

**Author:** ![RudiC](https://community.unix.com/letter_avatar/rudic/32/5_5575768a8748004e209b776fc1b2916d.png) [@RudiC](https://community.unix.com/u/RudiC)\
**Post date:** [July 11, 2016, 3:25am UTC](https://community.unix.com/t/extended-ascii-characters-keep-on-getting-reintroduced-to-text-files/361333/5 "2016-07-11T03:25:02Z")

</div>

Shooting in the dark: How about

```nohighlight
tr -dc '[:alnum:][:punct:][:cntrl:][:space:]' <file

```

or

```nohighlight
tr -dc '[:print:][:cntrl:]' <file

```

EDIT:  
or even

```nohighlight
tr -dc '\000-\177' <file

```