Search and replace from file in awk using a 16 bit text file

Shell Programming and Scripting
1

Hello,
Some time ago a helpful awk file was provided on the forum which I give below:

NR==FNR{A[NR]=$0;next}{for(j in A){split(A[j],P,"=");for(i=1;i<=NF;i++){if($i==P[1]){$i=P[2]}}}}1

While it works beautifully on English and Latin characters i.e. within the ASCII range of 127, the moment a character beyond 127 is provided for search and replace, the script fails
How do we modify the script to make it work for characters beyond

A-z

A short sample is provided below for the mapper

�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=
�=

and a sample file for conversion is also provided below:

����裪$#
���������$#
��������$#
���$#
������.�$#
�����������裪.�$#
������.��$#
����������.�$#
������.��$#
������.�$#
����.���$#
�����裪.��$#
�������$#A
��������몡�.���$#
�裪$#
������$#C
����衪$#
������.�$#
�����裪$#
������.�$#
��������衪$#
���������$#K
���窥��$#
������$#P
������������$#M
����衪$#
�����$#L
�窵����$#S
������.�$#

Many thanks in advance for the help

2

I can't see immediately why the script fragment should not work - it works on strings, not on chars unless you have omitted the FS="" assignment.
Pls post/attach input & desired output files, the entire script, and possibly errors that occurred.

3

Hello,
The mapper is the file in which the conversion rules are given. The character on the left hand side is the input and the one on the right hand side is the output.
The file on which the search and replace operation is to be carried out is given as the input.
I cannot provide output since the script does not work. Basically each 8 bit character should be mapped to a corresponding Unicode character and the output should be Unicode.
I have tried again and I get the same results. It spews out the input file as such.
Many thanks for replying.

4

These input files do not appear to be in UTF-8 format. Convert them to UTF-8 first.

5

Dear Scrutinizer,
The mapper file has the following structure:

Left hand before delimiter: 8 bit. Non Unicode/ upper ASCII in range 135 to 254
Right hand after delimiter. 16 bit Unicode or if so desired could also be UTF-8

The Input file is all in 8 bit Non Unicode i.e. Upper ASCII Range
The aim of the script would be to convert a non-Unicode 8 bit file into Unicode using the search and replace mappings stored in the mapper file.
I hope the explanation is clear.
Many thanks for your query and interest.

6

hexdump of above mapper file:

c2 a6 3d e0 af 81  
c2 a7 3d e0 ae 99
c2 a8 3d e0 ae bf

It's definitely not 8 bit on the left hand side (3d = "=" delimiter). Pls provide as attachments (advanced edit) input, mapper, and desired output files.

7

Many thanks I will try and provide the sample as an attachment and so also the mapper file.