How to create new empty utf8 file for appending?

Linux and Unix-Like Linux
1

Hey i try to create empty text file with utf-8 encoding
without success
what is the right way to do this ?

tried with

touch test.txt 
iconv -f UTF-8 -t UTF-8 test.txt

or

iconv -f latin1 -t UTF-8 test.txt
2

Try something like this

iconv -f ascii -t utf-8  inputfile -o outputfile
3

I don't think you can identify nor create an empty "UTF-8 file".

4 
touch test.txt

creates an empty file. Until you write data into it, it is an empty ASCII file, an empty EBCDIC file, an empty UTF-8 file, an empty ISO 8859-1 file, an empty ISO 8859-6 file, and an empty file encoded with any other codeset you choose to select.

5

well i did touch test.txt
and when i do file -bi output_gt/test.txt
im getting :

application/x-empty; charset=binary

and then i trying to write data in utf8 encoding to it with
$data >> test.txt
and all in gibberish
when i do on the file file -bi output_gt/test.txt:
im getting :

text/plain; charset=unknown-8bit

what can i do ?

6 
$data >> test.txt

does not give us sufficient information to determine how you generated the contents of test.txt. Please provide exact details.

7

ok more info as much as i can
this is part in the script that write into file :

#set string_to_print = `echo $STRING_CONCAT | awk '{$0=substr($0,1,length($0)-1); print $0}'` 
set string_to_print = `echo $STRING_CONCAT | awk '{gsub(/,$/,""); print}'`
echo $string_to_print>>$new_file_full_path
set STRING_CONCAT=""

now when i keep the second awk ( awk '{gsub(/,$/,""); print}'`)
it prints me gibberish
if i uncomment the first awk ( and comment the second )
the file remains utf-8 but the awk is wrong because in some cases it doesn't remove the last character

i have no idea why the first awk keeps the file utf-8
and the second not .
the text im writing to file is Russian

8

Some versions of awk do not conform to the standards and the length($0) command returns the number of bytes in the line instead of the number of characters in the line. These versions of awk also provide substr() functions that count bytes instead of characters. On these versions of awk , if the last character on a line before the <newline> line terminator is a multi-byte character, the substr() in your 1st awk will transform those multi-byte characters into illegal sequences of bytes that do not form valid UTF-8 characters.

What OS (including version) are you using?

9 
file fileX
fileX: ASCII text

Adding "�������" from my german keyboard makes it read

file fileX
fileX: UTF-8 Unicode text

So, as Don Cragun suspects, it might be the incorrect handling of multibyte chars. Where does the "gibberish" start?