Multibyte characters to ASCII

Shell Programming and Scripting
1

Hello,

Is there any UNIX utility/command/executable that will convert mutlibyte characters to standard single byte ASCII characters in a given file?

and

Is there any UNIX utility/command/executable that will recognize multibyte characters in a given file name?

The typical multibyte character set that we might encounter are Chinese and or Japanese.

Thanks
Jerardfjay :confused:

2

You want the conversion to be done within a script ?

Why do you want to do that ? Those things are usually done within the application.

If your intention is to just view those ZH*/JA locale characters, then you might as well change the locale information on your machine and then view those files. Instead of the machine, the locale change can be done to your xterm itself.

Very specifically, LC_ALL and LANG variables.

It would go something like this

LC_ALL=ja_JP.euc_jp
LANG=ja_JP.euc_jp

Vino

3

Vino,

Which is precisely what I intend to do If I can do it. Unfortunately there are several apps that feed into my script and changing the back-end system is not feasible.

Even though I may not be able to change the data content represented in multi-byte character to single, If I can perform the change atleast at the file name level I should be ok.

Please advise. Thanks
Jerardfjay

4

Vino,

I also noticed that I have two env vars set for the Language

shell>> env | grep LANG
LANG=ja_JP.euc_jp
NLS_LANG=AMERICAN_AMERICA.US7ASCII

Question: Should I unset the NLS_LANG to for my login to use the LANG variable? Please advise. Thanks.

Jerardfjay

5

POSIX locale and localedef don't support NLS_<anything> - there is an NLSPATH that is part of locale settings. With the exception of LANG, the locale variables all are LC_<something>. The NLS_<anything> variables I've seen have been in support of databases.

You will want to follow Vino's advice and use the LANG environment variable, but leave the NLS_ stuff alone. It doesn't affect locale.

6

Jim,

thanks for the clarification between the two vars. Would you happen to know if these variables are system wide settings or for individual logins? Would I require a special type of terminal emulator like exceed to see the character sets correctly or would a normal windows telnet session be okay. Please advise. Thanks.

Jerardjjay

7

You dont need to set the locale information explicitly.

when you set these variables to values of your choice, you have the luxury of having two locales in your machine. But each locale will be specific to a terminal.

I am not sure about a telnet window displaying those characters. But on kdesktop, I could see them.

Vino

8

I havnt seen NLS_* variables before. So I am not sure about them.

Issue a locale command, and it will show you the current locale settings.

vino

9

Try recode