Chapter 2 Internationalization Framework in the Solaris 8 Environment

This section discusses several internationalization features contained in the Solaris 8 environment.

• Support for Codeset independence

• Locale database

• Process code format (wide character expression)

• libw and libintl

• ctype macros

• genmsg utility

This section also contains information useful for developing internationalized applications such as:

• Dynamically linked applications

• Solaris 8 internationalized APIs

Support for Codeset Independence

The Solaris 8 operating environment supports non-EUC encodings such as PC-Kanji in Japan, Big-5 in Taiwan, and GBK in the People's Republic of China.

Because a large part of the computer market demands non-EUC codeset support, Solaris 8 provides a solid framework to enable both EUC and non-EUC codeset support. This support is called Codeset Independence, or CSI.

The goal of CSI is to remove EUC dependencies on specific codesets or encoding methods from Solaris OS libraries and commands. The CSI architecture allows the Solaris operating environment to support any UNIX file system safe encoding. CSI supports a number of new codesets, such as UTF-8, PC-Kanji, and Big-5.

The CSI Approach

Codeset Independence enables application and platform software developers to keep their code independent of encoding, such as UTF-8, and also provides the ability to adopt any new encoding without having to modify the source code. This architecture approach differs from Java internationalization in that Java requires applications to be Unicode-dependent and also requires code conversions throughout the application.

Many existing internationalized applications (for example, Motif) automatically inherit CSI support from the underlying system. These applications work in the new locales without modification. OPEN LOOK applications, however, that are XView/OLIT based, don't work in the new locales because XView is codeset-dependent.

CSI is inherently independent from any codesets. However, the following assumptions on file code encodings (codesets) still apply to Solaris 8:

• File code is a superset of ASCII.

  Unicode (16-bits fixed width) cannot be supported as file code.

• NULL (0x00) is not part of multibyte characters for support of null-terminated multibyte character strings.

• Slash / (0x2f) is not part of multibyte characters for support of the UNIX path names.

• Only stateless file code encodings are supported.

CSI-enabled Commands

Table 2-1 contains CSI-enabled commands in Solaris 8. These commands are marked with CSI capabilities on their man page.

All commands are in the /usr/bin directory, unless otherwise noted.

Solaris 8 CSI-enabled Libraries

Nearly all functions in Solaris 8 libc (/usr/lib/libc.so) are CSI-enabled. However, the following functions in libcare not CSI-enabled because they are EUC-dependent functions:

• csetcol() csetlen() euccol()

• euclen() eucscol() getwidth()

The following macros are not CSI-enabled because they are EUC dependent:

• csetno() wcsetno()

In the Solaris 8 product, libgen (/usr/ccs/lib/libgen.a) are internationalized, but not CSI enabled.

In the Solaris 8 product, libcurses (/usr/ccs/lib/libcurses.a) are internationalized, but not CSI enabled.

Here are the five deliverables:

• The utility (32-bit application):

  /usr/bin/geniconvtbl

• special iconv shared objects:

  /usr/lib/iconv/geniconvtbl.so

  /usr/lib/iconv/sparcv9/geniconvtbl.so

• Sample geniconvtbl(1) input source files and system-provided binary table files :

  /usr/lib/iconv/geniconvtbl/srcs/

  ISO8859-1_to_ISO646.txt

  ISO646_to_ISO8859-1.txt

  ISO8859-1_to_UTF-8.txt

  UTF-8_to_ISO8859-1.txt

  ShiftJIS_to_eucJP.txt

  eucJP_to_ShiftJIS.txt

  /usr/lib/iconv/geniconvtbl/binarytables/

  ISO8859-1%ISO646.bt

  ISO646%ISO8859-1.bt

• Changed iconv_open(3) at libc.so.1s:

  /usr/lib/libc.so.1

  /usr/lib/sparcv9/libc.so.1 (sparcv9 example)

• Man pages:

  /usr/share/man/sman1/geniconvtbl.1

  /usr/share/man/sman4/geniconvtbl.4

The section for geniconvtbl(1) describes how to use the utility and where to place the generated binary table files so that they can be used by the iconv functions and utilities.

See geniconvtbl(4)

Locale Database

The locale database format and structure is private and subject to change in a future release. Therefore, when developing an internationalized application, do not directly access the locale database. Instead, use the Solaris internationalization APIs.

When using Solaris 8, use the locale databases that are included with the Solaris 8 product. Do not use locales from previous Solaris versions.

Process Code Format

The process code format in the Solaris 8 product is private and subject to change in a future release. Therefore, when developing an international application, do not assume the process code format is the same. Instead use the Solaris internationalization APIs.

Multibyte Support Environment (MSE)

A multibyte character is a character that cannot be stored in a single byte, such as Chinese, Japanese, or Korean characters. These characters require two or three bytes of storage. A more precise definition can be found in ISO/IEC 9899:1990 subclause 3.13. The programming model enables these multibyte characters to be read in as logical units and stored internally as wide characters. These wide characters can be processed by the program as logical entities in their own right. Finally, these wide characters can be written out (undergoing appropriate translation) as logical units. This is analogous to the way single-byte characters are read in, manipulated, and written out again. The MSE provides a comparable set of interfaces to perform this processing. The MSE allows programs to be written to handle multibyte characters using the same programming model that is used for single-byte characters.

Dynamically Linked Applications

Solaris 8 users can choose how to link applications with the system libraries, such as libc, by using dynamic linking or static linking. However, any application that requires internationalization features in the system libraries must be dynamically linked. If the application has been statically linked, the operation to set the locale to other than C and POSIX using the setlocale function will fail. Statically linked applications can be operated only in C and POSIX locales.

By default, the linker program tries to link the application dynamically. If the command line options to the linker and the compiler include -Bstatic or -dn specifications, your application might be statically linked. You can check whether an existing application is dynamically linked using the /usr/bin/ldd command.

For example, if you type:

% /usr/bin/ldd /sbin/sh

the command displays the following message:

% ldd: /sbin/sh: file is not a dynamic executable or shared object

The message indicates the /sbin/sh command is not a dynamically linked program. Also, if you type:

% /usr/bin/ldd /usr/bin/ls

the command displays the following message:

% libc.so.1 => 	/usr/lib/libc.so.1
% libdl.so.1 => /usr/lib/libdl.so.1

This message indicates the /usr/bin/ls command has been dynamically linked with two libraries, libc.so.1 and libdl.so.1.

To summarize, if the message from the ldd command to the application does not contain a libc.so.1 entry, it indicates that the application has been statically linked with libc. In that case, you need to change the command line options to the linker so that dynamic linking is used instead, then re-link the application.

libw and libintl

These interfaces have moved to libc and are no longer in libw and libintl.

The shared objects ensure runtime compatibility for existing applications and, together with the archives, provide compilation environment compatibility for building applications. However, it is no longer necessary to build applications against libw or libintl.

For more information on filters see the Linker and Libraries Guide.

Table 2-2 shows the stub entry points in libw and libintl.

wscat

wscspn

ctype Macros

Character classification and character transformation macros are defined in /usr/include/ctype.h. The Solaris 8 environment provides a new set of ctype macros. The new macros support character classification and transformation semantics defined by XPG4. To access the new set of macros, one of the following conditions must be met:

• _XPG4_CHAR_CLASS is defined.

• _XOPEN_SOURCE and _XOPEN_VERSION=4 are defined.

• _XOPEN_SOURCE and _XOPEN_SOURCE_EXTENDED=1 are defined.

This means that all XPG4 and XPG4.2 applications automatically have the new macros. Since _XOPEN_SOURCE, _XOPEN_VERSION, and _XOPEN_SOURCE_EXTENDED bring in extra XPG4 related features in addition to new ctype macros, non-XPG4 or XPG4.2 applications should use __XPG4_CHAR_CLASS__.

There are corresponding ctype functions. The Solaris 8 functions also support XPG4 semantics.

Refer to the ctype(3C) man page for details.

Internationalization APIs in libc

Solaris 8 offers two sets of APIs:

• Multibye (file codes)

• Wide characters (process code)

Applications process in wide-character codes.

When a program takes input from a file, convert your file's multibyte data into wide character process code with the mbtwoc and mbtowcs APIs. To convert the file output data from wide character format into multibyte format, use the wcstombs and wctomb APIs.

Table 2-3 shows a list of internationalization APIs included in Solaris 8.

genmsg Utility

The new genmsg utility can be used with the catgets() family of functions to create internationalized source message catalogs. The utility examines a source program file for calls to functions in catgets and builds a source message catalog from the information it finds. For example:

% cat example.c
	...
	/* NOTE: %s is a file name */
	printf(catgets(catd, 5, 1, "%s cannot be opened."));
	/* NOTE: "Read" is a past participle, not a present
 
			tense verb */
	printf(catgets(catd, 5, 1, "Read"));
	...
% genmsg -c NOTE example.c
The following file(s) have been created.
			new msg file = "example.c.msg"
% cat example.c.msg
$quote "
$set 5
1			"%s cannot be opened"
	/* NOTE: %s is a file name */
2			"Read"
	/* NOTE: "Read" is a past participle, not a present
			tense verb */

In the above example, genmsg is run on the source file example.c, which produces a source message catalog named example.c.msg. The -c option with the argument NOTE causes genmsg to include comments in the catalog. If a comment in the source program contains the string specified, the comment appears in the message catalog after the next string extracted from a call to catgets().

You can use genmsg to number the messages in a message set automatically.

For more information, see the genmsg(1) man page.

The material in this section is used with permission from Creating Worldwide Software: Solaris International Developer's Guide, 2nd edition by Bill Tuthill and David A. Smallberg, published by Sun Microsystems Press/Prentice Hall. (c)1997 Sun Microsystems, Inc.