Ask Dr. International

Column #17

The Doctor has finally had the time to prepare a holiday present for everyone, and decided close the year with a bang. Amazing how quickly time flies when you're having fun!

This installment will address:

On This Page

	Exchanging non–English e–mails
	Windows XP/2000 default system font
	Which Windows charsets have support for the Euro symbol?
	Difference between Spanish traditional sort and modern sort
	How to perform string manipulation on UTF–8 strings?
	MLang and NLS compared
	Globalization of websites
	Multilanguage web forms

Exchanging non–English e–mails

Dear Dr. International,

I bought a new PC with Windows Me English version preinstalled. I'd like to exchange e–mail in Japanese. So, I installed the Global IME 5.02 for Japanese. I could read e-mails in Japanese. However, the e-mail that I wrote in Japanese was garbled at the recipient's.

Why were letters garbled even though I could type Japanese on the PC? Could you help me?

Ayako from Japan

Dr. International replies:

Dear Ayako,

A very good question. The situation happens when you want to send e–mails containing non–Latin text using Outlook or Outlook Express on all versions of Windows. The problem is in the language encoding that your mail client stamps the mail with. The default setting will make all messages marked to match your system's codepage, but you can change the settings manually.

Here are the detailed steps for Outlook 2000 or 2002 (part of Office XP):

There you can set the language for outgoing messages. It may be Japanese (or your desired language's codepage) or UTF–8. Both will work, but UTF–8 may not work with some recipients. Try both out and see which works best for you.

Top of page

Windows XP/2000 default system font

Dear Dr. International,

Are Windows 2000 default system fonts the same for other languages as for the US–English version? If not, where can I find a listing of default fonts by OS language?

Dion from the Carolinas

Dr. International replies:

Hi Dion,

The default Shell UI font varies from one set of languages to another. For all Latin and European languages, this font remains unchanged same as English (Tahoma). For East Asian and Middle–Eastern versions, here is a quick summary:

Note: This is also applicable to Windows XP.

Top of page

Which Windows charsets have support for the Euro symbol?

Dear Dr. International,

With the Euro being launched in January 2002, I am wondering which Windows character sets support the Euro symbol?

Euro Fan

Dr. International replies:

Dear Euro Fan,

Windows character sets/codepages that support Euro are:

Many of you have written to the Doctor concerning your inability to see the € symbol in the subject line or the body of your e–mail messages. I've done some investigation and I've seen some mails with the Euro character displayed properly in the message's body but not in the subject line. Usually it happened when the message containing Euro symbol was sent from one service provider to another – Euro's Hex number 80 was in its place. Strangely enough, when I sent the same message that exhibited the problem to different accounts with the same service provider, they remained intact – € was in the subject and body of the message. Other messages remained fully intact when sent amongst different service providers. The problem lies in:

The Doctor can only say that Outlook 2000 and 2002, Outlook Express 5.5 and 6.0, and Hotmail conform to RFC 2047 and keeps the Euro symbol intact. Outlook and Exchange server apply same character set to the header as to message body text parts. For other clients, please contact your vendor if you have problems.

FYI – Windows XP is Euro–ready. It defaults to Euro as the currency when the user locale (Standards and formats in Windows XP parlance) is set to Austria, Belgium, Finland, France, Germany, Greece, Italy, Luxemburg, Netherlands, Portugal, Republic of Ireland, and Spain. Monaco, which previously used the French currency, will also switch to the euro. Users in those 12 countries will see currency amounts appear in Euros rather than the national currency of their respective countries. You can switch to the national currency, or to any other currency, by customizing your currency settings in Regional and Language Options in the Control Panel.

To learn more about Euro support in Microsoft products, see Using Euro as your Currency on TechNet.

Top of page

Difference between Spanish traditional sort and modern sort

Dear Dr. International,

Can you kindly tell me what the difference is between Spanish_Traditional_Sort and Spanish_Modern_Sort? The break–down of the Spanish is quite extensive for the LDICs...however, I do not see a code for Spain or the US market....can you provide some insight for me?

I look forward to hearing back from you at your earliest possible convenience.

Curious Translator

Dr. International replies:

Dear Translator,

As you know there are two different flavors of Spanish sorting: modern and traditional. The traditional has become less preferred over time, and is being replaced by the modern sort. Within the traditional sort, the following rules apply:

For modern (international) sort, the only rule that applies is that N tilde is a unique letter between N and O.

Top of page

How to perform string manipulation on UTF–8 strings?

Dear Dr. International,

I am writing an application that handles UTF–8, and need to find out the length of UTF–8 strings being entered. I've used StrLen, and MbsLen, but they don't give me the right results. Please help.

String Counter

Dr. International replies:

Dear String Counter,

strlen and mbslen are both doing the same job: returning the number of characters in the string. The former deals with single–byte characters and the latter with MBCS (Multi–byte Character Sets). The generic function name is _tcslen. From an OS point of view we do not offer real string manipulation support for UTF–8 (except UTF16 <–>UTF8 conversion).

Let me explain it more in detail. Let's say that my original Unicode string is "سليم". This Arabic name is composed of 4 Unicode code points.

wcslen on this string will return 4 (in fact 4 wchars).

Now, if I convert this to ANSI (with Arabic code page), strlen on that ANSI string will still return 4 (this time 4 chars).

In your case, you are converting the original UTF–16 string to UTF–8. For that range of Unicode characters (U+0x0080 to U+0x07ff) each Unicode code point is represented by 2 bytes in UTF–8 encoding. So my conversion will in fact give 8 high ANSI chars that can not really be displayed in GUI mode (failure not in conversion, but string manipulation). So, running wcslen on that new chain of string will return in fact 8! Not the number of characters as expected, but the number of bytes!! The same goes for other scripts and characters (ASCII, Hebrew, CJK...).

In conclusion, UTF–8 string manipulation should be avoided. Convert your string to UTF–16 and do all analysis in this encoding. Once you're done, convert back to UTF–8 if needed.

Top of page

MLang and NLS compared

Dear Dr. International,

We are in a process of rewriting the code that handles internationalization. So far we work with NLS files and third party conversion tables (for codepages which are not supported by NLS) and we consider shifting to Mlang. Can you provide us with the pros and cons of using MLang versus NLS? Also, how do I perform CharNext on UTF–8 and MLang

Ori in Israel

Dr. International replies:

Dear Ori,

MLang is a COM component that provides a variety of services for detecting the character encoding used by web pages and emails, converting text from one encoding to another as part of an import or export operation, and the display of characters that are not included within the font specified for parts of a web page.

There is some overlap in features between MLang and NLS, but there are more differences than overlap.

Except for code page conversion, the decision to use NLS or MLang should usually be obvious – they provide different functionality and many programs use both.

As for you questions about CharNext, CharNext doesn't support UTF–8. It's easy to implement yourself though––look at the top 2 bits of each byte. If it's 0, then the character is one byte. If it's 1, then the character is 2 bytes. See table below for conversion. Finally, MLang doesn't do CharNext. This is one of the many areas where there is no overlap.

Top of page

Globalization of websites

Dear Dr. International,

I've read several of your articles on how to achieve globalization for an application, but have not been able to find any on the topic globalized web sites/platforms.

We are currently in the design stages of a project that calls for a single code base to support a web site that will be viewed by international users in several different countries (we're starting with UK, US, Israel and Germany).

We are running Windows 2000, IIS 5.0. All of our presentation logic is contained within ASP (VBScript) files. Our business logic is encapsulated in VB and VC++ DLLs. At the moment, we maintain separate code trees on the ASP layer to achieve multilingual support.

I am currently investigating ways in which the language/locale specific resources of each web site can be bundled together and "swapped" in/out on the presentation layer.

Can you point me to articles that specifically address web site globalization on all levels – database, business objects, and presentation.

Peter in UK

Dr. International replies:

Dear Peter,

You're right, we do not have a central reference point for web globalization and are working on it in near future. But for the time being, to answer to your question: Most of the globalization and localizability considerations are common between Win32 and Web development. However, within the web context, you can rely on your HTML/ASP rendering mechanism (example of Trident engine for Internet Explorer) do to the job for you. Here are globalization requirements that you should be aware of in a web context:

<style>
.myStyle {font-size: 10pt; font-family: Arial;}
</style>
<span class = myStyle> Hello </span>

In a Web context, mirroring and Right_To_Left (RTL) reading order go hand to hand and can be set by adding a direction attribute "DIR". The DIR attribute is added during the Arabic and Hebrew localization process and can be set:

Here is a small sample that demonstrates how to toggle mirroring on/off at run–time:

function mirror()
   {
   if (document.dir == "rtl")
      {
      document.dir = "ltr";
      }
   else
      {
      document.dir = "rtl";
      }
   }

Tips to accommodate BiDi localization:

As mentioned above, the DIR attribute leaves the orientation of stationary elements unchanged. However, for directional images (such as arrows), it might be required to change the direction of the image during the BiDi localization. A new filter style for images (IMG) allows developers to implement a flippable image: "<IMG style=filter:flipH SRC=arrow.jpg>". Tables and cells are left aligned by default. Avoid using an obsolete left alignment ("align = left") that would overwrite the DIR attribute and would cause the BiDi text to be left aligned as well. Upon application of the DIR attribute all controls are being rearranged in the web page with a right alignment. Avoid using absolute positioning for controls. Tables and cells are automatically mirrored, so take advantage of this feature and place your controls within cells for robust reversibility.

For more information about this subject, please search MSDN for scripting, and resource isolation. Also see the whitepaper outlining the international features of SQL Server 2000.

Top of page

Multilanguage web forms

Dear Dr. International,

I am currently facing the following problem while developing an international e–business site:

Inside a WEB form,

For each field filled with a specific language characters, the user will specify the language used.

When this form is submitted to the web server, I need to get all the values set by the user and stored them inside a SQL 7 database (nvarchar() columns).

I am using the following configuration:

Let's say my HTML form is like this one:

<html><head><title>Multi–language form</title></head>
<body> 
<form action=mypage.asp method=post> 
<input type=text name=field1>
<select name=lang_field1> 
<option value="">default</option> 
<option value="en">english</option> 
<option value="fr">french</option> 
<option value="gr">greek</option> 
<option value="ru">russian</option> 
</select> 
<br> 
<input type=text name=field2>
<select name=lang_field2> 
<option value="">default</option> 
<option value="en">english</option> 
<option value="fr">french</option> 
<option value="gr">greek</option> 
<option value="ru">russian</option>
</select> 
<br>
<input type=submit name=subf value="Submit these fields...">
</form>
</body>
</html> 

Now let's say user has filled the form with the following values:

Inside mypage.asp, when I call Request.Form(field_name), the current value of Session.Codepage is used and the translation to UNICODE seems to be applied on ALL the post data which is not convenient for me, as I need to use the codepage 1251 to get value of field1 and codepage 1253 to get value of field2.

Any help to solve this problem will be appreciate.

Webmaster from France

Dr. International replies:

Dear Webmaster,

The easiest way for you would be just to use Unicode (UTF–8) as your encoding. You may also set Session.Codepage based on the lang_fieldX settings to process each of the fields, but this is not a scenario Charset was supposed to be used in, it can be distracted by inclusion of the server includes, there are some other problems that can affect your data processing depending on the site architecture.

If your multi–lingual ASP uses string literals, then you need to set @CODEPAGE directive as well; without Unicode the architecture will be too complicated to make it workable. You'll need to have multiple language – specific ASP pages. UTF–8 is the definitely a way to go!

Note: If you are doing similar development work on IIS 5.0, please see Knowledge Base articles Q294831 and Q294833.

To set the compile–time encoding of an ASP page to UTF–8, start it with:

<%@ LANGUAGE="VBscript" CODEPAGE=65001 %>

('language' may be different, of course)

Then in the ASP code you should call

<% Session.CodePage = 65001 %>

before you process any data.

Finally, if you plan to display multilingual data, put the following in the HTML header:

<HEAD>
<meta http–equiv="Content–Type" content="text/html; charset=utf–8">
...

Top of page

Happy Holidays! See you all next year.

Dr. International
Windows Division

Top of page