(Unofficial) National Language Support (NLS)

The current version is 2005-09-22e-1.1.0. You can find

list of the previous versions,
working examples, and
the latest update (if any)

of this technique in my web site, though you cannot modify pages in my site. Please discuss here if you have any comment.

The National Language Support (NLS) feature introduced in this document is not a built-in functionality nor an official plugin of DokuWiki, but an unofficial patch applied to the 2005-09-22e version. A future version of DokuWiki may officially ship another NLS feature which is irrelevant to one described in this document. Please look for plugins or consult Browser Language Detection for more information about NLS in DokuWiki.

Suitable Situation

The technique introduced in this document is suitable for situations such that:

You cannot add or change the domain name and host name of your web site arbitrarilly. This is, for example, you cannot use en.wikipedia.org or ko.wikipedia.org according to the translating language as in the WikiPedia.
You want to operate only one installation of DokuWiki because of convenience of the user management and so on. In other word, you don't want to operate two or more DokuWiki installations according to the languages of UI's or translations.
Each document in your site will be translated into various languages, where the set of translation languages need not be the same for every document. For example, a document foo will be translated into three languages A, B, and C, whereas another document bar will be translated into four languages B, C, D, and E. Furthermore, yet another document baz will be only in one language, C.

Features (as visitors)

The technique in this page works as follows:

Readers visiting your web site can freely choose any existing translations for the documents they read. If they don't specify any language explicitly, they will be redirected to appropriate translations according to the language settings of their web browsers.
User Interface (UI) language is decided only by the language setting of the reader's web browser, independently from the translation language chosen. (If no language in the browser's setting is prepared by the DokuWiki installation, the default language configured by the site administrator will be used.)
The documents translated in different languages have different URL's, and readers can move from one translation to another not by submitting forms but via simple links. Therefore, every existing translations will be recorded to search engines properly.¹⁾

Prerequisites (as administrators and users)

In order to use the technique introduced in this document, the following conditions are required:²⁾

DokuWiki version 2005-09-22e is needed. (Other versions may work, though not tested.)
For ALL documents in the wiki site to which this technique is about to be applied, the 'translation flags' which denote what language they are translated in should be appended at the ends of their ID's. You will not be able to read any document without a translation flag after this techique is applied, and should append flags to all the documents in your site before applying this technique.

The 'translation flag' is a kind of language code that is appended at the end of page ID, which denotes in what language it is translated. When pageID is the valid ID format (including namespaces) permitted by DokuWiki, the new page ID with a translation flag should be

pageID\.[a-z][a-z](-[a-z][a-z])?

in regular expression. For example, foo:bar.ko is a Korean translation whereas foo:bar.en is an English translation for the same content.

Because the page ID foo:bar contains no translation flag, readers will be redirected to an appropriate translation pages according to their browser setting if they query this kind of page. Though they will be redirected basically to the most appropriate one among existing translations as far as possible, they will be redirected to a non-existing page if no existing translation matches to their browser's setting. In other words, readers will not be able to read foo:bar any more, therefore the site administrator should append translation flags to all the pages in his/her site before using this technique.

A method I used and recommend for appending translation flags to pre-existings page is:

Check in what language a pre-existing page foo:bar is written. → Let's assume that this page is written in zz language.
Create a new page foo:bar.zz copying the content of foo:bar verbatim.
Modify links ID's of all the internal links in the copied content appropiately.
Delete foo:bar.

This procedure, of course, is very tedious, but I don't know better way to change page ID's in DokuWiki.

Algorithm (as developers)

The algorithm is very simple:

A reader query a page.
If the queried page has no translation flag (e.g. foo:bar) and the reader is not doing indexing
1. Extract languages from the browser setting and sort them by their priority.
2. If one of these language (e.g. ko) matches an existing translation page, redirect the reader to that page (e.g. foo:bar.ko).
3. If none matches, redirect him/her to the default translation page configured by the administrator (e.g. foo:bar.{$conf['lang']}) regardless of whether such page exists or not.
If the queried page has a translation flag, show it to the reader.
Aumatically choose an appropriate language for UI by comparing browser settings and DOKU_INC/inc/lang/* in the DokuWiki installation. (If none matches, choose $conf['lang'].)
Overwrite chosen language for UI on $conf['lang'].
Reset locale.
Provide API in order for administrators to be able to add a menu (choosing other translations) to their template.

That's all.

Code and Usage

Please remind again that you should append translation flags to all the page ID's before using this.

Creating inc/NLS.php File

Create inc/NLS.php with the following code (Administrators can modify two arrays $NLS_locarr and $NLS_langname in the first part of this code):

<?php
/**
 * National Language Support (NLS) script for DokuWiki
 *
 * @version    2005-09-22e-1.1.0
 * @license    GPL 2 (http://www.gnu.org/licenses/gpl.html)
 * @author     CHA Reeseo <http://www.reeseo.net/>
 *
 * Usage:
 *     Include this script into 'doku.php',
 *     between the inclusions of 'inc/pageutils.php' and 'inc/html.php'
 *
 * CAUTION:
 *     This script demands a special policy that EVERY page should denote
 *     its own language (.xx or .xx-xx) at the end of its ID.
 *     For example, 'foo:bar.ko' is a page translated into Korean
 *     and 'foo:bar.en' is a page containing the same content in English.
 *
 *     Quering any page having ID without explicit language notation,
 *     you will be redirected to its appropriate 'localized' page.
 *     ('Unlocalized' page will be invisible.)
 *
 *     Please rename all the pre-existing pages before applying this script.
 *
 * To do:
 *     - Upgrade NLS_locale function
 */
 
 
/* ---------------------------------------------------------------
 * Configuration options: You can modify or add something to these */
 
$NLS_locarr['ko'] = 'ko_KR';
$NLS_locarr['en'] = 'en_US';
 
$NLS_langname['ko'] = '한국어';
$NLS_langname['en'] = 'English';
 
/* Configuration options end
 * --------------------------------------------------------------- */
 
 
 
// Setting $ID
$ID = getID();
 
// Redirecting if no language is specified at the end of $ID
if (! NLS_pagelang($ID)) {
        $target = NLS_page4browser($ID);
        if (! array_key_exists("idx", $_GET) && $_GET['do'] != 'recent')  // No redirection when indexing
                header('Location: ' . wl($target));
}
 
// Resetting $conf['lang'] according to the language setting of user's browser
$conf['lang'] = NLS_UI4browser();
// Resetting $lang array
@require_once(DOKU_INC.'inc/lang/'.$conf['lang'].'/lang.php');
// Resetting locale
setlocale(LC_ALL, NLS_locale($conf['lang']));
 
 
/**
 * Getting locale string
 *
 * FIXME: What a poor function this is!
 */
function NLS_locale($ln = NULL) {
        global $conf;
        global $lang;
        global $NLS_locarr;
        if (! $ln) $ln = $conf['lang'];
        $loc = array_key_exists($ln, $NLS_locarr) ? $NLS_locarr[$ln] : $ln;
        $loc .= '.';
        $loc .= array_key_exists('encoding', $lang) ? strtoupper($lang['encoding']) : 'UTF-8';
        return $loc;
}
 
 
/**
 * Printing links to other translations of the given page
 * (API for template files such as DOKU_TPL/main.php
 */
function NLS_transmenu($pid = NULL, $delimiter = ",\n", $withself = FALSE) {
        global $NLS_langname;
        if (! $pid) {
                global $ID;
                $pid = $ID;
        }
        $currplang = NLS_pagelang($pid);
        $tpages = NLS_transpages($pid);
        if ($currplang && ! $withself)
                unset($tpages[$currplang]);
        if ($tpages) {
                $first = TRUE;
                foreach ($tpages as $ln => $tid) {
                        $repr = array_key_exists($ln, $NLS_langname) ? $NLS_langname[$ln] : $ln;
                        if (! $first)
                                echo $delimiter;
                        $first = FALSE;
                        if ($currplang == $ln)
                                echo "<em>$repr</em>";
                        else
                                echo "<a href=\"".wl($tid)."\">$repr</a>";
                }
        } else {
                echo "None.";
        }
}
 
 
 
/**
 * Selecting a redirection target (page) which best match the browser setting
 *
 * Default: page of $conf['lang'] (whether it exists or not)
 * Choice : among existing translations, highest priority for the browser
 */
function NLS_page4browser($pid = NULL) {
        if (! $pid) {
                global $ID;
                $pid = $ID;
        }
        $blang = NLS_browserlang();
        $existing_pages = NLS_transpages($pid);
        $pid_base = NLS_ID_base($pid);
        $tmp_page = $pid_base . '.' . $conf['lang'];    // default page
        foreach ($blang as $lang_str => $priority) {
                $lang_str = str_replace("_", "-", strtolower($lang_str));
                if (array_key_exists($lang_str, $existing_pages))
                        $tmp_page = $pid_base . '.' . $lang_str;
                elseif (array_key_exists(substr($lang_str, 0, 2), $existing_pages))
                        $tmp_page = $pid_base . '.' . substr($lang_str, 0, 2);
        }
        return $tmp_page;
}
 
 
/**
 * Selecting a language for UI
 *
 * Default: $conf['lang']
 * Choice : among existing 'inc/lang/*', highest priority for the browser
 */
function NLS_UI4browser() {
        global $conf;
        $tmp_lang = $conf['lang'];      // This ($conf['lang']) is the default!
        $blang = NLS_browserlang();
        foreach ($blang as $lang_str => $priority) {
                $lang_str = str_replace("_", "-", strtolower($lang_str));
                if (is_dir(DOKU_INC . "inc/lang/" . $lang_str))
                        $tmp_lang = $lang_str;
                elseif (is_dir(DOKU_INC . "inc/lang/" . substr($lang_str, 0, 2)))
                        $tmp_lang = substr($lang_str, 0, 2);
        }
        return $tmp_lang;
}
 
 
/**
 * Getting a sorted (by priority) array of the languages
 * from the language setting of the user's web browser
 */
function NLS_browserlang() {
        $acclang_arr = split(" *, *", trim($_SERVER['HTTP_ACCEPT_LANGUAGE']));
        foreach ($acclang_arr as $acclang) {
                if (ereg("^(.+) *;.+= *(.+)$", $acclang, $acclang_parts))
                        $acclang_sorted[$acclang_parts[1]] = (double)($acclang_parts[2]);
                else
                        $acclang_sorted[$acclang] = 1.0;
        }
        asort($acclang_sorted, SORT_NUMERIC);
        reset($acclang_sorted);
        return $acclang_sorted;
}
 
 
/**
 * Getting an associative array (lang => ID) of
 * all the translated pages for the given page
 */
function NLS_transpages($pid = NULL) {
        global $conf;
        if (! $pid) {
                global $ID;
                $pid = $ID;
        }
        $transpages = array();
        $pid_base_path = $conf['datadir'] . '/' . str_replace(":", "/", NLS_ID_base($pid));
        foreach(glob($pid_base_path . ".*.txt") as $fn) {
                $aid = str_replace("/", ":", substr($fn, strlen($conf['datadir']) + 1, -4));
                if ($alang = NLS_pagelang($aid)) {
                        $transpages[$alang] = $aid;
                }
        }
        return $transpages;
}
 
 
/**
 * Drop language notation (.xx or .xx-xx) from the given ID
 */
function NLS_ID_base($pid = NULL) {
        if (! $pid) {
                global $ID;
                $pid = $ID;
        }
        if (substr($pid, -3, 1) == '.')
                return substr($pid, 0, -3);
        elseif (substr($pid, -6, 1) == '.' && substr($pid, -3, 1) == '-')
                return substr($pid, 0, -6);
        else
                return $pid;
}
 
 
/**
 * Get language (xx, xx-xx, or NULL) from the given ID
 */
function NLS_pagelang($pid = NULL) {
        if (! $pid) {
                global $ID;
                $pid = $ID;
        }
        if (substr($pid, -3, 1) == '.')
                return substr($pid, -2);
        elseif (substr($pid, -6, 1) == '.' && substr($pid, -3, 1) == '-')
                return substr($pid, -5);
        else
                return NULL;
}
?>

Modifying doku.php File

Now, let's add an inclusion of inc/NLS.php at the beginning part of doku.php. The location of this inclusion is very important! Though NLS script should be loaded as early as possible, it should be placed after inc/pageutils.php because NLS script uses it. Please insert the line of inclusion just after including inc/pageutils.php (This code shows only the first few lines):

<?php
/**
 * DokuWiki mainscript
 *
 * @license    GPL 2 (http://www.gnu.org/licenses/gpl.html)
 * @author     Andreas Gohr <andi@splitbrain.org>
 */
 
//  xdebug_start_profiling();
 
  if(!defined('DOKU_INC')) define('DOKU_INC',realpath(dirname(__FILE__)).'/');
  require_once(DOKU_INC.'inc/init.php');
  require_once(DOKU_INC.'inc/common.php');
  require_once(DOKU_INC.'inc/pageutils.php');
  require_once(DOKU_INC.'inc/NLS.php');       // insert this line!
  require_once(DOKU_INC.'inc/html.php');
  require_once(DOKU_INC.'inc/auth.php');
  require_once(DOKU_INC.'inc/actions.php');
 
  //import variables
...

Adding Menu (for Selecting Translations) to the Template

For the last, let's add a menu for selecting translations to the template in order for the readers to choose another translations of current page. Just insert the following code into the main.php or so in the template you are using:

Other translations of this page: <?php NLS_transmenu(); ?>

NLS_transmenu() function is an API which generate a menu to the template and defines following parameters:

NLS_transmenu($pid, $delimiter, $withself)

Each parameter is:

string $pid: The function generate a list of translation of the document which is identified by this variable. The default is DokuWiki's global variable $ID, that is, the current document.
string $delimiter: If there are more than one translations, they will be separated by the delimiter assigned to this variable. The default value is ",\n".
boolean $withself: Menu will include the language of the $pid document itself if the value assigned to this variable is True, and will exclude it if the value is False. The default is False.

To do

As written in the code above as a comment, NLS_locale() function sucks! An upgrade is seriously needed and it's the final goal of the version 1.1.x series.

Discussion

I like this concept. In reading through the dokuwiki site I see that namespaces are/can also be used. For reference please check multilingual_content and also local.php. At the bottom of the local.php discussion Stéphane Gully shows how he made http://www.pxxo.net/ with french and english translations and graphic language selectors. The end result seems very similar to your site http://www.reeseo.net/home.en.

I would love to see these two approaches merged…

The automatic creation of mirror documents in the first language in the alternate languages with a to be translated tag - as you suggest in this article.

The link between the documents is automatic (as in your site) and just clicking on a language will show the translated doc in the right place - or show that it has not yet been done…as in both of your approaches.

My main concern is the document naming convention. You suggest to add the language code at the end of your page name foo:bar.ko whereas the multilingual discussion suggests the creation of a language namespace at the front end ko:foo:bar.

What are the pro's and con's of these two approaches? - Tito Vergara 12/01/06

I like the present approach better than the multilang. content which adds the “lang:” in a too prominent place (IMHO) - in particular if a sidebar nav.menu (e.g. roundbox..) displays the namespace components as folders. Also since availability of a translation may rather depend on the given page, and some pages might not need translation.

OTOH, I'd like it to be even more discrete and at the same time “intelligent”. What about the following idea:

get $LANG from REQUEST or SESSION (see below)
if foo:bar is requested, check if foo:bar.LANG exists, then display it (without showing the extension in the URL), else display foo:bar (if exists) (or foo:bar.$DEFAULTLANG)

There are several advantages of the URLs not containing the .LANG : esthetical, portability, readability,…

Setting the $LANG (SESSION or COOKIE or GET) would be done

from GET (i.e. if “&LANG=xx” is added to the URL)
if avail, from the last click on a “lang. flag” or similar explicit choice
from browser setting
site default (questionable if this should set the LANG or leave it empty, which might be not equivalent)

— MFH, 25.6.07

¹⁾

This is only for the content translations. UI language has nothing to do with URL, and web crawlers will take only one language according to their own settings or site default.

²⁾

Some of these may change or disappear in a future version (1.2.x or above) of this technique.

Table of Contents