Libidn2 0.3

Libidn2

This manual is for Libidn2 (version 0.3, 20 April 2011), an implementation of IDNA2008 internationalized domain names.

Copyright © 2011 Simon Josefsson

• Introduction: What is Libidn2?
• Library Functions: Library functions.
• Examples: Demonstrate how to use the library.
• Invoking idn2: Command line interface to the library.
• Interface Index
• Concept Index

1 Introduction

Libidn2 is a free software implementation of IDNA2008.

2 Library Functions

Below are the interfaces of the Libidn2 library documented.

2.1 Header file idn2.h

To use the functions documented in this chapter, you need to include the file idn2.h like this:

     #include <idn2.h>

2.2 Core Functions

When you have the data encoded in UTF-8 form the direct interfaces to the library are as follows.

idn2_lookup_u8

idn2_register_u8

2.3 Locale Functions

As a convenience, the following functions are provided that will convert the input from the locale encoding format to UTF-8 and normalize the string using NFC, and then apply the core functions described earlier.

idn2_lookup_ul

idn2_register_ul

2.4 Control Flags

The flags parameter can take on the following values, or a bit-wise inclusive or of any subset of the parameters:

2.5 Error Handling

idn2_strerror

idn2_strerror_name

2.6 Return Codes

The functions normally return 0 on sucess or a negative error code.

2.7 Memory Handling

idn2_free

2.8 Version Check

It is often desirable to check that the version of Libidn2 used is indeed one which fits all requirements. Even with binary compatibility new features may have been introduced but due to problem with the dynamic linker an old version is actually used. So you may want to check that the version is okay right after program startup.

idn2_check_version

The normal way to use the function is to put something similar to the following first in your main:

       if (!idn2_check_version (IDN2_VERSION))
         {
           printf ("idn2_check_version() failed:\n"
                   "Header file incompatible with shared library.\n");
           exit(EXIT_FAILURE);
         }

3 Examples

This chapter contains example code which illustrate how Libidn2 is used when you write your own application.

• Lookup: Example IDNA2008 Lookup domain name operation.
• Register: Example IDNA2008 Register label operation.

3.1 Lookup

This example demonstrates how a domain name is processed before it is lookup in the DNS.

#include <stdio.h> /* printf, fflush, fgets, stdin, perror, fprintf */
#include <string.h> /* strlen */
#include <locale.h> /* setlocale */
#include <stdlib.h> /* free */
#include <idn2.h> /* idn2_lookup_ul, IDN2_OK, idn2_strerror, idn2_strerror_name */

int
main (int argc, char *argv[])
{
  int rc;
  char src[BUFSIZ];
  char *lookupname;

  setlocale (LC_ALL, "");

  printf ("Enter (possibly non-ASCII) domain name to lookup: ");
  fflush (stdout);
  if (!fgets (src, sizeof (src), stdin))
    {
      perror ("fgets");
      return 1;
    }
  src[strlen (src) - 1] = '\0';

  rc = idn2_lookup_ul (src, &lookupname, 0);
  if (rc != IDN2_OK)
    {
      fprintf (stderr, "error: %s (%s, %d)\n",
	       idn2_strerror (rc), idn2_strerror_name (rc), rc);
      return 1;
    }

  printf ("IDNA2008 domain name to lookup in DNS: %s\n", lookupname);

  free (lookupname);

  return 0;
}

3.2 Register

This example demonstrates how a domain label is processed before it is registered in the DNS.

#include <stdio.h> /* printf, fflush, fgets, stdin, perror, fprintf */
#include <string.h> /* strlen */
#include <locale.h> /* setlocale */
#include <stdlib.h> /* free */
#include <idn2.h> /* idn2_register_ul, IDN2_OK, idn2_strerror, idn2_strerror_name */

int
main (int argc, char *argv[])
{
  int rc;
  char src[BUFSIZ];
  char *insertname;

  setlocale (LC_ALL, "");

  printf ("Enter (possibly non-ASCII) label to register: ");
  fflush (stdout);
  if (!fgets (src, sizeof (src), stdin))
    {
      perror ("fgets");
      return 1;
    }
  src[strlen (src) - 1] = '\0';

  rc = idn2_register_ul (src, NULL, &insertname, 0);
  if (rc != IDN2_OK)
    {
      fprintf (stderr, "error: %s (%s, %d)\n",
	       idn2_strerror (rc), idn2_strerror_name (rc), rc);
      return 1;
    }

  printf ("IDNA2008 label to register in DNS: %s\n", insertname);

  free (insertname);

  return 0;
}

4 Invoking idn2

idn2 translates internationalized domain names to the IDNA2008 encoded format, either for lookup or registration.

If strings are specified on the command line, they are used as input and the computed output is printed to standard output stdout. If no strings are specified on the command line, the program read data, line by line, from the standard input stdin, and print the computed output to standard output. What processing is performed (e.g., lookup or register) is indicated by options. If any errors are encountered, the execution of the applications is aborted.

All strings are expected to be encoded in the preferred charset used by your locale. Use --debug to find out what this charset is. On POSIX systems you may use the LANG environment variable to specify a different locale.

To process a string that starts with -, for example -foo, use -- to signal the end of parameters, as in idn2 -r -- -foo.

4.1 Options

idn2 recognizes these commands:

  -h, --help               Print help and exit

  -V, --version            Print version and exit

  -l, --lookup             Lookup domain name (default)

  -r, --register           Register label

      --debug              Print debugging information

      --quiet              Silent operation

4.2 Environment Variables

On POSIX systems the LANG environment variable can be used to override the system locale for the command being invoked. The system locale may influence what character set is used to decode data (i.e., strings on the command line or data read from the standard input stream), and to encode data to the standard output. If your system is set up correctly, however, the application will use the correct locale and character set automatically. Example usage:

     $ LANG=en_US.UTF-8 idn2
     ...

4.3 Examples

Standard usage, reading input from standard input and disabling license and usage instructions:

     jas@latte:~$ idn2 --quiet
     räksmörgås.se
     xn--rksmrgs-5wao1o.se
     ...

Reading input from the command line:

     jas@latte:~$ idn2 räksmörgås.se blåbærgrød.no
     xn--rksmrgs-5wao1o.se
     xn--blbrgrd-fxak7p.no
     jas@latte:~$

Testing the IDNA2008 Register function:

     jas@latte:~$ idn2 --register fußball
     xn--fuball-cta
     jas@latte:~$

4.4 Troubleshooting

Getting character data encoded right, and making sure Libidn2 use the same encoding, can be difficult. The reason for this is that most systems may encode character data in more than one character encoding, i.e., using UTF-8 together with ISO-8859-1 or ISO-2022-JP. This problem is likely to continue to exist until only one character encoding come out as the evolutionary winner, or (more likely, at least to some extents) forever.

The first step to troubleshooting character encoding problems with Libidn2 is to use the ‘--debug’ parameter to find out which character set encoding ‘idn2’ believe your locale uses.

     jas@latte:~$ idn2 --debug --quiet ""
     Charset: UTF-8
     
     jas@latte:~$

If it prints ANSI_X3.4-1968 (i.e., US-ASCII), this indicate you have not configured your locale properly. To configure the locale, you can, for example, use ‘LANG=sv_SE.UTF-8; export LANG’ at a /bin/sh prompt, to set up your locale for a Swedish environment using UTF-8 as the encoding.

Sometimes ‘idn2’ appear to be unable to translate from your system locale into UTF-8 (which is used internally), and you will get an error message like this:

     idn2: lookup: could not convert string to UTF-8

One explanation is that you didn't install the ‘iconv’ conversion tools. You can find it as a standalone library in GNU Libiconv (http://www.gnu.org/software/libiconv/). On many GNU/Linux systems, this library is part of the system, but you may have to install additional packages to be able to use it.

Another explanation is that the error is correct and you are feeding ‘idn2’ invalid data. This can happen inadvertently if you are not careful with the character set encoding you use. For example, if your shell run in a ISO-8859-1 environment, and you invoke ‘idn2’ with the ‘LANG’ environment variable as follows, you will feed it ISO-8859-1 characters but force it to believe they are UTF-8. Naturally this will lead to an error, unless the byte sequences happen to be valid UTF-8. Note that even if you don't get an error, the output may be incorrect in this situation, because ISO-8859-1 and UTF-8 does not in general encode the same characters as the same byte sequences.

     jas@latte:~$ idn2 --quiet --debug ""
     Charset: ISO-8859-1
     
     jas@latte:~$ LANG=sv_SE.UTF-8 idn2 --debug räksmörgås
     Charset: UTF-8
     input[0] = 0x72
     input[1] = 0xc3
     input[2] = 0xa4
     input[3] = 0xc3
     input[4] = 0xa4
     input[5] = 0x6b
     input[6] = 0x73
     input[7] = 0x6d
     input[8] = 0xc3
     input[9] = 0xb6
     input[10] = 0x72
     input[11] = 0x67
     input[12] = 0xc3
     input[13] = 0xa5
     input[14] = 0x73
     UCS-4 input[0] = U+0072
     UCS-4 input[1] = U+00e4
     UCS-4 input[2] = U+00e4
     UCS-4 input[3] = U+006b
     UCS-4 input[4] = U+0073
     UCS-4 input[5] = U+006d
     UCS-4 input[6] = U+00f6
     UCS-4 input[7] = U+0072
     UCS-4 input[8] = U+0067
     UCS-4 input[9] = U+00e5
     UCS-4 input[10] = U+0073
     output[0] = 0x72
     output[1] = 0xc3
     output[2] = 0xa4
     output[3] = 0xc3
     output[4] = 0xa4
     output[5] = 0x6b
     output[6] = 0x73
     output[7] = 0x6d
     output[8] = 0xc3
     output[9] = 0xb6
     output[10] = 0x72
     output[11] = 0x67
     output[12] = 0xc3
     output[13] = 0xa5
     output[14] = 0x73
     UCS-4 output[0] = U+0072
     UCS-4 output[1] = U+00e4
     UCS-4 output[2] = U+00e4
     UCS-4 output[3] = U+006b
     UCS-4 output[4] = U+0073
     UCS-4 output[5] = U+006d
     UCS-4 output[6] = U+00f6
     UCS-4 output[7] = U+0072
     UCS-4 output[8] = U+0067
     UCS-4 output[9] = U+00e5
     UCS-4 output[10] = U+0073
     xn--rksmrgs-5waap8p
     jas@latte:~$

The sense moral here is to forget about ‘LANG’ (instead, configure your system locale properly) unless you know what you are doing, and if you want to use ‘LANG’, do it carefully and after verifying with ‘--debug’ that you get the desired results.

Interface Index

• idn2_check_version: Library Functions
• idn2_free: Library Functions
• idn2_lookup_u8: Library Functions
• idn2_lookup_ul: Library Functions
• idn2_register_u8: Library Functions
• idn2_register_ul: Library Functions
• idn2_strerror: Library Functions
• idn2_strerror_name: Library Functions

Concept Index

• command line: Invoking idn2
• Examples: Examples
• idn2: Invoking idn2
• invoking idn2: Invoking idn2
• Library Functions: Library Functions