perlunitut (1) Linux Manual Page

NAME

perlunitut – Perl Unicode Tutorial

DESCRIPTION

The days of just flinging strings around are over. It’s well established that modern programs need to be capable of communicating funny accented letters, and things like euro symbols. This means that programmers need new habits. It’s easy to program Unicode capable software, but it does require discipline to do it right.

There’s a lot to know about character sets, and text encodings. It’s probably best to spend a full day learning all this, but the basics can be learned in minutes.

These are not the very basics, though. It is assumed that you already know the difference between bytes and characters, and realise (and accept!) that there are many different character sets and encodings, and that your program has to be explicit about them. Recommended reading is “The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)” by Joel Spolsky, at <http://joelonsoftware.com/articles/Unicode.html>.

This tutorial speaks in rather absolute terms, and provides only a limited view of the wealth of character string related features that Perl has to offer. For most projects, this information will probably suffice.

Definitions

It’s important to set a few things straight first. This is the most important part of this tutorial. This view may conflict with other information that you may have found on the web, but that’s mostly because many sources are wrong.

You may have to re-read this entire section a few times…

Unicode

Unicode is a character set with room for lots of characters. The ordinal value of a character is called a code point. (But in practice, the distinction between code point and character is blurred, so the terms often are used interchangeably.)

There are many, many code points, but computers work with bytes, and a byte has room for only 256 values. Unicode has many more characters than that, so you need a method to make these accessible.

Unicode is encoded using several competing encodings, of which UTF-8 is the most used. In a Unicode encoding, multiple subsequent bytes can be used to store a single code point, or simply: character.

UTF-8

UTF-8 is a Unicode encoding. Many people think that Unicode and UTF-8 are the same thing, but they’re not. There are more Unicode encodings, but much of the world has standardized on UTF-8.

UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take only one byte per character. All other characters are encoded as two to four bytes using a complex scheme. Fortunately, Perl handles this for us, so we don’t have to worry about this.

Text strings (character strings)

Text strings, or character strings are made of characters. Bytes are irrelevant here, and so are encodings. Each character is just that: the character.

On a text string, you would do things like:

$text = ~s / foo / bar / ;
if ($string = ~ / ^\d + $ /) {
    ...
}
$text = ucfirst $text;
my $character_count = length $text;

The value of a character ("ord", "chr") is the corresponding Unicode code point.

Binary strings (byte strings)

Binary strings, or byte strings are made of bytes. Here, you don’t have characters, just bytes. All communication with the outside world (anything outside of your current Perl process) is done in binary.

On a binary string, you would do things like:

my(@length_content) = unpack “(V/a)*”, $binary;
$binary = ~s /