Perl5180delta (1) Linux Manual Page

NAME

perl5180delta – what is new for perl v5.18.0

DESCRIPTION

This document describes differences between the v5.16.0 release and the v5.18.0 release.

If you are upgrading from an earlier release such as v5.14.0, first read perl5160delta, which describes differences between v5.14.0 and v5.16.0.

Core Enhancements

New mechanism for experimental features

Newly-added experimental features will now require this incantation:

    no warnings "experimental::feature_name";
    use feature "feature_name";  # would warn without the prev line

There is a new warnings category, called “experimental”, containing warnings that the feature pragma emits when enabling experimental features.

Newly-added experimental features will also be given special warning IDs, which consist of “experimental::” followed by the name of the feature. (The plan is to extend this mechanism eventually to all warnings, to allow them to be enabled or disabled individually, and not just by category.)

By saying

    no warnings "experimental::feature_name";

you are taking responsibility for any breakage that future changes to, or removal of, the feature may cause.

Since some features (like "~~" or "my $_") now emit experimental warnings, and you may want to disable them in code that is also run on perls that do not recognize these warning categories, consider using the "if" pragma like this:


    no if $] >= 5.018, warnings => "experimental::feature_name";

Existing experimental features may begin emitting these warnings, too. Please consult perlexperiment for information on which features are considered experimental.

Hash overhaul

Changes to the implementation of hashes in perl v5.18.0 will be one of the most visible changes to the behavior of existing code.

By default, two distinct hash variables with identical keys and values may now provide their contents in a different order where it was previously identical.

When encountering these changes, the key to cleaning up from them is to accept that hashes are unordered collections and to act accordingly.

Hash randomization

The seed used by Perl’s hash function is now random. This means that the order which keys/values will be returned from functions like "keys()", "values()", and "each()" will differ from run to run.

This change was introduced to make Perl’s hashes more robust to algorithmic complexity attacks, and also because we discovered that it exposes hash ordering dependency bugs and makes them easier to track down.

Toolchain maintainers might want to invest in additional infrastructure to test for things like this. Running tests several times in a row and then comparing results will make it easier to spot hash order dependencies in code. Authors are strongly encouraged not to expose the key order of Perl’s hashes to insecure audiences.

Further, every hash has its own iteration order, which should make it much more difficult to determine what the current hash seed is.

New hash functions

Perl v5.18 includes support for multiple hash functions, and changed the default (to ONE_AT_A_TIME_HARD), you can choose a different algorithm by defining a symbol at compile time. For a current list, consult the INSTALL document. Note that as of Perl v5.18 we can only recommend use of the default or SIPHASH. All the others are known to have security issues and are for research purposes only.

PERL_HASH_SEED environment variable now takes a hex value

"PERL_HASH_SEED" no longer accepts an integer as a parameter; instead the value is expected to be a binary value encoded in a hex string, such as “0xf5867c55039dc724”. This is to make the infrastructure support hash seeds of arbitrary lengths, which might exceed that of an integer. (SipHash uses a 16 byte seed.)

PERL_PERTURB_KEYS environment variable added

The "PERL_PERTURB_KEYS" environment variable allows one to control the level of randomization applied to "keys" and friends.

When "PERL_PERTURB_KEYS" is 0, perl will not randomize the key order at all. The chance that "keys" changes due to an insert will be the same as in previous perls, basically only when the bucket size is changed.

When "PERL_PERTURB_KEYS" is 1, perl will randomize keys in a non-repeatable way. The chance that "keys" changes due to an insert will be very high. This is the most secure and default mode.

When "PERL_PERTURB_KEYS" is 2, perl will randomize keys in a repeatable way. Repeated runs of the same program should produce the same output every time.

"PERL_HASH_SEED" implies a non-default "PERL_PERTURB_KEYS" setting. Setting "PERL_HASH_SEED=0" (exactly one 0) implies "PERL_PERTURB_KEYS=0" (hash key randomization disabled); setting "PERL_HASH_SEED" to any other value implies "PERL_PERTURB_KEYS=2" (deterministic and repeatable hash key randomization). Specifying "PERL_PERTURB_KEYS" explicitly to a different level overrides this behavior.

Hash::Util::hash_seed() now returns a string

Hash::Util::hash_seed() now returns a string instead of an integer. This is to make the infrastructure support hash seeds of arbitrary lengths which might exceed that of an integer. (SipHash uses a 16 byte seed.)

Output of PERL_HASH_SEED_DEBUG has been changed

The environment variable PERL_HASH_SEED_DEBUG now makes perl show both the hash function perl was built with, and the seed, in hex, in use for that process. Code parsing this output, should it exist, must change to accommodate the new format. Example of the new format:

    $ PERL_HASH_SEED_DEBUG=1 ./perl -e1
    HASH_FUNCTION = MURMUR3 HASH_SEED = 0x1476bb9f

Upgrade to Unicode 6.2

Perl now supports Unicode 6.2. A list of changes from Unicode 6.1 is at <http://www.unicode.org/versions/Unicode6.2.0>.

Character name aliases may now include non-Latin1-range characters

It is possible to define your own names for characters for use in "\N{...}", "charnames::vianame()", etc. These names can now be comprised of characters from the whole Unicode range. This allows for names to be in your native language, and not just English. Certain restrictions apply to the characters that may be used (you can’t define a name that has punctuation in it, for example). See “CUSTOM ALIASES” in charnames.

New DTrace probes

The following new DTrace probes have been added:

•: "op-entry"
•: "loading-file"
•: "loaded-file"

${^LAST_FH}

This new variable provides access to the filehandle that was last read. This is the handle used by $. and by "tell" and "eof" without arguments.

Regular Expression Set Operations

This is an experimental feature to allow matching against the union, intersection, etc., of sets of code points, similar to Unicode::Regex::Set. It can also be used to extend "/x" processing to [bracketed] character classes, and as a replacement of user-defined properties, allowing more complex expressions than they do. See “Extended Bracketed Character Classes” in perlrecharclass.

Lexical subroutines

This new feature is still considered experimental. To enable it:

    use 5.018;
    no warnings "experimental::lexical_subs";
    use feature "lexical_subs";

You can now declare subroutines with "state sub foo", "my sub foo", and "our sub foo". ("state sub" requires that the “state” feature be enabled, unless you write it as "CORE::state sub foo".)

"state sub" creates a subroutine visible within the lexical scope in which it is declared. The subroutine is shared between calls to the outer sub.

"my sub" declares a lexical subroutine that is created each time the enclosing block is entered. "state sub" is generally slightly faster than "my sub".

"our sub" declares a lexical alias to the package subroutine of the same name.

For more information, see “Lexical Subroutines” in perlsub.

Computed Labels

The loop controls "next", "last" and "redo", and the special "dump" operator, now allow arbitrary expressions to be used to compute labels at run time. Previously, any argument that was not a constant was treated as the empty string.

More CORE:: subs

Several more built-in functions have been added as subroutines to the CORE:: namespace – namely, those non-overridable keywords that can be implemented without custom parsers: "defined", "delete", "exists", "glob", "pos", "prototype", "scalar", "split", "study", and "undef".

As some of these have prototypes, "prototype('CORE::...')" has been changed to not make a distinction between overridable and non-overridable keywords. This is to make "prototype('CORE::pos')" consistent with "prototype(&CORE::pos)".

kill with negative signal names

"kill" has always allowed a negative signal number, which kills the process group instead of a single process. It has also allowed signal names. But it did not behave consistently, because negative signal names were treated as 0. Now negative signals names like "-INT" are supported and treated the same way as -2 [perl #112990].

Security

Storable security warning in documentation

The documentation for "Storable" now includes a section which warns readers of the danger of accepting Storable documents from untrusted sources. The short version is that deserializing certain types of data can lead to loading modules and other code execution. This is documented behavior and wanted behavior, but this opens an attack vector for malicious entities.

Locale::Maketext allowed code injection via a malicious template

If users could provide a translation string to Locale::Maketext, this could be used to invoke arbitrary Perl subroutines available in the current process.

This has been fixed, but it is still possible to invoke any method provided by "Locale::Maketext" itself or a subclass that you are using. One of these methods in turn will invoke the Perl core’s "sprintf" subroutine.

In summary, allowing users to provide translation strings without auditing them is a bad idea.

This vulnerability is documented in CVE-2012-6329.

Avoid calling memset with a negative count

Poorly written perl code that allows an attacker to specify the count to perl’s "x" string repeat operator can already cause a memory exhaustion denial-of-service attack. A flaw in versions of perl before v5.15.5 can escalate that into a heap buffer overrun; coupled with versions of glibc before 2.16, it possibly allows the execution of arbitrary code.

The flaw addressed to this commit has been assigned identifier CVE-2012-5195 and was researched by Tim Brown.

Incompatible Changes

An unknown character name in \N{…} is now a syntax error

Previously, it warned, and the Unicode REPLACEMENT CHARACTER was substituted. Unicode now recommends that this situation be a syntax error. Also, the previous behavior led to some confusing warnings and behaviors, and since the REPLACEMENT CHARACTER has no use other than as a stand-in for some unknown character, any code that has this problem is buggy.

Formerly deprecated characters in \N{} character name aliases are now errors.

Since v5.12.0, it has been deprecated to use certain characters in user-defined "\N{...}" character names. These now cause a syntax error. For example, it is now an error to begin a name with a digit, such as in

 my $undraftable = "\N{4F}";    # Syntax error!

or to have commas anywhere in the name. See “CUSTOM ALIASES” in charnames.

\N{BELL} now refers to U+1F514 instead of U+0007

Unicode 6.0 reused the name “BELL” for a different code point than it traditionally had meant. Since Perl v5.14, use of this name still referred to U+0007, but would raise a deprecation warning. Now, “BELL” refers to U+1F514, and the name for U+0007 is “ALERT”. All the functions in charnames have been correspondingly updated.

New Restrictions in Multi-Character Case-Insensitive Matching in Regular Expression Bracketed Character Classes

Unicode has now withdrawn their previous recommendation for regular expressions to automatically handle cases where a single character can match multiple characters case-insensitively, for example, the letter LATIN SMALL LETTER SHARP S and the sequence "ss". This is because it turns out to be impracticable to do this correctly in all circumstances. Because Perl has tried to do this as best it can, it will continue to do so. (We are considering an option to turn it off.) However, a new restriction is being added on such matches when they occur in [bracketed] character classes. People were specifying things such as "/[

perl5180delta (1) Linux Manual Page