(This blog post will probably only interest Perl 6 hackers, since it talks only about infrastructure.)

One of the central pieces of Perl 6 infrastructure is the Pugs svn repository. It holds not only the Pugs source code, but also lots of other Perl 6 projects:

• Specification
• Test suite
• STD.pm (the standard grammar)
• SMOP
• mildew and mildew-js
• sprixel
• vill
• mp6, kp6, perlito
• elf
• various websites (perl6.org, pugscode.org, perlcabal.org/syn/)
• a host of scripts related to keep things running (rebuild the HTML version of the synopsis; smartlink checking; cronjobs for updating websites etc.)
• various documentation efforts
• An unknown number of projects more or less related to Perl 6

It's huge, but at the same time it's very practical: anybody who is interested can get write access very easily, create a new subfolder for a new project, or can fix a typo in someone else's README file without asking for commit access first.

The pugs repo is also viral: Anybody with commit access and invite new committers. Despite what you might think of it: it actually works in practice, so far I haven't seen a single case of abuse.

The pugs repository is hosted on feather, a server kindly provided by Juerd and his company. It contains three virtualized servers, feather{1,2,3}. feather2 is used for "sensitive" data (for example an IRC bot that has API keys for various github accounts, and the perl6.org website). Only a handful of "trusted" users have an account there. feather3 is used for low security stuff like evalbots which might go astray.

feather1 holds all the rest. That means the pugs repository, commitbit (the software we use for handing out commit bits and resetting svn passwords), various websites, and a whole bunch of Perl 6 developers and users have a shell account there, for trying out Perl 6 and hosting their screen + irssi sessions there.

I was about to write feather1 is maintained by a bunch of volunteers, but that would be a lie. It is "maintained" on an as-needed basis by whoever has time and feels half-way competent. It is an "interesting" mixture of Debian unstable and experimental. And it's becoming unmaintainable.

It seems clear what to do: set up a fourth virtual machine, set up a replacement for feather1, and en passant migrate some things (like websites and the pugs repo) to feather2.

But wait! There is an issue. There's always an issue. Setting up a new machine and migrating services takes time. Lots of time. And nobody wants to do it right now. Quite understandably.

Take the pugs repository for example: you might think it's easy enough to copy /var/svn/... recursively on the new host and set up Apache... except that you need authentication. And authentication is coupled to commitbit. And commitbit runs on Jifty. And if you want to install Jifty, you install half of CPAN. Does anybody know if commitbit is still maintained? and if it installs cleanly on a Debian stable?

So I thought about some changes to the infrastructure to make it easier to maintain. For example we handle commit bits differently for git projects on github: an IRC bot knows the API keys of the project owners, and can add committers to the projects. That's nice, and the IRC frontend is much leaner than the Jifty-based web frontend. But we lose virality. For security reasons the bot has to keep a whitelist of IRC users who are allowed to invite commiters. Since IRC nick names and (git|svn) user names don't always match, such a list has to be maintained manually.

The second question is: should we take the chance and move the pugs repo to git? I prefer git over svn by far, but there are also costs involved. For example Rakudo checks out a copy of the test suite via svn. The test suite is only a subdirectory of the pugs repo, so unless we split the pugs repo, we'd have to find a way to check out only a part of a git repository. I have no idea if that's possible.

Either way, I'd like to hear your thoughts, and learn from your experience:

• Do you know any "viral" git or svn hosting solution (ie where every committer can invite more committers)? (don't say commitbit - it's a PITA to maintain. Something more lightweight would be appreciated)
• If you use the pugs repository now: do you prefer svn or git? how strongly?

Much has happened in the Perl 6 land in 2009. Here is my humble attempt to summarize some of it; If you find something that I missed, feel free to contact me, I'll try to add it.

Specification

The year started with lots of improvements to S19. In January we also learned that *-1 constructs a closure, which means that Perl 6 has semi-automatic currying features built into most operators.

Lists, Captures and Parcels

We've seen a lot of talk about slices, lists, captures and parcels. The heart of the discussions is always how interpolation and non-interpolation of lists can be made both flexible and intuitive. For example: should 1, 2, 3 Z 'a', 'b', 'c' return a single, flat list? or instead a list of lists? How can a function which receives the result decide for itself what it want to receive? How does that mix with multi-dimensional arrays?

I haven't followed these discussions very closely, and so I'm hard pressed to give a good summary; however it seems that in the end an agreement was reached: each parenthesis constructs a Parcel, short for Parenthesis cell. A Parcel can behave context sensitively: A single-item Parcel degrades to its contents; as a signature list it is converted to a Capture object; code object also return parcels.

It remains to be seen how multi-dimensional slices (with the @@ sigil) evolve, and if we can't find anything suitable to replace them.

Built-in Routines

S29, the list of built-in functions and methods, finally got some long awaited attention in 2009, starting with Carl Mäsak's S29 Laundry List, and later carried on by Timothy Nelson, who split S29 into a set of documents summarized as S32.

In December it was decreed that most built-in methods have a candidate in a new class Cool, (Convenient OO Loopbacks), of which all value types and container types in Perl 6 inherit. That way maximal DWIMyness can be retained, while keeping user defined types clean of the more than hundred methods defined in Cool.

It is rather perlish to have a distinct name for each operation, and make it coerce its arguments. A few exceptions exist in Perl 5 (like reverse, which is list reverse in list context, and string reverse in string context); in Perl 6, most of these exceptions have been removed: reverse now only reverses lists, strings are reverted with flip, hashes with invert.

At the Nordic Perl Workshop, Larry decided that the prefix:<=> operator had to go, and replaced it with the .get and .lines methods.

Operators

The Cross Meta Operator is now Xop instead of XopX; in analogy the R meta operator reverses the argument list, so $a R- $b is the same as $b - $a.

Ranges served two purposes: one for denoting ranges in the sense that the mathematicians use them, and for generating lists according to simple schemes. These two functions have been separated: ranges are still constructed with two dots, but the :by adverb is gone; more intricate, lazy list generation can be achieved with the new series operator:

.say for 1, 1.1, 1.2 ... 5;
.say for 1 ... *+0.1, 5;

Numbers

The above actually works, and doesn't suffer from floating-point arithmetics, because 0.1 isn't stored as a floating-point number, but rather as a fractional number of type Rat.

Other languages decided against that approach, because some very simple loops quickly produce rather large numerators and denominators, degrading performance of the integer operations. Perl 6 instead has a limit in denominator size, and falls back to floating-point operations when that limit is crossed.

Implementations

Rakudo

A lot of work has been done in Rakudo; in fact it's hard to remember how it used to be in January 2009; Most features were implemented by Patrick Michaud and Jonathan Worthington, but we had a lot of other contributors too.

In January, Rakudo left the Parrot repository and since then lives on github as a git repository. It now relies on an installed parrot.

Rakudo implements many new features and lifts old limitations:

• Many built-in routines are now written in Perl 6
• eval() and classes now have access to outer lexical variables
• Much improved Unicode support, both in IO and regular expression
• punning of roles when .new is called
• Typed arrays and hashes, parametric roles
• Routine return types are now enforced
• Error messages now contain backtraces with filenames and line numbers
• Multi dispatch is now implemented with a custom dispatcher and signature binder, bringing much improvements over the dispatch and binding semantics that parrot supports.
• User-defined operators now possible, and automatically generate some of their associated meta-operators.
• Contextual variables
• User-defined traits are now possible; some of the built-in traits are now written in pure Perl 6.
• Rational numbers are now implemented, and support for Complex numbers has been much improved.
• routine signatures can now be introspected properly.

SMOP and Mildew

SMOP and Mildew have seen a major refactoring, connected to the changed semantics of slices, captures and parcels, and to the way method invocations are stored.

Paweł Murias implemented multi dispatch as a Summer of Code project. Mildew now supports an impressive set of features, but since it is not very user oriented, I know of no projects that actually use mildew as a platform.

Other implementations

Elf development seems to have stalled. Pugs mostly sleeps, too, though Audrey updated it to work with the latest Haskell compilers. (It doesn't live in the Pugs repository anymore though, and is distributed by cabal, the Haskell package manager).

New in the field are Sprixel, a Perl 6 to Javascript compiler, and vill, an experimental LLVM backend to STD.pm+viv.

Test Suite

The test suite continued to grow; most tests have now been moved to t/spec/, the official Perl 6 test suite. Most tests in the other remaining files are either rather dubious, or rely on behaviour that's not officially specified (or are specific to an implementation).

Many new tests have been contributed by two new faces: Solomon Foster contributed a large number of tests for trigonometric functions on the various number types, and rational and complex numbers. Kyle Hasselbacher provided us with many regression tests for Rakudo which are also useful to other implementations.

Documentation

Bemoaning the fact that Perl 6 has nearly no user-level documentation, Carl Mäsak started u4x, User-Level Documentation for X-Mas. Hinrik Örn Sigurðsson chimed in, and started to write grok, a tool for retrieving and showing documentation, sponsored by the Google Summer of Code project.

Patrick Michaud, Jonathan Worthington, Carl Mäsak, Jonathan Scott Duff and Moritz Lenz started to work on a Perl 6 book, with a few chapters already being written.

Websites

In an attempt to provide an up-to-date link list, Moritz registered perl6-projects.org and collected links. Later Susanne "Su-Shee" Schmitt contributed a nice design, and Daniel Wright made the domain perl6.org available to us.

So we now have a community driven, central Perl 6 site at perl6.org.

Leo Lapworth redesigned perl.org, and also the old Perl 6 development page, and updated it a bit.

Blogs

As an attempt to improve the visibility of the Perl community, Matt S. Trout issued the Ironman Perl Blogging Challenge. So far it's a huge success, and quite a few hackers blog about Perl 6 there. Also the blog roll of the Planetsix Blog Aggregator continued to grow, some excellent new blogs were added in 2009.

Carl Mäsak blogged at least once per day in Novemeber, same procedure as least year :-)

IRC

The #perl6 IRC channel has been very pleasant and active in 2009, with three times the activity of 2008.

The Future

For April 2010 the Rakudo developers have planned a big release called Rakudo *, not feature complete but still useful and usable. Around the same time the new Perl 6 book will be released.

The specification is still evolving, and has some areas that are in need of implementation before they can evolve more; among them are macros, concurrency and IO.

Update: improved floating point example as per comment from Matthias.

I think it's not too bold to call the Perl 6 Advent Calendar a full success: over 40k page views, more than 150 non-spam comments, and lots of new faces and nicknames in #perl6 - it's been a very pleasant surprise.

I asked myself why it became such a success, and came up with this list of things:

Limited in Scope

24 days are over rather quickly, so everyone who contributes knows when the deed is done

It appeals to very different contributors

Some people like the shiny new regex and grammar features, others are interested in the object model, operator overloading or complex numbers. Since we had no fixed topics, everybody could choose a topic that played to their strengths.

Each task is small and well limited

Writing a good post about a topic you know takes about half an hour up to an hour, maybe two if you need to do some research on the topic. Either way it's a relatively small task (compared to "write a regex engine" or "redesign the build system of $compiler" or so).

Schedule and polite nagging

We had a schedule for each day, and people could add their name to a free slot. When the day approached, the other authors started to ask politely how it was progressing, making sure people did not forget their posts. Also authors felt the subtle pressure to finish their posts on time.

Lots of positive feedback

We had lots of feedback, most of it quite positive: blog comments, visitors in our IRC channel, being featured on slashdot. That was very encouraging.

Also other authors would preview and proof-read the posts before they were published, pointing out falsities and gems.

We had a driving force

PerlJam and colomon took care. The decided to make the advent calendar happen, set up a blog, discussed things and so on - they made it happen.

All these factors encouraged contributions. I don't think all of them are necessary for a successful, lively projects, but they certainly help.

What other factors do you know that encourage contribution in open-source projects? What could we have done even better?

I've blogged about the Perl 6 Advent Calendar 2009, and want to follow up that it's been a huge success so far.

We're about half-way through, and you might be interested in our number of visits per day:

On 6th and 7th of December we had about 18k visitors in sum, courtesy of slashdot and Tim O'Reilly on Twitter.

As a result we got a lot of new people in our #perl6 IRC channel, asking for help on how to get Rakudo working, how to code some specific things in Perl 6, or asking about general design decisions.

Also mj41 reported that he has collected more than 50% more Perl 6/parrot blog posts than in the previous year.

We've also acquired perl6.org for our uses this year, and it has been promoted enough to be the third hit on a google search for Perl 6.

All in all I'm rather happy with the marketing state of Perl 6 in 2009, and hope to see similar efforts for 2010. With the upcoming releases of Rakudo Star and a Perl 6 book I'm pretty sure we'll do well, and I hope for more slashdot coverage :-).

In Perl 5 there is the undef value. Uninitialized variables contain undef, as well as non-existing hash values, reading from unopened or exhausted file handles and so on.

In Perl 6 the situation is a bit more complicated: variables can have a type constraint, and are initialized with the corresponding type object:

my Int $x;
say Int.WHAT();     # Int()

These type objects are also undefined, but in Perl 6 that doesn't mean they are a magical value named undef, but that they respond with False to the defined() subroutine and method.

In fact there is no undef anymore. Instead there are various values that can take its place:

Mu is the type object of the root type of the object hierarchy (or put differently, every object in Perl 6 conforms to Mu). It's the most general undefined value you can think of.

Nil is a "magic" value: in item (scalar) context it evaluates to Mu, in list context it evaluates to the empty list. It's the nothing to see here, move along value.

Each type has a type object; if you want to return a string, but can't decide which, just return a Str.

Other interesting undefined values are Exception (which usually contain a message and a back trace), Failure (unthrown exceptions), Whatever is a generic placeholder that can stand for "all", "infinitely many", "many" or as a placeholder for a real value.

How hard is it to build a good search engine? Very hard. So far I thought that only one company has managed to build a search engine that's not only decent, but good.

Sadly, they seem to have overdone it. Today I searched for tagged dfa. I was looking for a technique used in regex engines. On the front page three out of ten results actually dealt with the subjects, the other uses of dfa meant dog friendly area, department of foreign affairs or other unrelated things.

That's neither bad nor unexpected. But I wanted more specific results, so I decided against using the abbreviation, and searched for the full form: tagged deterministic finite automaton. You'd think that would give better results, no?

No. It gave worse. On the first result page only one of the hits actually dealt with the DFAs I was looking for. Actually the first hit contained none of my search terms. None. It just contained a phrase, which is also sometimes abbreviated dfa.

WTF? Google seemed to have internally converted my query into an ambiguous, abbreviated form, and then used that to find matches, without filtering. So it attempted to be very smart, and came out very stupid.

I doubt that any Google engineer is ever going to read this rant. But if one is: Please, Google, keep it stupid, stupid.

I'm fine with getting automatic suggestions on how to improve my search query; but please don't automatically "improve" it for me. I want to find what I search for. I'm not interested in dog friendly areas.

Most programming languages handle failures with either of two paradigms: failing routines return special values, or they throw exceptions.

Either way has its severe problems: in languages like C it can be very simple to forget to catch such a return value, and very tedious to propagate them to the caller; on the other hand throwing exceptions often clutters the code with way too many try blocks, and it's generally unfriendly if you try to automatically parallelize expressions.

So Perl 6 offers a middle ground: soft or unthrown exceptions. If a routine calls fail("message"), a new Failure object is created and returned from the current routine. That object behaves as an undefined value, which stores the message, file and line information of the fail() location, a backtrace and so on.

When you ask such an object whether it's true or false, or defined or undefined, you'll get a correct answer, and the exception is marked as handled. However if you try to use it as an ordinary value, it turns into an (ordinary) fatal exception. So both of these work:

# Variant 1: no exception thrown

my $handle = open('nonexistingfile');
if $handle {
    .say for $handle.lines;
} else {
    # do something else
}


# Variant 2

my $handle = open('nonexistingfile');

# throws a fatal exception while calling $handle.lines
.say for $handle.lines;

Now if you do some automatically parallelized operations, a single failure doesn't have to abort the whole operation, and neither is information lost

# divide @a1 by @a2 element-wise, a division by zero might occur:
@a1 »/« @a2;

The API for accessing the Failure objects isn't very mature yet, but the concept stands. See S04/Exceptions for the gory details, as they stand today.

<meta>From my useless musings series.</meta>

As a programmer you have to have confidence in your skills, to some extent, and at the same time you have to constantly doubt them. Weird, eh?

Confidence

You need some level of confidence to do anything efficiently. Planning ahead requires confidence that you can achieve the steps on your way.

As a programmer you also need some confidence with the language, libraries and other tools you're using.

If you program for money, you also have to assess what kind of programs you can write, and where you might have problems.

Doubt

In the process of programming you make a lot of assumptions, some of the explicit, some of them implicit. If you want to write a good program, it's essential that you are aware of as many assumptions as possible.

When you find a bug in your program, you have to challenge previous assumptions, and that's where doubt comes in. You not only suspect, but you know that at least one of the assumptions was false (or maybe just a bit too specific), and you know that you did something wrong.

Sometimes programmers make really stupid mistakes which are rather tricky to track down. That's when you have to question your own sanity.

One example (that luckily doesn't happen all that often to me) is when I edit my program, and nothing seems to change. Nothing at all. Depending on the setup it might be some cache, but something it is even more devious - for example I didn't notice that the console where I edit and the console where I test are on different hosts - and thus the edits actually have no effect at all.

After having done such a thing once or twice I adopted the habit of just adding a die('BOOM'); instruction to my code, to verify that the part I'm looking at is actually run.

These are moments when I question my own sanity, thinking "how could I have possibly done such a stupid thing?". Doubt.

The same phenomena applies when doing scientific research: since you usually do things that nobody has done before (or at nobody has published about it yet), you can't know the results beforehand -- if you could, your research would be rather boring. So you have no external reference for verification, only your intuition and discussion with peers.

In the great tradition of Perl Advent Calendars, colomon started and announced the 2009 Perl 6 Advent Calendar, with a post about Perl 6 each day.

After the first post many #perl6 regulars volunteered to contribute a post, so 20 of the 24 days are already allocated.

I'm looking forward to many nice posts, most of which will probably highlight a small Perl 6 feature.

Did you ever wonder how BEGIN, CHECK, END and so on are called in Perl? Well, they didn't have a good name, until recently.

The Perl 6 spec listed them under closure traits, which is unwieldy, and not really exact either. Now they are called phasers, because they tell you which execution phase the block or statement runs in.

There are so many possible puns that I'll refrain from writing any.

If you have an array @a and want to access the first element, in Perl 5 you write that as $a[0], in Perl 6 you write it as @a[0]. We call the former mutable sigil, and the latter immutable sigil.

You might think that's a small change, but the implications are rather deep, and we've had quite a few discussions about it in #perl6. In particular people often ask if it's possible to backport the Perl 6 behavior to Perl 5. The answer is "not easily".

In Perl 5 context propagates inwards, which means that in a statement like

... = func()

The compiler wants to know at compile time which context func() is in. If it doesn't, it complains:

2$ perl -ce '(rand() < 0.5 ? $a : @a) = func()'
Assignment to both a list and a scalar at -e line 1, at EOF
-e had compilation errors.

This also means that, in Perl 5, array slices and scalar array accesses have to be syntactically distinguished:

my @a;
$a{other_func()} = ...; # scalar context
@a{other_func()} = ...; # list context

So you can't just make sigils in Perl 5 immutable without also rewriting the whole context handling rules.

In Perl 6 that's not a problem at all, because functions don't know the context they're in, in fact can't know because of multi dispatch.

Instead functions return objects that behave appropriately in various contexts, and the context is determined at run time.

After getting used to it the immutable sigils are quite nice, and less complicated when references are involved. Anybody who objects without having tried it for at least three weeks, and is spoiled by Perl 5, will be shot.

A few days ago masak blogged about the social aspect of the is Perl 6 really Perl? question.

He presumes that the answer is yes, but doesn't tell us why. I'll try to give some reasons.

Perl 6 started as the successor to Perl 5

Perl 6 started off as the successor to Perl 5, at a Perl 5 meeting, by the Perl crowd. It was a plan to escape both the backwards compatibility trap (which meant that broken things couldn't be fixed without many people yelling), and the lack of momentum in the community.

Perl 6 embraces the Perl philosophy

What makes Perl Perl? In my opinion it's not the sigils on variables that make Perl Perl, or that writing a regex only need two characters and so on. It's mostly the philosophy that makes the difference.

There are some underlying principles like TIMTOWTDI, context sensitivity, convenience over consistency, making simple things easy and hard things possible, often used constructs short and less frequent things longer, and so on.

Perl 6 is founded on all those philosophies and ideals, and also shares some technical principles. For example sigils on variables (oh, I mentioned them already ...), easy access to powerful regexes, that fact that operations coerce their arguments (instead of the type of arguments determining the operation like in javascript, where a + can either mean addition or string concatenation).

So if you agree with my definition of what makes Perl Perl, Perl 6 is also Perl. If not, please tell me what's the essence of Perl!

When you learn a programming language, you not only learn about the syntax, semantics and core libraries, but also the coding style and common idioms.

Idioms are common usage patterns; learning and reusing them means you have to spend less time thinking on common things, and have more time working out the algorithms you deal with.

That's different if you learn Perl 6 - it's a largely unexplored field, and while there are loads of nice features, you might still feel a bit lost. At least I do. That's because I often think "There's got to be a much easier way to achieve $this, but it often takes time to find that easier solution - because nobody developed an idiom for it.

In those cases it helps to ask on the #perl6 IRC channel; many smart people read and write there, and are rather good in coming up with simpler solutions.

For example see masak's ROT13 implementation in Perl 6. In the right column you can see later revisions, and how they gradually improve, steady up to a one-liner.

I also made some simplifications to JSON::Tiny, which basically shows that when I wrote these reduction methods first I used Perl 6 baby talk language.

The nice things about exploring the Perl 6 wonderland of unexplored idioms is that it really pushes your ego if you find a nice simplification, and that you have something to blog about for the Planet Perl Iron man ;-)

These tidings posts seem to become rather sparse, but I hope you get some news by reading the Planet Six feed aggregator anyway.

Specification

• Larry lifted up the dual nature of Ranges. They mostly serve as an interval now, for smart iteration the series operator has been pimped up. You can now write for 1, 3 ... *+2, 9 { .say } to print all the odd numbers between 1 and 9. (r28344, r28348, r28351).
• Rational and Complex types now have their own literals (r28173).
• Stubbed classes are now documented (r28196, r28197).
• The new S08 documents Parcels and Captures.
• The numeric types have been cleaned up a lot (r28502 and later commits up to r28597).
• New and improved signature introspection (r28664, r28665).

Compilers

Rakudo

As opposed to two months ago, Rakudo now

• supports the Rat type
• supports overloading of many built-in operators
• has contextual variables
• has a faster and much better signature binder
• supports all kind of trigonometric functions, including on complex numbers
• implements sophisticated signature introspection

Patrick Michaud is also working on a new tool named npq-rx, which combines a self-hosting NQP compiler and a new regex engine, which already supports proto regexes, NQP code assertions and closures, and is generally much faster and better than PGE.

Sprixel

Mathew Wilson aka diakopter started sprixel, a Perl 6 to Javascript compiler.

Mildew

Mildew now has an experimental javascript emitter.

Other matters

perl6.org is redesigned again, this time spanning multiple pages, thus allowing much more stuff to be linked there.

Four Perl 6 and Rakudo hackers announced that they are writing a Perl 6 book, the print release of which shall coincide with the release of Rakudo Star.

We want a Perl 6 book. We want it badly enough to write it ourselves. So that's what we're doing: writing one.

We, that is Patrick Michaud (architect of the Rakudo Perl compiler), Jonathan Worthington (prolific contributor to both Rakudo and Parrot), Carl Mäsak (frenetic Rakudo user, and our number one bug finder) and Moritz Lenz (keeper of the Perl 6 test suite, and Perl 6 user and blogger). We are also open to contribution from others - already Jonathan Scott Duff has written an initial preface for us.

We don't have a name yet for our book. We want to cover the basics of Perl 6, enough to get your feet wet, and enough to make you want to use it. We want it to be based on useful examples. It is not going to be the definitive book, that task we leave to Larry Wall and Damian Conway.

Our vision is to present primarily the subset of Perl 6 that Rakudo understands, and have printed copies available by the time Rakudo Star is released, that is April or May 2010. chromatic and Allison Randal have kindly offered to published it via Onyx Neon Press.

Until then, monthly releases will be published under a Creative Commons license (noncommercial, attribution, share-alike).

Currently we have four chapters under construction, and the intention of writing the more introductory chapters later, when we know what we need to introduce for the later chapters. So far we have

• Multi dispatch
• Classes and Object
• Regexes
• Grammars

You can download the preliminary PDF version of the book here.

Interested? Check out the git repository, and join us in irc://freenode.net#perl6book.