Musings on Rakudo's spectest chart

In an attempt to track progress in a measurable way, the Rakudo people count the number of passing tests each night (ok, they don't do that manually :-) and write that into a CSV file.

The next step wasn't far: generate a chart from it, put it on a website, and tell everybody about it. Here's a snapshot:

Old news so far, as an eager Perl 6 follower you surely know that already, nothing new here. But the other day somebody asked me why the green (PASS) and the grey (number of spectests available) seem to be parallel.

My first reaction was "no, they aren't parallel" because the big jumps don't happen at the same time, but still there are large areas where they do seem to grow roughly at the same rate. So what's up?

As far as I can tell there are two mechanism at work here. The first one accounts for much of the past behaviour: tests were moved from the old part of the pugs test suite to the official test suite (where they are counted towards the grey line), and at the same time people also tried to get it to run under Rakudo - mostly successfully, as the green line shows.

Now most test files are actually moved to the official Perl 6 test suite, there are less than 200 tests left to move (as opposed to 17k already moved), and this mechanism is drained.

The second mechanism is that as soon as Rakudo implements a cool new feature, we realize that we need more tests for it, and people write more tests for it. (It's rather hard to write tests for language feature that doesn't run yet. "test first" is a nice idea, but it only works to some extent in reality). Also people explore the new features, find bugs, and bug tickets are only closed if there are tests for them, so even more spec tests that coincide with passing Rakudo tests.