Posts in this category
- Current State of Exceptions in Rakudo and Perl 6
- Meet DBIish, a Perl 6 Database Interface
- doc.perl6.org and p6doc
- Exceptions Grant Report for May 2012
- Exceptions Grant Report -- Final update
- Perl 6 Hackathon in Oslo: Be Prepared!
- Localization for Exception Messages
- News in the Rakudo 2012.05 release
- News in the Rakudo 2012.06 release
- Perl 6 Hackathon in Oslo: Report From The First Day
- Perl 6 Hackathon in Oslo: Report From The Second Day
- Quo Vadis Perl?
- Rakudo Hack: Dynamic Export Lists
- SQLite support for DBIish
- Stop The Rewrites!
- Upcoming Perl 6 Hackathon in Oslo, Norway
- A small regex optimization for NQP and Rakudo
- Pattern Matching and Unpacking
- Rakudo's Abstract Syntax Tree
- The REPL trick
- First day at YAPC::Europe 2013 in Kiev
- YAPC Europe 2013 Day 2
- YAPC Europe 2013 Day 3
- A new Perl 6 community server - call for funding
- A shiny perl6.org site
- Creating an entry point for newcomers
- An offer for software developers: free IRC logging
- Announcing try.rakudo.org, an interactive Perl 6 shell in your browser
- Another perl6.org iteration
- Blackjack and Perl 6
- Why I commit Crud to the Perl 6 Test Suite
- This Week's Contribution to Perl 6 Week 5: Implement Str.trans
- This Week's Contribution to Perl 6
- This Week's Contribution to Perl 6 Week 8: Implement $*ARGFILES for Rakudo
- This Week's Contribution to Perl 6 Week 6: Improve Book markup
- This Week's Contribution to Perl 6 Week 2: Fix up a test
- This Week's Contribution to Perl 6 Week 9: Implement Hash.pick for Rakudo
- This Week's Contribution to Perl 6 Week 11: Improve an error message for Hyper Operators
- This Week's Contribution to Perl 6 - Lottery Intermission
- This Week's Contribution to Perl 6 Week 3: Write supporting code for the MAIN sub
- This Week's Contribution to Perl 6 Week 1: A website for proto
- This Week's Contribution to Perl 6 Week 4: Implement :samecase for .subst
- This Week's Contribution to Perl 6 Week 10: Implement samespace for Rakudo
- This Week's Contribution to Perl 6 Week 7: Implement try.rakudo.org
- What is the "Cool" class in Perl 6?
- Report from the Perl 6 Hackathon in Copenhagen
- Custom operators in Rakudo
- A Perl 6 Date Module
- Defined Behaviour with Undefined Values
- Dissecting the "Starry obfu"
- The case for distributed version control systems
- Perl 6: Failing Softly with Unthrown Exceptions
- Perl 6 Compiler Feature Matrix
- The first Perl 6 module on CPAN
- A Foray into Perl 5 land
- Gabor: Keep going
- First Grant Report: Structured Error Messages
- Second Grant Report: Structured Error Messages
- Third Grant Report: Structured Error Messages
- Fourth Grant Report: Structured Error Messages
- Google Summer of Code Mentor Recap
- How core is core?
- How fast is Rakudo's "nom" branch?
- Building a Huffman Tree With Rakudo
- Immutable Sigils and Context
- Is Perl 6 really Perl?
- Mini-Challenge: Write Your Prisoner's Dilemma Strategy
- Longest Palindrome by Regex
- Perl 6: Lost in Wonderland
- Lots of momentum in the Perl 6 community
- Monetize Perl 6?
- Musings on Rakudo's spectest chart
- My first executable from Perl 6
- My first YAPC - YAPC::EU 2010 in Pisa
- Trying to implement new operators - failed
- Programming Languages Are Not Zero Sum
- Perl 6 notes from February 2011
- Notes from the YAPC::EU 2010 Rakudo hackathon
- Let's build an object
- Perl 6 is optimized for fun
- How to get a parse tree for a Perl 6 Program
- Pascal's Triangle in Perl 6
- Perl 6 in 2009
- Perl 6 in 2010
- Perl 6 in 2011 - A Retrospection
- Perl 6 ticket life cycle
- The Perl Survey and Perl 6
- The Perl 6 Advent Calendar
- Perl 6 Questions on Perlmonks
- Physical modeling with Math::Model and Perl 6
- How to Plot a Segment of a Circle with SVG
- Results from the Prisoner's Dilemma Challenge
- Protected Attributes Make No Sense
- Publicity for Perl 6
- PVC - Perl 6 Vocabulary Coach
- Fixing Rakudo Memory Leaks
- Rakudo architectural overview
- Rakudo Rocks
- Rakudo "star" announced
- My personal "I want a PONIE" wish list for Rakudo Star
- Rakudo's rough edges
- Rats and other pets
- The Real World Strikes Back - or why you shouldn't forbid stuff just because you think it's wrong
- Releasing Rakudo made easy
- Set Phasers to Stun!
- Starry Perl 6 obfu
- Recent Perl 6 Developments August 2008
- The State of Regex Modifiers in Rakudo
- Strings and Buffers
- Subroutines vs. Methods - Differences and Commonalities
- A SVG plotting adventure
- A Syntax Highlighter for Perl 6
- Test Suite Reorganization: How to move tests
- The Happiness of Design Convergence
- Thoughts on masak's Perl 6 Coding Contest
- The Three-Fold Function of the Smart Match Operator
- Perl 6 Tidings from September and October 2008
- Perl 6 Tidings for November 2008
- Perl 6 Tidings from December 2008
- Perl 6 Tidings from January 2009
- Perl 6 Tidings from February 2009
- Perl 6 Tidings from March 2009
- Perl 6 Tidings from April 2009
- Perl 6 Tidings from May 2009
- Perl 6 Tidings from May 2009 (second iteration)
- Perl 6 Tidings from June 2009
- Perl 6 Tidings from August 2009
- Perl 6 Tidings from October 2009
- Timeline for a syntax change in Perl 6
- Visualizing match trees
- Want to write shiny SVG graphics with Perl 6? Port Scruffy!
- We write a Perl 6 book for you
- When we reach 100% we did something wrong
- Where Rakudo Lives Now
- Why Rakudo needs NQP
- Why was the Perl 6 Advent Calendar such a Success?
- What you can write in Perl 6 today
- Why you don't need the Y combinator in Perl 6
- You are good enough!
Sun, 13 Sep 2009
When we reach 100% we did something wrong
<meta> This is not a marketing blurb as I originally intended to write. It is quite a lengthy and a bit philosophical brain dump. If my regular readers skip it that's fine; I promise that my next post will be shorter, more exciting and easier to digest.</meta>
This is a very useful information, but still its use is limited. Among other reactions we also received two kinds of responses that I want to talk about. One is "And when Rakudo passes all tests, Perl 6 is ready, right?", the other is "I don't care how many tests you pass until it's 100% (the latter seen in a reddit or Y-combinator comment, for example).
The first reaction points to a larger problem - how can we know if our tests actually cover the whole specification? The answer is we can't. Ever. We can identify parts of the spec that are very well covered by tests (for example multi dispatch), and parts that are covered badly or not at all (IO, and way too many other areas). But there is a huge number of ways the various parts of the specification interact (growing exponential in the number of specified facts), which makes it impossible to reach complete covering.
Still we can read each paragraph in the specs and see if we have an appropriate test, achieving some kind of shallow coverage.
But worse still there are many things implicit but not explicit in the spec. I write "worse" because for the test suite writer it's a pain, but on the other hand if everything was stated explicitly, we'd waste too much time on trivialities.
One example is a bug which Rakudo had in the beginning: recursion would screw up some lexical variables rather badly. When Patrick and Jonathan looked into fixing it, I wanted to contribute my part and provided some tests. And then I didn't know where to put them. The specs talk about subroutines, and about lexical variables. But it does not talk about the interaction between recursion and lexicals - because it's obvious that each instance of a function gets its own lexical pad on recursion. Perl 5 does this just fine, we carry that semantics over to Perl 6. Perfectly sane, but it's an implication nonetheless.
And, after much talking, I finally come to my first main point: We can't measure test coverage of such implications. We have no automatic way to turn implications into explicit statements either. We can't know if spec and the implications have a decent test coverage.
We won't ever reach 100% passing tests
Most people consider Perl 5 "done". Not in the sense that there's nothing left to do, but that there are stable releases, wide acceptance, on generally technical and social maturity. And yet Perl 5 does not pass all of its test.
It might come as a shock to you if you haven't looked at Perl 5's source tree, and never compiled it from source and run the tests (and actually looked at the results), but it's true.
$ ~/tmp/perl> ack -w 'skip(?:_all)?' t|wc -l 381 $ ~/tmp/perl> ack -w 'todo' t|wc -l 48
There are many valid reasons why some tests should not be run - for example the tested feature might not be available on the current platform, or maybe a previous failure would make their outcome inconclusive.
There are also valid reasons why some tests are marked as TODO (which means they are run, but don't pass) - for example to test a bug that is not yet fixed, or for behavior that is subject to change but has not yet been adapted.
So even with huge efforts our chances of ever unconditionally passing each and every test in the Perl 6 test suite are practically 0. When a run over the whole test suite shows that all tests passed, the first thing I'll do is to check our test module and test harness for errors - maybe some bug prevented them from successfully identifying errors.
If you compare how the tests for Perl 5 and Perl 6 were originally written, you'll see that in the case of Perl 5 a huge amount was written against a working implementation, while many Perl 6 tests were whipped up by reading the spec and transforming it into code, or even as feature requests for pugs. So it won't surprise you that in the Perl 6 test suite there is a larger amount of contradictory tests, written either against different versions of the specification, or by various people who understood the spec differently. We'll work hard to weed out such contradictions, but it would be an illusion to assume that we will fully succeed.
Are you moody?
When you read the previous paragraphs you might think I'm moody, and quite negative about Perl 6. I'm not. I just faced the reality that each software project of at least modest complexity faces, and brain-dumped it here.
In fact I'm quite optimistic. Many areas of Perl 6 improve at an impressive rate these days: the specification, Rakudo, smop+mildew, the websites, available libraries and applications, and last but not least: the community.
Having written quite some Perl 6 code (see my SVG related posts on this blog) over the previous weekend I think that developing Perl 6 has never been more fun - at least not to me ;-)
Comments / Trackbacks:
Write a comment