Categories
Posts in this category
- A shiny perl6.org site
- Creating an entry point for newcomers
- Sprixel, a 6 compiler powered by JavaScript
- Another perl6.org iteration
- Blackjack and Perl 6
- Why I commit Crud to the Perl 6 Test Suite
- Report from the Perl 6 Hackathon in Copenhagen
- Custom operators in Rakudo
- Defined Behaviour with Undefined Values
- Dissecting the "Starry obfu"
- Perl 6: Failing Softly with Unthrown Exceptions
- The first Perl 6 module on CPAN
- Google Summer of Code Mentor Recap
- Building a Huffman Tree With Rakudo
- Immutable Sigils and Context
- Is Perl 6 really Perl?
- Perl 6: Lost in Wonderland
- Lots of momentum in the Perl 6 community
- Musing and the future of feather and the Pugs repository
- Musings on Rakudo's spectest chart
- My first executable from Perl 6
- Trying to implement new operators - failed
- Let's build an object
- Perl 6 is optimized for fun
- How to get a parse tree for a Perl 6 Program
- Perl 6 in 2009
- Perl 6 ticket life cycle
- The Perl 6 Advent Calendar
- How to Plot a Segment of a Circle with SVG
- Publicity for Perl 6
- Rakudo architectural overview
- Rakudo Rocks
- Rakudo "star" announced
- Rakudo's rough edges
- Rats and other pets
- Releasing Rakudo made easy
- Set Phasers to Stun!
- Starry Perl 6 obfu
- Recent Perl 6 Developments August 2008
- Strings and Buffers
- Subroutines vs. Methods - Differences and Commonalities
- A SVG plotting adventure
- A Syntax Highlighter for Perl 6
- Test Suite Reorganization: How to move tests
- The Happiness of Design Convergence
- Perl 6 Tidings from September and October 2008
- Perl 6 Tidings for November 2008
- Perl 6 Tidings from December 2008
- Perl 6 Tidings from January 2009
- Perl 6 Tidings from February 2009
- Perl 6 Tidings from March 2009
- Perl 6 Tidings from April 2009
- Perl 6 Tidings from May 2009
- Perl 6 Tidings from May 2009 (second iteration)
- Perl 6 Tidings from June 2009
- Perl 6 Tidings from August 2009
- Perl 6 Tidings from October 2009
- Timeline for a syntax change in Perl 6
- Visualizing match trees
- We write a Perl 6 book for you
- When we reach 100% we did something wrong
- Where Rakudo Lives Now
- Why was the Perl 6 Advent Calendar such a Success?
- What you can write in Perl 6 today
- Why you don't need the Y combinator in Perl 6
Fri, 15 Jan 2010
Musing and the future of feather and the Pugs repository
Permanent link
(This blog post will probably only interest Perl 6 hackers, since it talks only about infrastructure.)
One of the central pieces of Perl 6 infrastructure is the Pugs svn repository. It holds not only the Pugs source code, but also lots of other Perl 6 projects:
- Specification
- Test suite
- STD.pm (the standard grammar)
- SMOP
- mildew and mildew-js
- sprixel
- vill
- mp6, kp6, perlito
- elf
- various websites (perl6.org, pugscode.org, perlcabal.org/syn/)
- a host of scripts related to keep things running (rebuild the HTML version of the synopsis; smartlink checking; cronjobs for updating websites etc.)
- various documentation efforts
- An unknown number of projects more or less related to Perl 6
It's huge, but at the same time it's very practical: anybody who is interested can get write access very easily, create a new subfolder for a new project, or can fix a typo in someone else's README file without asking for commit access first.
The pugs repo is also viral: Anybody with commit access and invite new committers. Despite what you might think of it: it actually works in practice, so far I haven't seen a single case of abuse.
The pugs repository is hosted on feather, a server kindly provided by Juerd and his company. It contains three virtualized servers, feather{1,2,3}. feather2 is used for "sensitive" data (for example an IRC bot that has API keys for various github accounts, and the perl6.org website). Only a handful of "trusted" users have an account there. feather3 is used for low security stuff like evalbots which might go astray.
feather1 holds all the rest. That means the pugs repository, commitbit (the software we use for handing out commit bits and resetting svn passwords), various websites, and a whole bunch of Perl 6 developers and users have a shell account there, for trying out Perl 6 and hosting their screen + irssi sessions there.
I was about to write feather1 is maintained by a bunch of volunteers, but that would be a lie. It is "maintained" on an as-needed basis by whoever has time and feels half-way competent. It is an "interesting" mixture of Debian unstable and experimental. And it's becoming unmaintainable.
It seems clear what to do: set up a fourth virtual machine, set up a replacement for feather1, and en passant migrate some things (like websites and the pugs repo) to feather2.
But wait! There is an issue. There's always an issue. Setting up a new machine and migrating services takes time. Lots of time. And nobody wants to do it right now. Quite understandably.
Take the pugs repository for example: you might think it's easy enough to
copy /var/svn/... recursively on the new host and set up
Apache... except that you need authentication. And authentication is coupled
to commitbit. And commitbit runs on Jifty. And if you want to install Jifty,
you install half of CPAN. Does anybody know if commitbit is still maintained?
and if it installs cleanly on a Debian stable?
So I thought about some changes to the infrastructure to make it easier to maintain. For example we handle commit bits differently for git projects on github: an IRC bot knows the API keys of the project owners, and can add committers to the projects. That's nice, and the IRC frontend is much leaner than the Jifty-based web frontend. But we lose virality. For security reasons the bot has to keep a whitelist of IRC users who are allowed to invite commiters. Since IRC nick names and (git|svn) user names don't always match, such a list has to be maintained manually.
The second question is: should we take the chance and move the pugs repo to git? I prefer git over svn by far, but there are also costs involved. For example Rakudo checks out a copy of the test suite via svn. The test suite is only a subdirectory of the pugs repo, so unless we split the pugs repo, we'd have to find a way to check out only a part of a git repository. I have no idea if that's possible.
Either way, I'd like to hear your thoughts, and learn from your experience:
- Do you know any "viral" git or svn hosting solution (ie where every committer can invite more committers)? (don't say commitbit - it's a PITA to maintain. Something more lightweight would be appreciated)
- If you use the pugs repository now: do you prefer svn or git? how strongly?
Comments / Trackbacks:
Trackback URL:
/blog-en/perl-6/musings-on-feather-and-pugscode.trackback
Kyle Hasselbacher wrote
I prefer Git over SVN, but it's not a strong preference since I learned to use git-svn.
I think checkout of a subtree in Git is still, um, undesirable. They do, however, have subproject support. http://git.or.cz/gitwiki/SubprojectSupport We could break the Pugs repo into subprojects (perhaps based on the bullet points you have above) and have a master superproject that contains them. I say this without having tried it or fully understanding subproject support as it is.
How complex is commitbit? I don't see it in the pugs repo... Would it help to rewrite it with Catalyst? If there were a move to Git, it would have to change anyway, right?
Juerd wrote
Re-use the existing authorization mechanism! Anyone with a commit bit can... commit something! You could create a folder that is a queue for new commit bits. To hand out a bit, touch $queuedir/email@example.org - a cronjob can poll this and move the files to $queuedir/done as messages are sent.
Martin Berends wrote
The subdirectory checkout capability may have more applications than those listed, for example mildew depends on 2 or 3 subtrees. The Git subproject alternative does not seem flexible enough. That's my vote in favour of retaining Subversion.
Building a replacement for feather1 makes a lot of sense. Because of the "issues" mentioned, let's document a migration plan (somewhere in the Pugs repo) and include testing of the new system before we retire feather1. Let's also be willing to change the plan and re-execute in a mini-whirlpool if necessary. I'm happy to look into Debian aspects if desired (or Ubuntu Server).
Write a comment
The comments on this blog post have been disabled; the comment form below will not work.