tag:blogger.com,1999:blog-66233146312793662212024-03-13T12:42:46.263-07:00Developing with InfernoNotes and articles vaguely related to the Inferno distributed system and the Limbo programming languageforsythhttp://www.blogger.com/profile/00068719978877211213noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-6623314631279366221.post-56748114212498881502008-04-14T05:22:00.000-07:002008-04-16T09:30:28.702-07:00Compensating for Spamhaus<span style="font-style: italic;font-size:85%;" ><span style="font-weight: bold;">Exercise</span>. </span><span style="font-size:85%;">Some pundits claim that (so-called) end-users should never ever send their own mail directly. You beg to differ, especially after experience with rubbish ISPs, and for years at work and at home have happily delivered your e-mail directly from your network to destinations, using Plan 9's pleasant little <a style="font-style: italic;" href="http://plan9.bell-labs.com/magic/man2html/8/smtp">upas/smtp</a>. Why bother with middlemen?<br /><br />One day, your Internet supplier provides a shiny new set-top box with separate cable modem. Plug in the RJ45, hit a few configuring web pages, DHCP, update Sender Policy entries on <span style="font-style: italic;">dyndns</span>, and away you go. Except that your mail is now being rejected by some sites. Yahoo? MSN? Hate them anyway. Gmail?! Oh dear! Why? There is a peculiar organisation, let's call it <span style="font-style: italic;">Spamhaus</span> (which really ought to be the name of a group that sends spam), that busies itself making lists of IP address ranges that supposedly belong to these horrible end-user people. (The ones that pay to connect to the Internet, but never mind.) It turns out that your new address, unlike the old, is on a list. (Same supplier, but again, never mind.)<br /><br />Others in your house are now distressed by rejected mail (they actually know people on Yahoo and MSN!). Fix it, using other computers and software as needed, but without introducing a store-and-forward phase or changing your mail domain.<br /><br /><span style="font-style: italic;"><span style="font-weight: bold;">Solution</span>.</span> The network in the exercise is running Plan 9, but saddled with a Spamhosed address. You have fortunately got access to a virtual server<span style="font-style: italic;"></span> elsewhere with a safe address, but it is running Linux. Run hosted Inferno on <span style="font-style: italic;">theserver</span> and export <span style="font-weight: bold;">/net</span>:</span><span style="font-size:85%;"></span><br /><div style="text-align: left;"><blockquote><span style="font-weight: bold;font-size:85%;" ><span style="font-family:courier new;">exec /usr/inferno/Linux/386/bin/emu /dis/sh.dis -c "\<br /></span></span><div style="text-align: center;"><span style="font-weight: bold;font-size:85%;" ><span style="font-family:courier new;"> </span></span><span style="font-size:85%;"><span style="font-family:courier new;"><span style="font-weight: bold;">listen 'tcp!*!</span><span style="font-style: italic; font-weight: bold;">port</span><span style="font-weight: bold;">' {export -a /net}"</span></span></span><span style="font-weight: bold;"></span><br /></div></blockquote></div><span style="font-size:85%;">That makes the socket-based interfaces of the Linux system accessible through the name space exported on <span style="font-style: italic;">port</span> by the Inferno system using the <a href="http://www.vitanuova.com/inferno/papers/styx.html">Styx protocol</a>.<br /><br />On the Plan 9 system, add a chunk similar to the following to <span style="font-style: italic;">/mail/lib/remotemail</span>, before it calls <span style="font-style: italic;">smtp:</span></span><br /><div style="text-align: left;"><span style="font-weight: bold;"><span style="font-size:85%;"><blockquote>while(! mount /srv/netexp /n/remote){<br /><div style="text-align: center;"> {rm -f /srv/netexp && srv tcp!<span style="font-style: italic;">theserver</span>!<span style="font-style: italic;">port</span> netexp} ||<br /></div><div style="text-align: center;"> exit "import failed"<br /><div style="text-align: left;">}<br /></div></div>bind /n/remote/tcp /net/tcp || exit "failed bind"</blockquote></span></span><span style="font-size:85%;">The Styx (=9P) connection for the <span style="font-weight: bold;">/net</span> exported from <span style="font-style: italic;">theserver</span> is cached in <span style="font-weight: bold;">/srv/netexp</span>, and mounted at <span style="font-weight: bold;">/n/remote</span> in <span style="font-style: italic;">remotemail</span>'s own name space, allowing just <span style="font-weight: bold;">/net/tcp</span> to be bound in from the other machine. (If the cached connection has hung up, we get another.) When <span style="font-style: italic;">upas/smtp</span> is later invoked by <span style="font-style: italic;">remotemail,</span> it will dial the destination machines using the name <span style="font-weight: bold;">/net/tcp</span> <a href="http://plan9.bell-labs.com/sys/doc/net.html">as usual for Plan 9</a>, but the name will refer to the instance bound from the </span><span style="font-size:85%;">other machine, which thus acts as a TCP/IP gateway, transparently. Because only the remote <span style="font-weight: bold;">/net/tcp</span> is used, DNS lookups will use the local <span style="font-weight: bold;">/net/udp</span>, and local name server <span style="font-weight: bold;">/net/cs</span>. The outgoing TCP/IP traffic (just for <span style="font-style: italic;">smtp)</span> will have <span style="font-style: italic;">theserver</span>'s IP address, because it is using the server's TCP/IP sockets via the hosted Inferno. Adjust the Sender Policy Framework and related DNS entries to suit, and get back to productive work.<br /><br /></span><br /></div>forsythhttp://www.blogger.com/profile/00068719978877211213noreply@blogger.com0tag:blogger.com,1999:blog-6623314631279366221.post-33233335501533872492008-04-13T05:09:00.000-07:002008-04-14T05:21:12.381-07:00Summer of Code 2007 results and experience<span style="font-size:85%;">Inferno projects in 2007 ended up within the <span style="font-style: italic;">Plan 9 from Bell Labs </span>organisation. You can read about it in last year's <a href="http://gsoc.cat-v.org/blog/">blog</a>. I was mentor for three of those projects: <a href="http://world.std.com/%7Ecme/html/spki.html">SPKI</a> infrastructure for Inferno (Katie Reynolds/katelyn); <a href="http://plan9.bell-labs.com/sys/doc/venti.html">Venti</a>-like system in Limbo for Inferno, with added Rabin fingerprinting (Mechiel Lukkien/mjl); and a port of Inferno to the Nintendo DS (Noah Evans). The students were all talented, which was just as well, since otherwise acting as mentor for three projects (and helping a bit on some others) would have been quite impossible. The projects were modular, so that timing and expectations could be adjusted fairly easily as the summer wore on, and there was something to show for it all early on.<br /><br />Here is an edited version of a post I made elsewhere just after the programme ended, of our experience of GSoC 2007.<br /><br /></span><div style="text-align: left;"><div style="text-align: justify;"><div style="text-align: justify;"><span style="font-size:85%;"> I had three students to mentor and they all produced work that is</span><span style="font-size:85%;"> being included in the organisation's distributions. One <span class="nfakPe">of</span> the</span><span style="font-size:85%;"> projects (SPKI) finally</span><span style="font-size:85%;"> wrote <span class="nfakPe">code</span> to implement some ideas I had originally intended to</span><span style="font-size:85%;"> implement three years ago,</span><span style="font-size:85%;"> but had to put aside for lack <span class="nfakPe">of</span> time. That success in turn is</span><span style="font-size:85%;"> leading to a significant change to ancient <span class="nfakPe">code</span> and mechanisms in one</span><span style="font-size:85%;"><span class="nfakPe"> of</span> our systems, mainly by deleting <span class="nfakPe">code</span> from its kernels. (So in a</span><span style="font-size:85%;"> way, for us it was the Google <span class="nfakPe">Summer</span> <span class="nfakPe">of</span> Anti-<span class="nfakPe">code</span>,</span><span style="font-size:85%;"> which seems good to me.)</span><br /></div><br /><span style="font-size:85%;">Those three projects were all quite hard. The SPKI one</span><span style="font-size:85%;"> required installing</span><span style="font-size:85%;"> two related but different operating systems and a large application</span><span style="font-size:85%;"> suite, and then </span><span style="font-size:85%;">writing <span class="nfakPe">code</span> in both C (for one part) and a concurrent programming</span><span style="font-size:85%;"> language the student had never seen before</span><span style="font-size:85%;"> (Limbo, for other parts). Another project required writing an archival</span><span style="font-size:85%;"> storage subsystem</span><span style="font-size:85%;"> broadly based on an existing design (Venti) but including some new</span><span style="font-size:85%;"> techniques. In the third project,</span><span style="font-size:85%;"> the student got the Inferno operating system settled as a native kernel on a new platform — first</span><span style="font-size:85%;"> on an emulator, then on the hardware —</span><span style="font-size:85%;"> without previous experience <span class="nfakPe">of</span> doing kernel ports.</span><br /><br /><span style="font-size:85%;"> Despite the relative difficulty, the projects all worked out well,</span><span style="font-size:85%;"> because the students settled down</span><span style="font-size:85%;"> and did the work, and they kept it up. </span><span style="font-size:85%;">Sometimes I would </span><span style="font-size:85%;">receive e-mail starting "<span style="font-style: italic;">This is probably a silly question, but ...</span>", and not only was it not silly, it revealed</span><span style="font-size:85%;"> some long-standing flaw/deficiency/confusion in system or</span><span style="font-size:85%;"> documentation or both. The students have also expressed</span><span style="font-size:85%;"> interest in continuing to contribute to the underlying systems, time</span><span style="font-size:85%;"> and graduate study permitting.</span><br /><br /><span style="font-size:85%;"> None <span class="nfakPe">of</span> this would have happened without GSoC.</span><br /><br /><span style="font-size:85%;"> The style <span class="nfakPe">of</span> interaction was different for each student. E-mail was</span><span style="font-size:85%;"> used quite a bit, partly because <span class="nfakPe">of</span> timezone</span><span style="font-size:85%;"> differences, and partly because I prefer it because it offers a chance to think about the</span><span style="font-size:85%;"> questions or responses, but one <span class="nfakPe">of</span> the students</span><span style="font-size:85%;"> used Google chat with me quite a bit at critical points to good</span><span style="font-size:85%;"> effect. Interaction generally increased after mid-term</span><span style="font-size:85%;"> evaluation, partly because the more complex stages had been reached, but mainly because I felt guilty about not spending as much time on the programme as I had originally intended, owing to a change in my own workload. </span><span style="font-size:85%;">Generally I found it similar to supervising a student project, but</span><span style="font-size:85%;"> with the added complications <span class="nfakPe">of</span> not being</span><span style="font-size:85%;"> able to assess the student's ability as easily, having to cope with time zones, and</span><span style="font-size:85%;"> deadlines in the day job.</span><br /><span style="font-size:85%;"><br />Another GSoC organisation suggested that detailed specifications might improve the outcome.</span><span style="font-size:85%;"> One <span class="nfakPe">of</span> my three projects was defined in considerable detail</span><span style="font-size:85%;"> (including external references), another was fairly</span><span style="font-size:85%;"> obvious in scope, and the third had an overall aim but no real</span><span style="font-size:85%;"> detail. Thus, from my own experience, I cannot</span><span style="font-size:85%;"> conclude that either over- or under-specification is critical (or not)</span><span style="font-size:85%;"> to success. "It depends."</span><br /><br /><span style="font-size:85%;">What else might help? All three projects were committing source <span class="nfakPe">code</span> well before</span><span style="font-size:85%;"> the mid-term point. Each <span class="nfakPe">of</span> the projects</span><span style="font-size:85%;"> had distinct stages identified, and all were deliberately or</span><span style="font-size:85%;"> inherently open-ended (making it easy to add or remove</span><span style="font-size:85%;"> items depending on actual progress). Insisting on fairly regular</span><span style="font-size:85%;"> supply <span class="nfakPe">of</span> <span class="nfakPe">code</span> (or at least design material and discussion)</span><span style="font-size:85%;"> helps to highlight writer's block.</span><br /><br /><span style="font-size:85%;"> Now, the Plan 9 organisation as a whole had 13 projects and 5 failed</span><span style="font-size:85%;"> the final evaluation, which is</span><span style="font-size:85%;"> not so much too high in absolute terms as too high too late (we ought</span><span style="font-size:85%;"> to have failed more at mid-term).</span><span style="font-size:85%;"> But then again, there were 8 successful projects, and none <span class="nfakPe">of</span> those</span><span style="font-size:85%;"> would have happened without GSoC.</span><span style="font-size:85%;"> It's a bit like one <span class="nfakPe">of</span> those tabloid newspaper articles bemoaning that</span><span style="font-size:85%;"> "20% <span class="nfakPe">of</span> people do/believe/hope some horrible thing"</span><span style="font-size:85%;"> when a reasonable response might be that 80% <span class="nfakPe">of</span> people do not. Our</span><span style="font-size:85%;"> students succeeded</span><span style="font-size:85%;"> more often than not. Still, we all hate wasting time and money on</span><span style="font-size:85%;"> failure.</span><br /><br /><span style="font-size:85%;"> I later read through the original applications, the</span><span style="font-size:85%;"> mid-term reviews and the</span><span style="font-size:85%;"> final reviews, to see whether there was anything to distinguish the</span><span style="font-size:85%;"> set that succeeded from the set that did not.</span><span style="font-size:85%;"> I was reminded that during the application competition, I thought the</span><span style="font-size:85%;"> quality <span class="nfakPe">of</span> both GSoC applicants and</span><span style="font-size:85%;"> the coherence <span class="nfakPe">of</span> their applications was good. We had well over 100</span><span style="font-size:85%;"> applications, and there were only a few</span><span style="font-size:85%;"> that were spam or no-hopers. The ones we accepted that subsequently</span><span style="font-size:85%;"> failed still looked plausible to me.</span><br /><br /><span style="font-size:85%;"> For the projects I was going to mentor, I was particular about</span><span style="font-size:85%;"> assessing the students' portfolios, which included</span><span style="font-size:85%;"> any sort <span class="nfakPe">of</span> code they'd previously written, ideally on their own,</span><span style="font-size:85%;"> perhaps as a student project. I would not </span><span style="font-size:85%;">have declined to mentor someone who was relatively</span><span style="font-size:85%;"> inexperienced, but I would never have</span><span style="font-size:85%;"> agreed to mentor them on an ambitious project</span><span style="font-size:85%;">.<br /><br />Still, only one <span class="nfakPe">of</span> the three</span><span style="font-size:85%;"> that I did mentor had any relevant experence <span class="nfakPe">of</span> their chosen project area, so the</span><span style="font-size:85%;"> previous work wasn't directly applicable, and</span><span style="font-size:85%;"> they still had quite a bit <span class="nfakPe">of</span> work to do.</span><span style="font-size:85%;"></span><span style="font-size:85%;"> On the other hand, at least one project that failed had an apparently</span><span style="font-size:85%;"> experienced student who could point to previous, plausible,</span><span style="font-size:85%;"> and even relevant work. As a rule, though, we'd probably pay special</span><span style="font-size:85%;"> attention to evident capability in future.</span><br /><br /><span style="font-size:85%;"> All but two <span class="nfakPe">of</span> the failed projects were reasonable ones that could be</span><span style="font-size:85%;"> expected to be done (to an adequate level) in the time available.</span><span style="font-size:85%;"> We ought, however, to have failed the three simpler projects at mid-</span><span style="font-size:85%;">term for lack <span class="nfakPe">of</span> progress.</span><span style="font-size:85%;"> One <span class="nfakPe">of</span> those would have been especially good fun, and had a good</span><span style="font-size:85%;"> mentor for it, who helped agree a simplified project</span><span style="font-size:85%;"> at mid-term, but the student simply did not seem to come to grips with</span><span style="font-size:85%;"> it at all. I eventually realised that all the projects that did fail had a worried mentor at mid-term, and the ones that succeeded looked fine to their mentors (even when the students were worried about progress).<br /></span><span style="font-size:85%;"><br />One <span class="nfakPe">pleasant surprise for me was that the native</span> kernel port to Nintendo DS succeeded.</span><span style="font-size:85%;"> One rule <span class="nfakPe">of</span> thumb I originally had for GSoC projects was "never</span><span style="font-size:85%;"> suggest doing a device driver let alone a kernel port", but although one</span><span style="font-size:85%;"> port did fail, one</span><span style="font-size:85%;"> worked. The reason for avoiding drivers and ports</span><span style="font-size:85%;"> is that although they seem like fine projects, we know historically it</span><span style="font-size:85%;"> is quite difficult to help debug them at a distance,</span><span style="font-size:85%;"> even when both parties ostensibly have the same hardware, and that</span><span style="font-size:85%;"> seemed a big risk for a short <span class="nfakPe">summer</span> project. The Nintendo port has attracted an enthusiastic group, and <a href="http://code.google.com/p/inferno-ds/">work continues in a project on Google code</a>.</span><br /><br /><span style="font-size:85%;">Our rather loose (in several ways) organisation</span><span style="font-size:85%;"> overall received a big benefit from participating in GSoC:</span><span style="font-size:85%;"> there is now more <span class="nfakPe">code</span> in public, and just as important, more visibility and more participants in</span><span style="font-size:85%;"> the larger project.</span><br /></div><br /></div>forsythhttp://www.blogger.com/profile/00068719978877211213noreply@blogger.com0tag:blogger.com,1999:blog-6623314631279366221.post-33968113019199656492007-03-05T06:43:00.000-08:002008-04-14T05:20:37.183-07:00Summer of Code 2007<div>It is at last lighter earlier in the morning and later in the evening here in England, hinting at the end of hibernation, the arrival of spring and with it, the start of this year's <a href="http://code.google.com/soc/">Google Summer of Code</a>. Last year we inadvertently <a href="http://www.amazon.com/While-England-Select-Bibliiographic-Reprint/dp/0829008047">slept through that</a>. <i>This</i> year we are better prepared.</div><div align="left"><br />During this last week, we have extended <a href="http://code.google.com/p/inferno-os/wiki/Project_Suggestions">the set of projects</a> on the Wiki at <a href="http://code.google.com/p/inferno-os/">the Inferno-os site at code.google.com</a>, adding a new section specifically for Summer of Code projects. Of course, those projects could be done at any other time, and other projects on that page might also be good projects for students. Still, the hope was that the Summer suggestions themselves would be a little different. The topics include naming in networks, language implementation, constraint-based systems, data archiving, design and implementation of library modules, and Inferno-related plug-ins for browsers. Whether they have been given short paragraphs or just bullet-points as descriptions, all the projects are essentially open-ended. Even so, they are intended to allow development to be done in stages during the time available. In every case, at the end of the summer there should be something substantial and useful to show for it.<br /><br />I thought the work should also be reasonably self-contained, without programmers having to become familiar with large chunks of existing code. After all, those that have not seen or used <a href="http://www.vitanuova.com/inferno/">Inferno</a> before will usually take a little while to come to grips with the <a href="http://www.vitanuova.com/inferno/papers/limbo.html">Limbo language</a> and the Inferno environment. There is plenty of interesting work to be done in each project, making use of the novel aspects of Inferno to be sure, but also learning about a given application area. What do I mean by "interesting"? Every one of them is a project I should like to do myself, and perhaps has languished on my own TODO list, and none of them is "grunge work".<br /><br /><div align="justify">Notably excluded from the Summer section are native ports of the Inferno kernel (<i>native</i> ports are ports to raw hardware). Native ports are good projects, but I have experience now of people doing ports remotely and it is often just too hard to diagnose trouble at a distance, even if both parties have supposedly identical bits of hardware. The debugging support is often a little too Spartan for novices. As summer projects they are quite risky. By contrast, there has been more success with so-called <i>hosted</i> ports, where the Inferno kernel runs as an application under another operating system, and a few such projects are included.<br /></div><br />One of the earliest mediaeval rounds merrily celebrates the lively arrival of early summer after the chill of winter: <blockquote><i>Sumer is icumin in,<br />Lhude sing, cuccu!</i></blockquote><br />The <a href="http://www.bbc.co.uk/schoolradio/music/musicextra_spring06_programme08.shtml">BBC Schools' Radio web site</a> notes about it that:<blockquote>As the cannon [sic!] develops, the melody weaves around itself to create a rich and complex rhythmic and harmonic pattern</blockquote><div align="left">That rich, constructive result is one of the hopes for this collection of projects.<br /></div><div align="left"><br /></div></div>forsythhttp://www.blogger.com/profile/00068719978877211213noreply@blogger.com0tag:blogger.com,1999:blog-6623314631279366221.post-85788626891249224762006-12-21T16:42:00.000-08:002008-04-14T05:21:27.590-07:00Subversive Inferno<span style="font-family: lucida grande;">Now that Inferno's source code is available via Subversion on Googlecode, with incremental updates there (if it all works), I thought I'd try to keep a related blog. Fortunately it is Christmas, and I am on holiday. Meanwhile, Caerwyn's Inferno Programmer's Notebook (<a href="http://www.caerwyn.com/ipn/">http://www.caerwyn.com/ipn/</a>) sets a sufficiently high standard to satisfy readers (and put me off even trying) until the New Year. Still, why "Subversive" in the title of my initial post?</span><span style="font-family: lucida grande;"></span><span style="font-family: courier new;"><br /></span><span style="font-family: courier new;"><blockquote></blockquote></span><ol><li> Sub*vert" (?), v. t. [imp. & p. p. /Subverted/; p. pr. & vb. n. /Subverting/.] [L. subvertere, subversum; sub under + vertere to turn: cf. F. subvertir. See /Verse/.]<br /></li></ol><blockquote>1. To overturn from the foundation; to overthrow; to ruin utterly.<br /> These are his substance, sinews, arms, and strength,<br /> With which he yoketh your rebellious necks,<br /> Razeth your cities, and subverts your towns. Shak.<br /> This would subvert the principles of all knowledge. Locke. </blockquote><blockquote>2. To pervert, as the mind, and turn it from the truth; to corrupt; to confound. 2 Tim. iii. 14. Syn. -- To overturn; overthrow; destroy; invert; reverse; extinguish.<br /><br />...<br /></blockquote>It seems to have been a poor choice: ruination, corruption, turning from the truth etc. are certainly not the targets. Changing something at the foundation is probably closer to the real aim, and given the vast amount of existing code and systems, aiming to displace them is not realistic, but turning them all into run-time platforms for Inferno is both practical and sound.<br /><br /><span style="font-family: courier new;"></span>forsythhttp://www.blogger.com/profile/00068719978877211213noreply@blogger.com3