-
Parsing Quoted Strings in Ruby April 28th, 2008
Python has a nice module in the standard library called shlex that parses strings as a Unix shell would. Here’s a Python interpreter session demonstrating its usage:
>>> import shlex >>> shlex.split('foo "bar baz" qux') ['foo', 'bar baz', 'qux']
It’s useful for creating your own mini-languages or external DSLs that need to parse quoted strings like the one shown above.
We recently built an inventory tracking application in Ruby that has a user interface for selecting search filters. To expose the same search capabilities as a web service, we created a simple query language. I was looking for an
shlexequivalent in Ruby.It turns out that the Ruby Standard Library has a module called Shellwords:
>> require 'shellwords' => true >> Shellwords::shellwords('foo "bar baz" qux') => ["foo", "bar baz", "qux"]
Shellwordsis a little less capable thanshlexbut handles the most common use case just fine. It’s a convenient solution for a problem that comes up too often. -
Speaking at OSCON 2008 April 2nd, 2008
The OSCON 2008 website has published its talk schedule. I’ll be giving two talks at OSCON this year; one on the Python track and one on the PHP track.Supervisor as a Platform
I will quickly introduce you to Supervisor and the immediate benefits of running your server processes under it. We will then dive into how applications written specifically for Supervisor can take advantage of it as a platform — writing your own event listeners to observe Supervisor and the process lifecycle, controlling with XML-RPC, and extending the Supervisor core with your own Python extensions.
This will be an expanded version of the talk I gave with Chris McDonough at PyCon 2008. Since PyCon, there’s been quite a few interesting developments in Supervisor like the ability to extend
supervisorctland progress made on configuration reloading. We’ll touch on these as well, so if you attended the PyCon talk there will still be new and interesting material in this talk for you.Integration Testing PHP Applications
While more PHP developers are recognizing the importance and benefits of unit testing, the uptake of PHP developers using automated integration or acceptance testing is relatively slow. This testing is equally crucial to maintaining the integrity of applications.
I’m going to help get you started testing at the application level with practical tips and source code. We’ll look at how to structure your HTML markup so it’s more easily testable, making tests easier to write and maintain with CSS selectors, organizing your tests, and testing with or without a browser.
We write a lot of PHP applications with this kind of testing at Maintainable. Before the conference, we plan to release some open source PHP code to help you test that we’ll cover in the talk as well. I’d also suggest you check out Sebastian Bergmann’s tutorial session on PHPUnit’s integration with Selenium RC.
-
PyCon 2008 Wrap-Up March 20th, 2008
At PyCon 2008, I gave a talk titled Supervisor as a Platform together with Chris McDonough. Our talk was very well attended and the audience even included Python creator Guido van Rossum. We received a lot of positive feedback about our recent work and I suspect we picked up a good number of new users as well.
About two months ago, we began a push to build quality documentation for Supervisor using Docbook. After this was complete, we set out to build a new web presence. Our efforts culminated at PyCon 2008 when we unveiled the new Supervisor website, built by the team at Maintainable Software.
PyCon was also very productive, with hacking on Supervisor, Repoze, and general WSGI fun. I worked on a new feature to allow Supervisor’s process group configurations to be reloaded without restarting Supervisor. Meanwhile, Chris made
supervisorctlpluggable so extensions can add new commands to the Supervisor command line.I also attended quite a few interesting talks, met some new people, enjoyed hanging out with usual the Zope and Plone folks, and overall just had a good time. PyCon 2008 was great conference and I’m looking forward to next year.
-
Supervisor Twiddler 0.2 February 17th, 2008
Supervisor Twiddler is an RPC extension for Supervisor that allows Supervisor’s configuration and state to be manipulated in ways that are not normally possible at runtime.
Changes in release 0.2:
Renamed
addProcessToGroup()toaddProgramToGroup(). The method
now supports program definitions withnumprocs> 1 and will add
all resulting processes from the program definition. It also
fixes a bug where the process config was not added to the group
properly. Requested by Roger Hoover.Added new method
log()that writes an arbitrary message to the
main supervisord log. This is useful for recording information
about your twiddling.You can download the new version and read the updated documentation on the Supervisor Twiddler page of Maintainable Software’s open source project pages.
-
Supervisor at PyCon 2008 December 21st, 2007
I’ll be presenting at PyCon 2008, together with Chris McDonough.
Supervisor is a tool for managing UNIX processes. Supervisor starts arbitrary processes at its own startup and allows otherwise unprivileged users to start, stop and restart its subprocesses and view its subprocess’ logs using a command line or web interface. Other programs exist which do this, but what sets Supervisor apart is that it is written in Python and built with extension points that can be leveraged by Python developers. In this talk, we’ll look at Supervisor as a platform, and how Python programs written to run under Supervisor can use its unique capabilities.
Our talk will cover basic usage but will focus on the more advanced features we’ve added recently, such as adding custom RPC interfaces and event listeners.
PyCon 2008 will in Chicago on March 13-16th. From the list of talks, it looks like this is going to be a great conference. I am especially looking forward to pounding out some new code during the sprints following the conference on the 17-20th.
-
Supervisor Extensions Released November 5th, 2007
Supervisor is a process management system written in Python.
One of the major features we added to Supervisor recently is the ability to extend its XML-RPC interface functionality by registering new top-level namespace factories.
Today, I released two new extension packages for Supervisor:
- Supervisor Twiddler allows Supervisor’s configuration and state to be manipulated in ways that are not normally possible at runtime.
- Supervisor Cache provides the ability to cache arbitrary data in the Supervisor instance as key/value pairs.
Both of these packages add capabilities to Supervisor through this new extensibility. Since they are the first packages available of their kind, they also serve as a template for building your own Supervisor extensions.
Update: The extensions were featured on Chris McDonough’s blog.
-
Repoze: Retooling Zope November 1st, 2007
During the last couple of weeks, the Maintainable team has been privileged to help out with Repoze, a new open source Python project. Repoze is “plumbing Zope into the WSGI pipeline” but also much more.
I’m not a Zope developer but I’ve benefited from Zope indirectly in several ways. The most significant for me has been Supervisor, a process management system. It’s not part of Zope itself but it originated from a need to keep production Zope servers running smoothly. I started using it for other kinds of servers, and fast forward a couple of years and now I’m very involved and help develop it.
Supervisor is interesting to many Zope outsiders. It might have grown out of the needs of running Zope servers, but it’s a tool that many others can benefit from using. I think parts of Zope itself also have similar potential for outsiders.
Zope has been around for a long time and for many was thought to be the “killer app” for Python. Philip J. Eby wrote about the significance of Zope in Where Zope Leads, Python Follows. I’m interested in Zope not because of the application server as a whole but because it has many technologies like an object database, transactions, and object publishing that I’d like to explore.
It seems that quite a few technologies that are gaining mindshare now through new developments have been available and stable in Zope for many years. As an outsider, I want to be able to consume bits of Zope technologies for other kinds of projects.
The problem that Zope has is that its technologies aren’t accessible to outsiders in the same way as Supervisor. To the rest of us, the Zope world is largely a mysterious monolith. Projects like Grok make it easier to get started by bringing Rails-like conventions to Zope development, but many of the core Zope technologies remain largely inaccessible for new purposes without putting in a lot of effort.
Repoze is changing that in a significant way. Repoze is packaging Zope technologies into components and middleware that can be easily deployed on WSGI servers. This is a win for Zope applications because it means better deployment options like Apache and mod_wsgi. Breaking up the monolith and providing more options will help the longevity of their platform.
For the rest of us, a more modular world of Zope means that eventually, many of the interesting technologies that Zope has to offer will be closer in reach and more practical through a combination of WSGI middleware and better packaging.
Repoze already seems to have great momentum and I’ll be watching it with interest. I look forward to exploring what the Zope world has to offer through Repoze.
Update: This post was featured on Repoze Notes.
-
New in Horde: Routes September 15th, 2007
I’m pleased to announce the first release of Horde/Routes, a new URL mapping system for PHP 5. This package provides classes for mapping URLs into the controllers and actions of an MVC system, inspired by Ruby on Rails.
There are already quite a few existing libraries that do this sort of thing for PHP. Horde/Routes is a compelling alternative.
At Maintainable, we examined most of these PHP solutions and found them all inadequate for various reasons, particularly because we wanted RESTful routing, named routes, sophisticated matching, PHP 5 E_STRICT, and extensive test coverage.
Since we do quite a bit of Ruby and Python programming, we surveyed the options in those languages and decided to do a full port of the Python library, Routes.
Horde/Routes provides these features and more:
- Supports route recognition and generation
- Sophisticated matching conditions like subdomains
- Named routes and RESTful route generation
- PEAR-style naming and coding standards
- PHP 5 E_STRICT compliant, web framework agnostic
- A comprehensive unit test suite
Maintainable decided to contribute the code to Horde’s Rampage project because Horde is one of PHP’s oldest and most successful projects. We’re using and contributing to other Horde libraries and we think Rampage is worth your attention.
The Python version has been around for some time and is very popular with different Python web frameworks. We’re happy to be part of that ecosystem now and our full port has already resulted in patches being committed back to the Python version.
At Maintainable, we’ve already used Horde_Routes on several applications. While Horde/Routes is relatively new, it is very feature-rich and well-tested.
Currently, Horde/Routes is a beta release. Over the coming weeks, we’ll be making some minor changes to the API and adding more documentation, and then it will quickly move to stable.
Update: The project now has its own pages on the Horde website and was featured on Chuck Hagenbuch’s blog.
-
Faster TDD with Stakeout.rb September 4th, 2007
I’m a big fan of Autotest and it runs almost constantly on my machine. Autotest automatically reruns your tests whenever your files change. Instead of constantly flipping to another shell to rerun your tests, just let Autotest cheerfully do it for you in the background. It’s highly addictive.
The only problem with Autotest is that it is specific to Ruby. At work, I do a mix of different kinds of programming including Ruby, PHP, Python, and C. I’d like my TDD to be accelerated for all of these languages.
Thanks to Geoffrey Grossenbach, last week I came across stakeout.rb from Mike Clark. This is a tiny, dead simple Ruby script that runs an arbitrary command when certain files change. This is a stripped-down Autotest for everybody else. I’m sure it has all kinds of other uses as well.
To get started testing with
stakeout.rb, you’ll need Ruby installed. Any recent version is fine and you might already have it installed. Next, grab the stakeout.rb script and add the shebang line to the top (Unix-like OS assumed):#!/usr/bin/env ruby -w if ARGV.size < 2 puts "Usage: stakeout.rb <command> [files to watch]+" ...
Make the file executable and put it somewhere in your
PATH. You can test it out by typingstakeout.rbfrom an arbitrary directory and you should see the help message.Next, change over to a project directory where you have some test files. Most of the projects that I am involved with tend to use some directory structure similar to this:
/project_name /lib /test ...
To test such a project, run
stakeout.rbfrom the/project_namedirectory. Most PHP projects using PHPUnit tend to have anAllTests.phpfile or equivalent to run all the tests, so we’ll assume this for the example:project_name$ stakeout.rb "php test/AllTests.php" **/*
The first argument is what command to run when the tests change. The second argument, and any subsequent arguments, are the files to watch for changes. These can use a Ruby globbing pattern. The pattern
**/*will watch all files underproject_namerecursively, which includeslib/andtest/.Once
stakeout.rbis run, it will show no output but will sit and wait for changes. As soon as you change a watched file,stakeout.rbwill automatically rerun you tests and will continue to do so until you exit with Control-C. -
MashupCamp February 23rd, 2006
Earlier this week, I spent two days at MashupCamp. This great event was held at the Computer History Museum in Mountain View, California.
Mashups are an interesting topic. “Mashup” is one of those terms like “Web 2.0″ that’s trendy and really hard to define in any concrete terms. Basically, mashups are taking two or more web service APIs (or data somehow gleamed from the web) and then building a useful application by combining them.
The caliber of developers that attended was top notch and this could easily be seen by the great apps that were demonstrated. These were the kind of folks that know XML inside and out, know everything annoying about SOAP headers, know XHTML strict better than HTML 4, and can tell you the method signatures of the services they use from memory. Every developer that I talked with had experience with at least two scripting languages, usually more. It was a great crowd with some very smart and talented people in attendance.
Mashups themselves tend to get put together very quickly. Sometimes people get an idea and implement a mashup in a few hours or a few days. It was great to be around so many developers who really grok the newest web technologies and know the fastest ways to develop applications to take advantage of them. So, what were they using to get things done fast? Besides the mashups themselves, that’s what I was interested in learning about.
It turns out these developers were really pragmatic. They use what works and gets their job done the fastest without getting in the way — even if that means Perl. I don’t mean Catalyst or any other kind of fancy frameworkish thing, I mean straight Perl CGI. If you read the blogs these days, you’d think Perl is almost dead for web applications. I counted four mashups using Perl in some way. The most common application was backend processing but I saw one using it on the frontend as well.
Three mashups that I saw were using Python. One of them was built on the Twisted asynchronous framework. I’ve done some work with Medusa but haven’t worked on a project needing all the power of Twisted yet. It certainly looks impressive. Another Python mashup was the excellent chicagocrime.org website, built on the Django framework by its co-creator Adrian Holovaty. A lot of people I talked with at MashupCamp were interested in Python, especially at the Googleplex where I attended an afterparty (thanks, Adam!). Several Googlers told me Python is Google’s standard scripting language and were quick to mention that Google recently picked up Python creator Guido van Rossum. Of course, everyone reading his blog already knows that.
Ruby has a nice buzz going these days and it wasn’t surprising that it made a showing as well. I counted one mashup using it, powered by Ruby-on-Rails, although it was sort of a joke by a PHP 5 developer (the author of TagCloud). I heard that there was another Ruby-based mashup there but to my knowledge it wasn’t demoed. Coming from more of a Python background, I spent quite a bit of time with a couple of the Ruby developers at the conference to see what they liked best about the language and why they’d picked it over Python. I’m not sure that they drew any significant conclusions but they were down-to-earth and really nice guys. One of the Ruby users was also telling me about Nitro and seemed particularly enthusiastic about Og. A company called Rubyred Labs brought some nice grapefruit.
ASP.net had a presence with one mashup using it. As with the Zend PHP Conference, I did get some nice freebies from Microsoft and noticed all the coffee cups said “Microsoft” on them. The coffee wasn’t bad, either. Visual Studio was given out as one of the prizes. I’ve already got my copy and you can download Express online.
Java didn’t make a showing as far as I saw, although Stephen O’Grady noted one that I apparently missed. However, there was an appearance by Johnathan Schwartz who gave away a T2000 server.
PHP was unquestionably the language of choice for developers at MashupCamp. I counted thirteen mashups using it. I made a point to talk to as many PHP developers as I could and nearly everyone had moved to PHP 5, although some were still supporting customers with legacy PHP 4 code. I believe nine out of the thirteen mashups powered by PHP were on PHP 5, with at least one more in transition. Most of the mashups were built within the past few months, or in many cases, the past few weeks. Everyone seemed very excited about PHP 5 in general.
Amongst all of the hi-tech showings at this event, I have to say that the highlight for me was definitely meeting Bob Frankston. Bob, together with Dan Bricklin, developed the world’s first spreadsheet for microcomputers: Visicalc. This might be showing my age but it was truly an honor to meet this tech luminary and chat with him about the first generation of microcomputers, the 6502 microprocessor, Commodore computers, and everything in between. Thanks, Bob. I won’t soon forget it.
All in all, it was a great event and I’m looking forward to the next one.
