• PyCon 2008 Wrap-Up Mar 20, 2008

    Posted by Mike Naberezny in Python

    At PyCon 2008, I co-presented a talk, Supervisor as a Platform. The talk was well attended and the audience even included Python creator Guido van Rossum. We received a lot of positive feedback about our recent work and I suspect we picked up a good number of new users as well.

    About two months ago, we began a push to build quality documentation for Supervisor using Docbook. After this was complete, we set out to build a new web presence. Our efforts culminated at PyCon 2008 when we unveiled the new Supervisor website.

    PyCon was also very productive, with hacking on Supervisor, Repoze, and general WSGI fun. I worked on a new feature to allow Supervisor’s process group configurations to be reloaded without restarting Supervisor. Meanwhile, Chris made supervisorctl pluggable so extensions can add new commands to the Supervisor command line.

    I also attended quite a few interesting talks, met some new people, enjoyed hanging out with usual the Zope and Plone folks, and overall just had a good time. PyCon 2008 was great conference and I’m looking forward to next year.

  • PyCon 2008: Supervisor as a Platform Mar 16, 2008

    Posted by Mike Naberezny in Python

    We presented Supervisor this week at PyCon 2008. We enjoyed meeting many new users and hearing how Supervisor has helped you. Thanks for your support.

    If you didn’t make it to PyCon, here’s an overview of our talk:

    Supervisor is a tool for managing UNIX processes. Supervisor starts arbitrary processes at its own startup and allows otherwise unprivileged users to start, stop and restart its subprocesses and view its subprocess’ logs using a command line or web interface. Other programs exist which do this, but what sets Supervisor apart is that it is written in Python and built with extension points that can be leveraged by Python developers. In this talk, we’ll look at Supervisor as a platform, and how Python programs written to run under Supervisor can use its unique capabilities.

    You can now download the slides from our talk in PDF format.

  • Supervisor Twiddler 0.2 Feb 17, 2008

    Posted by Mike Naberezny in Python

    Supervisor Twiddler is an RPC extension for Supervisor that allows Supervisor’s configuration and state to be manipulated in ways that are not normally possible at runtime.

    Changes in release 0.2:

    Renamed addProcessToGroup() to addProgramToGroup(). The method
    now supports program definitions with numprocs > 1 and will add
    all resulting processes from the program definition. It also
    fixes a bug where the process config was not added to the group
    properly. Requested by Roger Hoover.

    Added new method log() that writes an arbitrary message to the
    main supervisord log. This is useful for recording information
    about your twiddling.

    You can download the new version from PyPI.

  • Supervisor at PyCon 2008 Dec 21, 2007

    Posted by Mike Naberezny in Python

    I’ll be presenting at PyCon 2008:

    Supervisor is a tool for managing UNIX processes. Supervisor starts arbitrary processes at its own startup and allows otherwise unprivileged users to start, stop and restart its subprocesses and view its subprocess’ logs using a command line or web interface. Other programs exist which do this, but what sets Supervisor apart is that it is written in Python and built with extension points that can be leveraged by Python developers. In this talk, we’ll look at Supervisor as a platform, and how Python programs written to run under Supervisor can use its unique capabilities.

    Our talk will cover basic usage but will focus on the more advanced features we’ve added recently, such as adding custom RPC interfaces and event listeners.

    PyCon 2008 will in Chicago on March 13-16th. From the list of talks, it looks like this is going to be a great conference. I am especially looking forward to pounding out some new code during the sprints following the conference on the 17-20th.

  • Supervisor Extensions Released Nov 5, 2007

    Posted by Mike Naberezny in Python

    Supervisor is a process management system written in Python.

    One of the major features we added to Supervisor recently is the ability to extend its XML-RPC interface functionality by registering new top-level namespace factories.

    Today, I released two new extension packages for Supervisor:

    • Supervisor Twiddler allows Supervisor’s configuration and state to be manipulated in ways that are not normally possible at runtime.
    • Supervisor Cache provides the ability to cache arbitrary data in the Supervisor instance as key/value pairs.

    Both of these packages add capabilities to Supervisor through this new extensibility. Since they are the first packages available of their kind, they also serve as a template for building your own Supervisor extensions.

  • Repoze: Retooling Zope Nov 1, 2007

    Posted by Mike Naberezny in Python

    During the last couple of weeks, I have been privileged to help out with Repoze, a new open source Python project. Repoze is “plumbing Zope into the WSGI pipeline” but also much more.

    I’m not a Zope developer but I’ve benefited from Zope indirectly in several ways. The most significant for me has been Supervisor, a process management system. It’s not part of Zope itself but it originated from a need to keep production Zope servers running smoothly. I started using it for other kinds of servers, and fast forward a couple of years and now I’m very involved and help develop it.

    Supervisor is interesting to many Zope outsiders. It might have grown out of the needs of running Zope servers, but it’s a tool that many others can benefit from using. I think parts of Zope itself also have similar potential for outsiders.

    Zope has been around for a long time and for many was thought to be the “killer app” for Python. Philip J. Eby wrote about the significance of Zope in Where Zope Leads, Python Follows. I’m interested in Zope not because of the application server as a whole but because it has many technologies like an object database, transactions, and object publishing that I’d like to explore.

    It seems that quite a few technologies that are gaining mindshare now through new developments have been available and stable in Zope for many years. As an outsider, I want to be able to consume bits of Zope technologies for other kinds of projects.

    The problem that Zope has is that its technologies aren’t accessible to outsiders in the same way as Supervisor. To the rest of us, the Zope world is largely a mysterious monolith. Projects like Grok make it easier to get started by bringing Rails-like conventions to Zope development, but many of the core Zope technologies remain largely inaccessible for new purposes without putting in a lot of effort.

    Repoze is changing that in a significant way. Repoze is packaging Zope technologies into components and middleware that can be easily deployed on WSGI servers. This is a win for Zope applications because it means better deployment options like Apache and mod_wsgi. Breaking up the monolith and providing more options will help the longevity of their platform.

    For the rest of us, a more modular world of Zope means that eventually, many of the interesting technologies that Zope has to offer will be closer in reach and more practical through a combination of WSGI middleware and better packaging.

    Repoze already seems to have great momentum and I’ll be watching it with interest. I look forward to exploring what the Zope world has to offer through Repoze.

    Update: This post was featured on Repoze Notes.

  • New in Horde: Routes Sep 15, 2007

    Posted by Mike Naberezny in PHP,Python

    I’m pleased to announce the first release of Horde/Routes, a new URL mapping system for PHP 5. This package provides classes for mapping URLs into the controllers and actions of an MVC system, inspired by Ruby on Rails.

    There are already quite a few existing libraries that do this sort of thing for PHP. Horde/Routes is a compelling alternative.

    I examined most of these PHP solutions and found them all inadequate for various reasons, particularly because we wanted RESTful routing, named routes, sophisticated matching, PHP 5 E_STRICT, and extensive test coverage.

    Since I do quite a bit of Ruby and Python programming, I surveyed the options in those languages and decided to do a full port of the Python library, Routes.

    Horde/Routes provides these features and more:

    • Supports route recognition and generation
    • Sophisticated matching conditions like subdomains
    • Named routes and RESTful route generation
    • PEAR-style naming and coding standards
    • PHP 5 E_STRICT compliant, web framework agnostic
    • A comprehensive unit test suite

    I decided to contribute the code to Horde’s Rampage project because Horde is one of PHP’s oldest and most successful projects. I’m using and contributing to other Horde libraries and I think Rampage is worth your attention.

    The Python version has been around for some time and is very popular with different Python web frameworks. Horde/Routes is now part of that ecosystem now and has already resulted in patches being committed back to the Python version.

    I’ve already used Horde_Routes on several applications. While Horde/Routes is relatively new, it is very feature-rich and well-tested.

    Currently, Horde/Routes is a beta release. Over the coming weeks, we’ll be making some minor changes to the API and adding more documentation, and then it will quickly move to stable.

    Update: The project now has its own pages on the Horde website and was featured on Chuck Hagenbuch’s blog.

  • MashupCamp Feb 23, 2006

    Posted by Mike Naberezny in PHP,Python,Ruby

    Earlier this week, I spent two days at MashupCamp. This great event was held at the Computer History Museum in Mountain View, California.

    Mashups are an interesting topic. “Mashup” is one of those terms like “Web 2.0” that’s trendy and really hard to define in any concrete terms. Basically, mashups are taking two or more web service APIs (or data somehow gleamed from the web) and then building a useful application by combining them.

    The caliber of developers that attended was top notch and this could easily be seen by the great apps that were demonstrated. These were the kind of folks that know XML inside and out, know everything annoying about SOAP headers, know XHTML strict better than HTML 4, and can tell you the method signatures of the services they use from memory. Every developer that I talked with had experience with at least two scripting languages, usually more. It was a great crowd with some very smart and talented people in attendance.

    Mashups themselves tend to get put together very quickly. Sometimes people get an idea and implement a mashup in a few hours or a few days. It was great to be around so many developers who really grok the newest web technologies and know the fastest ways to develop applications to take advantage of them. So, what were they using to get things done fast? Besides the mashups themselves, that’s what I was interested in learning about.

    It turns out these developers were really pragmatic. They use what works and gets their job done the fastest without getting in the way — even if that means Perl. I don’t mean Catalyst or any other kind of fancy frameworkish thing, I mean straight Perl CGI. If you read the blogs these days, you’d think Perl is almost dead for web applications. I counted four mashups using Perl in some way. The most common application was backend processing but I saw one using it on the frontend as well.

    Three mashups that I saw were using Python. One of them was built on the Twisted asynchronous framework. I’ve done some work with Medusa but haven’t worked on a project needing all the power of Twisted yet. It certainly looks impressive. Another Python mashup was the excellent chicagocrime.org website, built on the Django framework by its co-creator Adrian Holovaty. A lot of people I talked with at MashupCamp were interested in Python, especially at the Googleplex where I attended an afterparty (thanks, Adam!). Several Googlers told me Python is Google’s standard scripting language and were quick to mention that Google recently picked up Python creator Guido van Rossum. Of course, everyone reading his blog already knows that.

    Ruby has a nice buzz going these days and it wasn’t surprising that it made a showing as well. I counted one mashup using it, powered by Ruby-on-Rails, although it was sort of a joke by a PHP 5 developer (the author of TagCloud). I heard that there was another Ruby-based mashup there but to my knowledge it wasn’t demoed. Coming from more of a Python background, I spent quite a bit of time with a couple of the Ruby developers at the conference to see what they liked best about the language and why they’d picked it over Python. I’m not sure that they drew any significant conclusions but they were down-to-earth and really nice guys. One of the Ruby users was also telling me about Nitro and seemed particularly enthusiastic about Og. A company called Rubyred Labs brought some nice grapefruit.

    ASP.net had a presence with one mashup using it. As with the Zend PHP Conference, I did get some nice freebies from Microsoft and noticed all the coffee cups said “Microsoft” on them. The coffee wasn’t bad, either. Visual Studio was given out as one of the prizes. I’ve already got my copy and you can download Express online.

    Java didn’t make a showing as far as I saw, although Stephen O’Grady noted one that I apparently missed. However, there was an appearance by Johnathan Schwartz who gave away a T2000 server.

    PHP was unquestionably the language of choice for developers at MashupCamp. I counted thirteen mashups using it. I made a point to talk to as many PHP developers as I could and nearly everyone had moved to PHP 5, although some were still supporting customers with legacy PHP 4 code. I believe nine out of the thirteen mashups powered by PHP were on PHP 5, with at least one more in transition. Most of the mashups were built within the past few months, or in many cases, the past few weeks. Everyone seemed very excited about PHP 5 in general.

    Amongst all of the hi-tech showings at this event, I have to say that the highlight for me was definitely meeting Bob Frankston. Bob, together with Dan Bricklin, developed the world’s first spreadsheet for microcomputers: Visicalc. This might be showing my age but it was truly an honor to meet this tech luminary and chat with him about the first generation of microcomputers, the 6502 microprocessor, Commodore computers, and everything in between. Thanks, Bob. I won’t soon forget it.

    All in all, it was a great event and I’m looking forward to the next one.

  • Python Module “xmlrpclib” Apr 27, 2005

    Posted by Mike Naberezny in Python

    Whether you are a fan of Python or not, Python’s bundled module xmlrpclib from PythonWare is extremely useful for testing an XML-RPC server when used from the Python interactive shell. Since xmlrpclib is bundled with the standard Python 2.3 distribution, no additional installation is required. Python itself comes preinstalled on most Linux distributions and Mac OS X. Start the Python shell, and enter these commands:

    import xmlrpclib
server = xmlrpclib.Server('http://your/server/url')

    Now, you can interactively test the methods of your XML-RPC server from the Python shell. As an example, most XML-RPC server libraries include support for the system.listMethods() and system.methodSignature() methods. To list all of the available methods on the remote server from Python, simply enter:

    server.system.listMethods()

    The methods will be returned as a Python list and printed to stdout. Calling XML-RPC methods really couldn’t be any simpler. To get the signature of a method, enter:

    server.system.methodSignature('method_name')

    The conversion of data types between Python and XML-RPC is automatically performed by the Server object. You can test any of your remote server’s methods as shown above. The Server object also provides a verbose mode that will display the raw XML-RPC transactions with the server for troubleshooting. To enable verbose mode, instantiate the Server like this:

    server = xmlrpclib.Server('http://your/server/url', verbose=True)

    Interactively testing your server API like this can greatly reduce your debugging time. It is also very handy for getting to know your way around the APIs of other XML-RPC servers, such as WordPress or Flickr.

  • Python Module “re” Apr 15, 2005

    Posted by Mike Naberezny in Python

    Without question, one of my favorite bundled Python modules is re, a PCRE (Perl Compatible Regular Expression) engine.

    Python’s string object has many useful methods that I use frequently such as split(), strip(), replace(), and especially find(). When more sophisticated operations are needed, import re.

    A good deal of the work I do involves interpreting short messages sent from instrumentation. For most of these communications, I need to quickly evaluate a small quantity of data in three ways:

    1. Validate that the response is formatted as expected.
    2. Extract specific data from the response.
    3. Validate and then act on the data extracted.

    re‘s object-oriented interface is very nice, however I find that it can overcomplicate very small operations where a simple function like PHP’s preg_match or preg_match_all would be entirely sufficient. Luckily, identical functionality is easily recreated with re by cascading the re objects on one line and using the findall() method of the pattern object to return a list:

    import re
testString = 'Light Switch = ON'
matches = re.compile('Light Switch = (\w+)').findall(testString)
if matches == []:
    # If an empty list is returned, the string under test
    # was nothing like we expected.
elif matches[0] not in ('ON', 'OFF'):
    # The string under test was formatted as expected,
    # however the extracted data is unusual.
else:
    # Success.  The string under test was formatted as
    # expected, and the expected data was extracted from it.