2010 is the year for Python 3!

New year, new hopes, new energy! And I’ve already said that 2010 is the year for porting packages/libraries to Python 3, and it seems like I’m right, because the Python 3 buzz is rising. There is more talk about Python 3 on the mailing lists, more blog posts and more packages. And there is even rumors that Google want to merge their Unladen-swallow project to Python 3 trunk.

There is now support in Distribute that will help you, and the amount of packages that support Python 3 is slowly rising. There are branches of several basic packages in the Zope Component Architecture that supports Python 3 already, and I really hope we’ll have a Python 3 compatible version of the basic ZCA libraries out this year.

Fittingly then, I’m going to have a series of articles (we’ll see how many it turns out to be) in Python Magazine about porting to Python 3. I also intend to compile those articles together with additional material into a book that should come out during the second half of 2010.

But why do I say 2010 is the year for porting only if you make packages? Well, because for your project, you will have to wait for all your dependencies to be ported to Python 3 first. And even if you help with that porting, it’s quite likely that most big projects will have a couple of dependencies that doesn’t support Python 3 yet. But I hope that by the end of the year, all of the most popular packages on PyPI will support Python 3, which means that in 2011 it’s definitely time to port even applications to Python 3.

And in 2012 Python 3 will take over and save the world from evil Maya-planets from outer space, or whatever the 2012 disaster has become by that time. :)

New year’s Python meme

(From Tarek)

1. What’s the coolest Python application, framework or library you have discovered in 2009?

BFG. It’s a minimal web framework that still does things the way that feels natural from me. This is because the authors are all ex Zope-programmers and therefore have learned from Zopes mistakes. I hope to be able to do a BFG project also using a lot of Grok technologies in the future. That would be cool, and maybe it would make web development fun again.

2. What new programming technique did you learn in 2009?

Once I’ve learnt something, I feel like I always knew it. I have no idea what I learned in 2009, specifically, except how zope.interfaces metaclass hack works. It’s one of those tricks that once the penny drops you start laughing out loud, because it’s just way too nasty and clever. In short, the implements() method will hack a __metaclass__ variable into the namespace during the execution of the class body, which will cause the class construction to use that metaclass, except that the metaclass’s class constructor will actually return the standard class, but only after setting a bunch of variables on it. I guess that can count as a technique. Or butt ugly hack, depending on your view of such things.

3. What’s the name of the open source project you contributed the most in 2009 ? What did you do?

Well, Distribute came out with a Python 3 port. It was largely done by Martin v Löwis, who went through my changes in my Setuptools port one by one, and made sure they made sense, improved them and added more fixes. So the Python 3 port I did for Setuptools ended up being a decent contribution to Distribute. I also added much of the support for automatically running 2to3 on install. As Setuptools/Distribute is very popular, it has been one of the big stumbling blocks for Python 3 migration, so that was a biggie. And it wasn’t easy either. Complicated project, Setuptools.

4. What was the Python blog or website you read the most in 2009?

Well, I read my own posts several times before posting… oh, that doesn’t count? Hmmm. I’ll have to say “Planet Python” then. I do read it in an RSS reader, of course. I have no idea which specific blog there I read most.

5. What are the three top things you want to learn in 2010?

  1. Understanding the Abstract Syntax Tree, so I can write fixers without doing it mostly by trial and error.
  2. LaTeX, as I’ll plan to self-publish a book about Porting to Python 3.
  3. Everything else.

About Open source, patents and copyright

I’ve made my talk on open source at the Plone Conference in Budapest into three blogposts. I’ve put them on the blog I have about the open society, as they really aren’t about Plone or Python. But as it is about open source many of you will be interested, so here are the links:

By popular demand: Playlist

Several people have asked me for what “that song” I played in my presentation on “Why Open Source Works” at the Plone conference. Usually they mean the song I finished with, sometimes the song I payed before the presentation. But just to be sure, here is the complete play list:

The song played before the presentation was: Darwin DeezRadar Detector

During the presentation I played excerpts from:
Neil Innes – How Sweet to Be an Idiot
Oasis – Whatever
Bobby McFerrin – Don’t Worry, Be Happy
4 Non Blondes – What’s Up
Rolling Stones – The Last Time
The Andrew Oldham Orchestra – The Last Time
The Verve – Bittersweet Symphony

After the presentation I played Baller in it’s entirety.

Radar Detector and Baller are both available at the sixtyone. You don’t
need an invite, but if anyone wants one, mail me and I’ll send you an
invite anyway. We’ll get connected automatically when you join (and I
get points, haha!)

If Open Source business is so elusive, then why do most of us have jobs?

Via Karl Horak I found a very strange article by Ashlee Vance about open source as a business model. It’s clear he doesn’t understand open source, but that’s not the strange thing, the strange thing is the obvious faults and self contradictions. Mentioning that MySQLs sales hasn’t matched the downloads is a strange claim for example. Of course it has. Since the software is open source and free to use, the sales would be only a fraction of the downloads, which is true. But they were still big enough to enable the owners to sell MySQL to Sun Microsystems for 1 billion dollars last year. Would MySQL if it was a closed database ever have reached that value? Of course not. There were already very good SQL databases covering most of the market niches when MySQL was created. If it had been a closed source company, they would never have gotten off the ground. The company only became a billion dollar company because of open source. Exactly how is that an example of open source failing as a business?

The debate over Oracle owning MySQL is also a bit strange. This is in the end a matter of a big company buying up a smaller competitor that serves a slightly different market segment. This is nothing new or strange, and is usually not stopped unless the resulting company gets a monopoly, which is not the case here. That MySQL is available in an open source license is no argument to stop such an acquisition. The open source world isn’t even affected by it, as it can simply continue to use and improve the open source version. They can’t relicense it as a commercial version, meaning that those who use the commercial version is in the hands of Oracle. Just as they were in the hands of Sun before. And before that in the hands of MySQL.

But how strong are those hands? These companies have bought a commercial license of MySQL. I don’t know what that license entails, it might be that they are not allowed to modify the code. In that case, they are in the hands of Oracle in the same way as anyone who uses Windows are in the hands of Microsoft. If they find a bug or want a feature, they have to go to the owner to make that happen. But if they are allowed to modify the code, then they can still do this under the commerical license. The commercial license means only that they are allowed to distribute MySQL as a part of a commercial system, where the open source license doesn’t allow that. That type of license does not stop anyone from innovating and improving MySQL.

The open source community is in no way in the hands of Oracle. Because the Open Source licenced version of MySQL can be modified. The license restriction there means you can’t take MySQL and sell it, and that you must also share your improvements. This allows the open source community to continue to improve MySQL forever.

Simon Crosbys claim that only Red Hat makes “real money” out of open source is also strange. It depends on what you mean with “real money”. I for example use almost exclusively open source software, and I create only open source software. The money I get from that seems real to me. But if you mean that only one company are getting stinking rich, then yes, this is true. But that statement misunderstands what open source is. It’s a statement that has a world view centered only on products. And yes, if you want to make a lot of money, you need to make a product and sell that to a lot of people. With open source, the product is free, so you can’t make a lot of money. That means the open source worlds income is based on consulting and support. But that doesn’t mean people don’t make real money, or that all open source companies are in trouble. That’s complete bullshit. Most open source companies are consulting firms who are getting along quite nicely, thank you, even if the current crisis of course affects them too. Will the owner of these companies ever become multi-billionaires? Of course not, but then that’s perfectly true for most owners of companies that make closed source softwares too. Just as plumbers rarely become millionaires, software developers rarely become millionaires. There is limited place at the top. This is hardly news to anyone, but I think it is surprising news to most company owners that unless you can make millions and retire after an IPO your income is not real.

Most interesting of all though, is Ashlee Vances comment that “Many of the top open-source developers are anything but volunteers tinkering in their spare time”. It may come as a surprise to him to hear that in fact very few open source developers at all are volunteers tinkering in their spare time. The vast majority of open source developers develop open source as a part of their job. Quite often open source software start out as a spare-time project. But if the software is useful you’ll soon have people using it on their job, and spending their payed work hours improving it, because it makes their job easier. Open source developers are generally not some sort of quasi-socialist idealists who work on open source out of the goodness of their hearts. They make their software open source because it makes their software better, and because they like to see other people use and improve their software.

It is now also more and more common that open source software is created by companies because they need it, but they decide to make it open source because it’s good for them to make it open source. Yes, the only way to make money directly from a product is to sell it. But you can make money indirectly, either by support or because you use the software in your job somehow. But to make money that way, your software must be really good, and the more people you have working on it, the better it will be. So by making it open source you ensure that everyone who wants to can help. And they will do so without monetary compensation.

So open source is a viable business model. But it’s only viable if you understand that there are other ways of running a company than to make a commodity, sell it to millions and then make an IPO.

Review: Veda Williams – Plone 3 Theming

I got a review copy of Veda Williams book “Plone 3 Theming“, which I have now finally read. I was positively surprised about the depth and detail of the book, it contains information not only on skinning, but explanations of how the Zope Component Architecture works, and how to override a portlet view. That means the book is useful all the way up until you start actually developing new products yourself. It also means that Packt Publishing has covered the Plone development spectrum, with Practical Plone 3 as an introduction into using and managing Plone, Plone 3 Theming to cover how to start making modifications, and finally Professional Plone Development when it’s time to go hardcore. Practical Plone 3 and Plone 3 theming also works to teach a beginning developer those skills needed to really grasp Professional Plone Development, which is a tersely written book that doesn’t explain much, but relies on code examples to teach you what you need. Efficient, but not easy.

There is a big caveat emptor with Veda Williams Plone 3 Theming, though. Recently in the Plone world, a new technique for theming sites known as Deliverance or XDV has started to get popular. Using Deliverance is a completely different technique and mindset from the old way of theming a Plone site, and although Plone 3 Theming has a chapter on this, it’s very brief. That’s understandable because it’s a new technique where people haven’t yet discovered all the pitfalls and best procedures, but  it does mean that of you are handed a new project today, and need to theme that site, this book may not be what you are looking for.

However, the depth of detail and clear explanations in Veda Villiams books means that it’s going to work well for anybody who have an ongoing Plone 3 project or need to take over an Plone 3 site that is in production and needs to know how to theme that site. I also think it’s a good book for somebody who has read “Practical Plone 3″ and wants to know more of what is going on behind the scenes of Plone, but find Martin Aspelis book too hardcore.

Time for Python 3!

The main problem with Python 3’s lack of backwards compatibility is not that you need to change the code. That’s pretty easy. No, the main problem is that you need to wait for all the libraries you use to be ported first. And when you use a whole framework that can be a lot. Zope + Plone involves some 170 packages or so.

And amongst the absolutely most popular packages out there has been Setuptools. So when I wanted to look into using the Zope Component Architecture on Python 3, I realized I needed Buildout, and to use Buildout I needed Setuptools. And many others, I’m sure, has had the same experience. Setuptools has become a block to Python 3.

But no longer! As mentioned earlier, Distribute now has Python 3 support, and 0.6.3 of Distribute has been out for some weeks now and seems reasonably stable. And that means, that Yes! It is finally time for You to port your library to Python 3!

Install Python 3.1 (forget 3.0, it’s too buggy) and Distribute and  read the documentation on Python 3 support. Then subscribe to the python-porting mailing list, and off you go!

Use key when sorting, not cmp

I just read Jarret Hardies article in Python Magazine on How to Survive Sorting in Python 3. There it is claimed that sort() and sorted() takes both cmp and key as parameters (which is correct) but that cmp is easier to understand and more popular.

But is it easier to understand? cmp requires you to compare two objects. It requires you to first extract a value from the two objects that can be compared, then compare these values and then return -1 if the first object is smaller than the object passed in, 0 if they are equal, and +1 if the first object passed in to the comparison function is larger than the second. Or, if it’s multiple values that should be compared, you typically compare them one by one.

OK, admittedly, that is not very hard to understand. But what about using key? Well, for key you pass in a method that takes only one parameter, the object. The method then extracts a value that is used by the sorting. Done. It honestly doesn’t get simpler or easier to understand than that. It’s also faster, as the sorting methods will call the key function on each object only once.

The reason Python programmers are still using cmp instead of key (and also list.sort() instead of sorted()) is that these didn’t exist before Python 2.4, and therefore people use the old methods out of habit. No other reason. I didn’t even know about key until a year ago when I started looking into Python 3.

There is one usecase where key has one minor issue of “uhm…”, and that is if there is no obvious way of extracting one value. Maybe you should sort fruit so you want it sorted by type of fruit, and then size. So that’s two values, but key requires you to generate just one value. Well, you can simply return a tuple of the values, in the order they should be sorted. So for the fruit case (hehe), you simply do “return fruit.type, fruit.size”. It’s that simple, again.

Jarret also mentions a use case that key doesn’t cover, namely if you want to sort one value ascending and another descending. Well, it is possible to do, by generating some sort of hash that takes this into consideration, but that’s silly. Instead Jarret recommends doing tow consecutive sorts. He also demonstrates this to be faster than a cmp based comparison, so there you go.

Key is easier to implement and faster to run. You should already now use key instead of cmp. And you should buy Python Magazine, it’s getting really good.

(Also, most of the time you should use sorted() instead of sort. Because sorted can sort anything, while list.sort() of course only works on lists. That way people can pass in generators to your methods where you expected lists, and use the dynamicism of Python even more!)

Setuptools on Python 3 Status: Pretty damn good!

At DZUG-Tagungs last weekend it turned out that Martin v Löwis had a bunch of patches for my Python 3 port of setuptools. I merged them in, and Martin then proceeded to look through all my change sets and try to actually make sure they wouldn’t break edge cases (which I never bothered with). And then who ported my changes, change for change, to Distribute, the new fork of Setuptools. He wasn’t the only one working on this during the sprint, but he did the main body of the work, and embarassingly I’ve forgotten the names of the others. Shame on me.

This means, that amazingly enough, Distrubute now has Python 3 support! The next version 0.6.2, hasn’t been released yet, but it works fine for me for Python 3. And that’s really nice! If you need setuptools for Python 3, please try out the trunk version of Distribute. Martin put in support for automatically running 2to3 on code so you can support both Python 2 and Python 3 from the same source code easily, and I put in support so that the test command also does this conversion.

So thanks to Martin and the others! Also thanks to Alex Grönholm who has fixed some bugsin the Python 3 port, and thanks to Tarek for managing the Distribute project!

Some fat presentation tips if you use Linux or Open Office

Recommendation 1: xrandr

At OSCON I had problems with getting my Ubuntu MacBook to talk to the overhead projects. Yes, MacBooks are notorious for this. At the excellent DZUG-Tagungs I had the same troubles, but time to look for a fix. (At PyCon everything worked fine though, so it’s only certain projectors).

Basically, the monitor configuration tool will present me with three screens if I connect the VGA dongle. One being the LCD screen, another the VGA and a third screen seems to be the DVI screen, even though I connect the VGA dongle. Crazy stuff. The problem is that whatever you do with the VGA screen, nothing actually happens. Resolution makes no difference, mirroring makes no difference. Pressing Apply will make the screen blink and you get asked if you want to save the settings, but the external monitor never turns on.

Aftersome serious Googling I found xrandr. This saved my life. Well, OK it didn’t, but it saved the presentation. xrandr does the same thing as the screen tool, but on the command like, and unlike it, it works. Possibly it’s way more stupid, and maybe doens’t ask the monitor if it can handle the resolution or something, it just sets it. Just typing xrandr will tell you what screens you have and the modes that they support.

I’ll give you a couple of handy typical settings. For other stuff, read the manual:

Single-screen presentation setting; Mirror the LCD and the VGA in 1024×768:

xrandr --output LVDS --mode 1024x768 --output VGA --mode 1024x768

Dual screen presentation; Separate screen for the LCD and the VGA, VGA to the right of the LCD:

xrandr --output LVDS --mode 1280x800 --output VGA --mode 1024x768 \
       --right-of LVDS

As you see it’s pretty straight forward to use.

Recommendation 2: Presenter view plugin for Impress

Open Office Impress is not the worlds best presentation software. In many ways it’s pain in the ass, and I did a lightning talk at Pycon using the normal image viewer in fullscreen mode and just pictures I Googled. That was actually quite nice and much easier to do.

But that’s not always enough, and then you do need a presentation software, and for Linux the choices are limited. Now that I have discovered xrandr and know how to get separate screens in a reliable way when I’m at conferenses, I would like to have a separate presenter view, where you see not only the current slide, but the next slidem the notes and the time.

And whaddayounkow, there is one for Impress. Now, stupdly it’s not included, you need to download it as a separate plugin. Silly really. But it seems to work fine, and I’m totally using it for my next presentations. No more forgetting what the next slide is, no more looking at the clock worringly, no more forgetting to say stuff. It’s going to make me a better presenter!

So why did I tell you guys this? I should have kept the secret to myself, right? No, I shouldn’t, and with some luck I’ll tell you why not at the Plone conference!