An excellent update from one of the Jython hackers (and followup). I particularly liked this:

>>> from __future__ import GIL
Traceback (most recent call last):
(no code object) at line 0
File "", line 0
SyntaxError: Never going to happen!

For me the most useful thing is that Jython SVN now supports decorators. I use decorators fairly extensively in my Python code, and they were the main thing blocking me from using Jython more widely.

Some people may wonder – why Jython? One reason is that the tooling around the JVM is awesome. JDWP (and clients like Eclipse‘s debugger) blow away gdb. For a while now I’ve been trying to debug a periodic lockup in a multithreaded Python process, and it’s been immensely painful with gdb. There are a lot of other extremely useful FOSS tools for heap analysis, profiling, etc. Even if your entire world currently runs on Python, and you aren’t interested in easily taking advantage of some of the other awesome JVM software out there like Hadoop, running on Jython makes sense alone just for OpenJDK being among the best Free VMs out there.

I liked this blog entry from Planet KDE. That entry reminded me to talk about one way I like to think about how to improve the user experience with software – and that is by what I call the “forehead mashing” method.

This is where as you’re developing software, write out a few problems that you’re trying to solve. Then, for every step the user is required to make – for every dialog that pops up, for every few seconds of keystrokes, every dialog to click, every checkbox to check…lower your head onto the desk and hit your forehead.

You don’t have to do this with a lot of force. Just enough so that you feel it.

This calls for several head-desk hits

One of the top things that will quickly make your forehead sore are Wizards/Assistants. The way I think of these is that the programmer is so proud of their work and they think it’s so awesome and complicated that they want you to acknowledge it.

What if programmers inflicted these kinds of things on ourselves? Say in gcc.

If you all keep making wizards, I’ll create this patch for GCC

It’s not just wizards of course that are the source of pointless user time spent reading text and clicking buttons; random dialogs are another source. Often with some thought, it’s really not hard to reduce and simplify.

This is davidz’s theme, don’t blame me

So I promised in the comments in my previous post that I would do something to reduce the pain from the notification bubble I added; to keep the universe in balance as it were. So I’m happy to say that in Fedora rawhide now, the above dialog is no more. It was just a bad idea, and you will never see it again.

One of the things I’ve been looking at again recently is the initial experience for the desktop. Previously we investigated reducing the number of steps by starting the browser by default; incidentally I have plan for a more refined version of that kicking around that isn’t quite ready but should hopefully be soon.

In the context of the Online Desktop work we want to get the user logged in online. However, this requires a network connection. Pretend for a moment you’re new to the system. Look at a default desktop, say one from the Phoronix Fedora 9 review, without the obligatory window to 1970 in the middle.

How do you know what to do to get online? If you’re using a wired connection then you don’t have to know anything; NetworkManager does it automatically. However a much more common case now is wireless networks, and you would have to know to click on the “two computer and broken X” icon in the top right. A simple solution is to have a notification pointing to it:

How to connect to a wireless network

Cooking up the patch for that, I quickly ran into the problem though of notifications pointing to the wrong place. If you’ve used GNOME for long enough you probably know what I mean; on login sometimes getting a notification in the top left instead of the top right, or it could just be off by 10 pixels. This usually wasn’t too bad because the notification was more about the content, but in this case a notification pointing to the wrong thing was actually worse than nothing.

Fixing this right ended up taking me though a number of layers in the stack, from the application displaying the notification, to libnotify, to GtkStatusIcon, to the notification protocol, and finally to the notification daemon, which needed a fairly big patch. However after spending a bit of time this weekend I finally got it to work:

Link to Theora video

Not earth shattering, but it has been a pretty longstanding bug in the desktop. It was actually sort of fun to get back to grubbing around with the low level of Xlib.

As you may have gathered from previous entries, I’ve recently become interested in programming languages again.
I’m almost done (though skimming some parts) with my copy of the aforementioned Ruby book. and though I haven’t done any substantial Ruby programming (just playing around in jirb while reading), I think I have a good idea now why so many people love the language.

Closures

Most of my real code over the last number of years has been in C, Java, and Python, and I know those languages and their runtime libraries pretty well, but reading the Ruby book I was struck by how really useful closures can be in an Algol-family language (i.e. not Lisp). Well, Ruby calls them “blocks” and has infrastructure on top in the form of yield, etc., but that’s fundamentally what they are. C/Java/Python all lack them (no, Python’s single-line lambdas are too restrictive to count).
Closures…environment, get it?

Closures are incredibly powerful, in fact you might say they’re the ultimate language construct. Neal Gafter has a good description of the kinds of things you can do with them, going from a language that doesn’t currently have them.

Polyglot programming

While I was thinking about this blog entry, I reread one of Steve Yegge’s great blog posts, and decided to look up more about his reference to the author of a design pattern book “leaving Java to go to Ruby”. After going to Martin’s home page, I found on his wiki he has a good entry which is pretty close to what I wanted to talk about. That is basically: it makes sense for large software systems to have multiple layers in different languages.

Now, if you’re thinking “Wow, that’s obvious”, that’s good; but there is more to the story here. So let’s look at some rationales. If your app is large enough, you probably have parts which need to be fast. And you probably have other parts which cry out for a domain-specific language.

So for speed, you’ll want a lower layer which is usually characterized by manifest typing and direct vtable function dispatch. Read: C++/Java/C#.

But it makes sense often to have a higher layer which is agile. It is better for the parts of your program which change rapidly – this could be user interface bits you’re prototyping, or rapidly creating test cases. This layer is usually characterized by implicit typing (possibly with type inference), metaprogramming capabilities, and (ideally) good integration with the lower level language. Read: Groovy,JavaScript,Python,Ruby

There is a lot of software out there split in exactly this way; in fact, you’re almost certainly reading this blog entry in one of them, Firefox, where the answer is C++ and JavaScript. A lot of computer games are built in this way too – for Civilization 4, the answer is C++ and Python, and for World of Warcraft it’s C++ and Lua. If you’re familiar with Java, just think about JSP and Ant – they’re really DSLs. If you mostly know Python or Ruby, think about how much of the underlying platform is actually written in C/Java/.NET.

So it’s fairly easy to dismiss anyone who says something like “everything must be written in language X”, for values of X like C,Ruby,Python,Java. Which reminds me to say: Eclipse really needs to embrace Eclipse Monkey.

The impedance mismatch

So we accept that it makes sense to have multiple languages with different characteristics. One important issue then becomes – how similar are our two different layers? Taking the example of C++ and Python as in Civilization 4. The gap is enormous. C++ containers are not the same as Python containers. C++ strings are not the same as Python strings. C++ objects and Python objects are wildly different. The answer to this problem is to create a special glue layer; in Mozilla, it’s called XPCOM. In GNOME, it’s called pygobject. These layers are very painful to create and maintain.

An interesting question is – what if our two languages shared more? Do we really need to have separate container types just to get agility and dynamism? The answer turns out to be – no, which we’ll get to in a minute. As we know, the fact that there are a lot of things that every modern language shares lower level components (like garbage collection, JIT compilation) led Microsoft to brand .NET as a multi-language runtime (as an aside, plenty of languages ran on the JVM long before .NET was created; for example Kawa, which dates to 1996). Now here’s the thing, though. Running on .NET does not make Python objects same as .NET objects, nor does it make their containers the same.

What is an object?

Let’s briefly take a look at what a Python object is. It’s a fairly illustrative example of just how different languages can be. In Python, every object instance is by default a dictionary (hash table), with data stored in the __dict__ member. Every property lookup or method call has to in general traverse a chain of hash table lookups. At any point, some other code can come along and add a new entry in an object’s dictionary:

#!/usr/bin/python

class Test(object):
def __init__(self, a):
self.a = a

t = Test("hello")
t.b = 42
print t.b

Supporting this level of dynamism is expensive, both in time and space, again because every object instance carries along a mutable hash table under the covers. It means you can’t share very much between processes. It makes multi-threading much slower because everything has to be synchronized on that dictionary. Besides being expensive, it’s almost never what you actually want, at least by default. You usually want t.b to be an error. This is by far my biggest issue with Python. In fairness to Python, it predates almost every other language discussed here.

Stealing and language evolution

Languages are clearly stealing things from each other, and evolving together. In the Ruby book they often mention how certain parts were taken from other languages. Java and C# are stealing ideas from each other. ECMAScript 4 is clearly rebuilding itself on a more JVM/.NET like class model.

What I’ve been looking at lately is a new dynamic language that has clearly stolen a lot of the good ideas from Ruby and Python, but is a lot more “native” to a modern runtime (in this case, the JVM): Groovy.

#!/usr/bin/env groovy

class Test {
String a
}

def t = new Test(a: "hello")
t.b = 42
println b

This results in an exception about a missing b, because its idea of a class is exactly the same as the underlying JVM platform, where objects are much more static by default (this is also true of .NET). Note we can even declare types if we like (or we can just use def). I really like how default constructors work – it’s even less typing than both Python and Ruby! It has useful closures, regular expression and hash table literals. Pretty cool. For a more complex example, here’s an example of a fairly typical scripting task of log file processing I wrote a few days ago. I’m fairly sold so far, but there is still more to learn. I spent a bit of spare time poking at getting it packaged, but ran into some Maven bootstrapping issues.

More on languages

One random link: An awesome feature of Python is Generators, and if you aren’t familiar with them and think of yourself as a “systems programmer”, check out this very good slide set.

Second to last: some recent additions to my Google Reader feed: Charles Oliver Nutter, John Rose, Lambda the Ultimate.

As an aside that’s not directly language related, but also new to my feed list is Why, who is like a great artist-programmer churning out amazing works like Shoes. Does anyone else have the feeling that for Why all of these code projects are just what he does in his idle time, and in the next few years he’ll emerge from his underground hideout with an army of giant robots and take over the earth?

As I’ve mentioned before, in my spare time I’ve been working on a fun project to replace the Unix terminal and shell with native graphics and a high-level language runtime. One thing that I quickly found out though was that certain applications require a real VT100 terminal emulator; for example, less is a highly optimized program, and while replacing it with a proper X11 app would be possible, it would not be trivial and would largely be a distraction from my goals anyways. Another situation where you need a terminal emulator is connecting to remote Unix hosts; while I have complete control over the software on my laptop, it’s hard to change all the existing servers out there.

SSH in particular is something that really cries out for a nice application around it, rather than launching it primitively from an existing terminal emulator. Earlier I mentioned the result of a few hours of PyGTK hacking. Over the last week or two I’ve been working on improving it. Today David Ascher of Thunderbird fame asked for it, so I should probably kick it out the door (though HotSSH is primarily for X11/Unix and not the Mac).

New connection with history

As you can see, the new version of HotSSH now remembers what you do, automatically. For example:

• If you connect to a host and use a custom username, the next time you open HotSSH and type the host again, it defaults to that custom username, without you having to do anything
• It has pretty good completion on the Host entry, based both on ~/.ssh/known_hosts as well as its own history
• Open windows and tabs are saved automatically, Firefox style

The only dependencies are CPython, GTK+, DBus, and OpenSSH. I’ve broken it out of where it previously lived in the Hotwire tarball into a separate source tarball. You can currently get it from the Hotwire downloads page, though my long term plan is to import it into GNOME SVN and ideally have it come with GNOME out of the box. The code is currently here, and bug reports here. It works well enough for me now; about all it might make sense to add is some sort of color scheme (we currently just pull from the gnome-terminal defaults). To run, you can just unpack the tar and python ui/hotwire-ssh. If you install it, it should create a default bash alias for ssh, but this is something that package distributors will need to make sure integrates well.

So you’re writing an application. At some point, you have a problem where you want to be able to talk to an instance of your program while it’s running. It turns out, this is fiendishly difficult for Free Unix-based desktops, and many programs get it wrong. On my Fedora 9 system, looking in /tmp, I see listings like gconfd-walters (GConf), hsperfdata_walters (OpenJDK), pulse-walters (Pulseaudio), emacs500 (GNU Emacs). All of these programs are broken. The root cause of this is the failure of Unix to provide a standard per-user, per-machine temporary directory. If we had that, things would be dramatically simpler, but we don’t.

You can’t use /tmp because it’s not per-user. Any time you rely on a well-known name in /tmp (like the above list), your application is subject to denial-of-service attacks if another user creates that file before you log in. Security aside, it’s also just harder to deal with because you have to check for whether or not the data in your temporary directory is stale or not (is something listening on that socket, or is it left over from a session crash?), and that’s yet more painful code.

You can’t use $HOME because it’s not per-machine (think of all the sites that subject users to the pain of NFS), and won’t always support Unix domain socket files.

Now, there are other solutions; you can use the X server, which is what Firefox does I believe. If your application is single-instance this isn’t too bad, but there is an easier way:

Enter solution: DBus

The best solution is to use DBus. The key feature here is that the DBus session bus provides exactly what we want: a per-user, per-machine namespace, as well as a fairly sane IPC system we can use to pass messages. Here’s how you use it.

• Decide on a service name for your application. For the purposes of this discussion, we’ll call it org.openssh.Ssh, for no particular reason.
• In your application startup, try to acquire this name on the session bus, using DBus. If you’re successful, your application can now provide a service. To pick a completely random example, this service might be multiplexing SSH connections. If something else already owns the service, then you can act as a client; you might send the message org.openssh.GetControlMasterSocketPath to return the path to another Unix-domain socket, because your application has a non-DBus protocol for communication.

In this example, we just used DBus to retrieve the path to another local, randomly-named Unix domain socket. That’s perfectly fine – DBus is not the one true IPC system. Programs like Pulseaudio and OpenSSH already have defined protocols, and there is nothing wrong with that, we don’t need to replace them with DBus. However, DBus is the best candidate for a one true atomic session namespace.

Again, all examples used in this blog post are completely hypothetical, and have no relation to the silliness of requiring users to try to hack around this issue on their own computers with variables like ControlMaster and ControlPath to enable a useful feature of your application.

I’m going to learn Ruby. My copy of The Ruby Programming Language arrives today.

I see from this history meme that a lot of people are still using prehistoric shells! Low numbers like 303 or even 24 total times for top execution? Now, my shell knows how to store data properly, so for example it doesn’t toss away your entire session if you don’t exit it cleanly, and has scaled so far with near-instant searches for a 20000+ history size.

My code:

import os,sys,sqlite3
c = sqlite3.connect(os.path.expanduser('~/.hotwire/state/history.sqlite'))
cmds = map(lambda x: x[0].split(' ', 1)[0], c.execute('SELECT cmd FROM Commands'))
stats = {}
for cmd in cmds:
count = stats.get(cmd, 0)
stats[cmd] = count+1
cmds = stats.keys()
cmds.sort(lambda a,b: cmp(stats[b],stats[a]))
for cmd in cmds:
print "%s: %s" % (cmd, stats[cmd])

Output:

cd: 2280
ls: 2003
ed: 1823
git: 1362
python: 1187
make: 817
svn: 751
sudo: 669
py: 543
cat: 442
rpm: 380
rm: 370
less: 366
cvs: 306

Mailing lists

Benjamin discovers that yes, pretty much no one except hardcore hackers use mailing lists (my thoughts on them here). I don’t see the disconnect between normal users and developers as “scaling” – it’s not a good thing. I’ll try to look at Fedora Forum more…must train self to ignore the flashing “NEW” icon.

New journal

I spent a little bit of free time a few days ago to make the Firefox Journal extension work again. As mentioned before, you’ll need to create an AMO account to install because it’s still in the “sandbox”.

Now it needs some CSS love for sure. But, it works and I find it pretty useful for a few tasks, such as “go back to that new bug I was looking at 10 minutes ago”.

Speaking of extensions…Password Maker is awesome. I’ve now redone most of my web passwords using it.

Account system

I’ve been thinking lately about how to improve the online.gnome.org account system. If you’re curious, my proposal is linked from here

Languages

• Clojure definitely shows that there is a lot of room left for innovation in programming languages. I’d only briefly heard of transactional memory before; this is something to learn.
• ECMAScript 4 looks like a massive improvement over the old JavaScript. I, for one, will not miss the old bizarre prototype system.

A good post by one of the PHP authors on dynamic vs static languages (though he phrases it as Ruby/Python/PHP vs Java). I think my feelings on this lately are that if I were going to work on a website frontend process, it makes total sense to use Ruby or Python. But there’s a lot more to the world than website frontends. For example, I don’t think I would choose a dynamic language to write Hadoop or HDFS. The dynamic vs static debate is of course a very old one, but what is new is the dramatic progress of dynamic languages on VMs designed for static languages, which I was playing with before.

What I think people in these debates greatly underestimate though is the importance and power of the Free Software ecosystem, which could also be called the “default factor”. /usr/bin/ruby has always been a simple yum/apt-get install away on Free operating systems if it wasn’t already there; no license clickthrough web pages. That matters.

It’s the community, stupid

Speaking of technology though, a lot of us sysadmins, programmers and free software hackers can get easily get caught up in the implementation details of our systems. Now for sure, some things are just technically better than others; for example, the JVM is years ahead of CRuby’s green threads and basic bytecode interpreter. But what ultimately matters is the people behind the technology. It’s easy to get confused because the technology names are staring us in the face; when you want to run your program, you type /usr/bin/ruby, and not /usr/bin/a-language-from-matz-and-and-now-a-global-community.

A key example where I see this happen often is in “packaging systems”; many people confuse the program apt-get, which is just an overgrown wget, with all the hard work that goes into packaging and maintaining software. In the beginning, people had to go through and make sure it worked. For certain types of software, it is actually hard and there’s no getting around it; an example is JNA which deals with some lower-level JVM access. In that case, a lot of the work was done by two Fedora community members. And we even sent the patch for our work upstream, so that all Free operating systems can benefit.

Anyways, that’s enough for a Friday pre-lunch update…