As I’ve mentioned before, in my spare time I’ve been working on a fun project to replace the Unix terminal and shell with native graphics and a high-level language runtime. One thing that I quickly found out though was that certain applications require a real VT100 terminal emulator; for example, less is a highly optimized program, and while replacing it with a proper X11 app would be possible, it would not be trivial and would largely be a distraction from my goals anyways. Another situation where you need a terminal emulator is connecting to remote Unix hosts; while I have complete control over the software on my laptop, it’s hard to change all the existing servers out there.

SSH in particular is something that really cries out for a nice application around it, rather than launching it primitively from an existing terminal emulator. Earlier I mentioned the result of a few hours of PyGTK hacking. Over the last week or two I’ve been working on improving it. Today David Ascher of Thunderbird fame asked for it, so I should probably kick it out the door (though HotSSH is primarily for X11/Unix and not the Mac).

New connection with history

As you can see, the new version of HotSSH now remembers what you do, automatically. For example:

• If you connect to a host and use a custom username, the next time you open HotSSH and type the host again, it defaults to that custom username, without you having to do anything
• It has pretty good completion on the Host entry, based both on ~/.ssh/known_hosts as well as its own history
• Open windows and tabs are saved automatically, Firefox style

The only dependencies are CPython, GTK+, DBus, and OpenSSH. I’ve broken it out of where it previously lived in the Hotwire tarball into a separate source tarball. You can currently get it from the Hotwire downloads page, though my long term plan is to import it into GNOME SVN and ideally have it come with GNOME out of the box. The code is currently here, and bug reports here. It works well enough for me now; about all it might make sense to add is some sort of color scheme (we currently just pull from the gnome-terminal defaults). To run, you can just unpack the tar and python ui/hotwire-ssh. If you install it, it should create a default bash alias for ssh, but this is something that package distributors will need to make sure integrates well.

So you’re writing an application. At some point, you have a problem where you want to be able to talk to an instance of your program while it’s running. It turns out, this is fiendishly difficult for Free Unix-based desktops, and many programs get it wrong. On my Fedora 9 system, looking in /tmp, I see listings like gconfd-walters (GConf), hsperfdata_walters (OpenJDK), pulse-walters (Pulseaudio), emacs500 (GNU Emacs). All of these programs are broken. The root cause of this is the failure of Unix to provide a standard per-user, per-machine temporary directory. If we had that, things would be dramatically simpler, but we don’t.

You can’t use /tmp because it’s not per-user. Any time you rely on a well-known name in /tmp (like the above list), your application is subject to denial-of-service attacks if another user creates that file before you log in. Security aside, it’s also just harder to deal with because you have to check for whether or not the data in your temporary directory is stale or not (is something listening on that socket, or is it left over from a session crash?), and that’s yet more painful code.

You can’t use $HOME because it’s not per-machine (think of all the sites that subject users to the pain of NFS), and won’t always support Unix domain socket files.

Now, there are other solutions; you can use the X server, which is what Firefox does I believe. If your application is single-instance this isn’t too bad, but there is an easier way:

Enter solution: DBus

The best solution is to use DBus. The key feature here is that the DBus session bus provides exactly what we want: a per-user, per-machine namespace, as well as a fairly sane IPC system we can use to pass messages. Here’s how you use it.

• Decide on a service name for your application. For the purposes of this discussion, we’ll call it org.openssh.Ssh, for no particular reason.
• In your application startup, try to acquire this name on the session bus, using DBus. If you’re successful, your application can now provide a service. To pick a completely random example, this service might be multiplexing SSH connections. If something else already owns the service, then you can act as a client; you might send the message org.openssh.GetControlMasterSocketPath to return the path to another Unix-domain socket, because your application has a non-DBus protocol for communication.

In this example, we just used DBus to retrieve the path to another local, randomly-named Unix domain socket. That’s perfectly fine – DBus is not the one true IPC system. Programs like Pulseaudio and OpenSSH already have defined protocols, and there is nothing wrong with that, we don’t need to replace them with DBus. However, DBus is the best candidate for a one true atomic session namespace.

Again, all examples used in this blog post are completely hypothetical, and have no relation to the silliness of requiring users to try to hack around this issue on their own computers with variables like ControlMaster and ControlPath to enable a useful feature of your application.

I’m going to learn Ruby. My copy of The Ruby Programming Language arrives today.

I see from this history meme that a lot of people are still using prehistoric shells! Low numbers like 303 or even 24 total times for top execution? Now, my shell knows how to store data properly, so for example it doesn’t toss away your entire session if you don’t exit it cleanly, and has scaled so far with near-instant searches for a 20000+ history size.

My code:

import os,sys,sqlite3
c = sqlite3.connect(os.path.expanduser('~/.hotwire/state/history.sqlite'))
cmds = map(lambda x: x[0].split(' ', 1)[0], c.execute('SELECT cmd FROM Commands'))
stats = {}
for cmd in cmds:
count = stats.get(cmd, 0)
stats[cmd] = count+1
cmds = stats.keys()
cmds.sort(lambda a,b: cmp(stats[b],stats[a]))
for cmd in cmds:
print "%s: %s" % (cmd, stats[cmd])

Output:

cd: 2280
ls: 2003
ed: 1823
git: 1362
python: 1187
make: 817
svn: 751
sudo: 669
py: 543
cat: 442
rpm: 380
rm: 370
less: 366
cvs: 306

Mailing lists

Benjamin discovers that yes, pretty much no one except hardcore hackers use mailing lists (my thoughts on them here). I don’t see the disconnect between normal users and developers as “scaling” – it’s not a good thing. I’ll try to look at Fedora Forum more…must train self to ignore the flashing “NEW” icon.

New journal

I spent a little bit of free time a few days ago to make the Firefox Journal extension work again. As mentioned before, you’ll need to create an AMO account to install because it’s still in the “sandbox”.

Now it needs some CSS love for sure. But, it works and I find it pretty useful for a few tasks, such as “go back to that new bug I was looking at 10 minutes ago”.

Speaking of extensions…Password Maker is awesome. I’ve now redone most of my web passwords using it.

Account system

I’ve been thinking lately about how to improve the online.gnome.org account system. If you’re curious, my proposal is linked from here

Languages

• Clojure definitely shows that there is a lot of room left for innovation in programming languages. I’d only briefly heard of transactional memory before; this is something to learn.
• ECMAScript 4 looks like a massive improvement over the old JavaScript. I, for one, will not miss the old bizarre prototype system.

A good post by one of the PHP authors on dynamic vs static languages (though he phrases it as Ruby/Python/PHP vs Java). I think my feelings on this lately are that if I were going to work on a website frontend process, it makes total sense to use Ruby or Python. But there’s a lot more to the world than website frontends. For example, I don’t think I would choose a dynamic language to write Hadoop or HDFS. The dynamic vs static debate is of course a very old one, but what is new is the dramatic progress of dynamic languages on VMs designed for static languages, which I was playing with before.

What I think people in these debates greatly underestimate though is the importance and power of the Free Software ecosystem, which could also be called the “default factor”. /usr/bin/ruby has always been a simple yum/apt-get install away on Free operating systems if it wasn’t already there; no license clickthrough web pages. That matters.

It’s the community, stupid

Speaking of technology though, a lot of us sysadmins, programmers and free software hackers can get easily get caught up in the implementation details of our systems. Now for sure, some things are just technically better than others; for example, the JVM is years ahead of CRuby’s green threads and basic bytecode interpreter. But what ultimately matters is the people behind the technology. It’s easy to get confused because the technology names are staring us in the face; when you want to run your program, you type /usr/bin/ruby, and not /usr/bin/a-language-from-matz-and-and-now-a-global-community.

A key example where I see this happen often is in “packaging systems”; many people confuse the program apt-get, which is just an overgrown wget, with all the hard work that goes into packaging and maintaining software. In the beginning, people had to go through and make sure it worked. For certain types of software, it is actually hard and there’s no getting around it; an example is JNA which deals with some lower-level JVM access. In that case, a lot of the work was done by two Fedora community members. And we even sent the patch for our work upstream, so that all Free operating systems can benefit.

Anyways, that’s enough for a Friday pre-lunch update…

“What it lacks is an extensive screen reader like GNOME’s Orca”: Do you really think our time is best spent spending time reimplementing Orca just so we have something in Qt? Maybe someday someone will, but right now we have better things to do. This kind of mentality of “my toolkit or DIE!” is a disease. It makes us waste more effort and time when it isn’t specifically needed. In this case, just use Orca with KDE4 apps.
— Aaron Segio.

Agreed! — Colin, who is currently using the official Last.fm client written in Qt because, hey it worked, and it’s Free Software.

Very good post I saw while scanning Planet KDE. Which reminds me – besides keeping up with what’s happening in KDE via their planet, other recent additions to my feed list are Stevey’s Blog Rants and the Hadoop blog.

The previous post spawned a lot of discussion, a lot of which was surprisingly technical and on-topic. But after talking with some people I realized that OpenJDK can do a lot just on its own. Here’s my wishlist:

OpenJDK->OpenVM plan

1. Split up source distribution into OpenVM core, place things like Swing into separate source project
2. OS-specific integration in core; e.g. javax.unix namespace (e.g. Unix domain sockets), javax.windows (similar to python-win32), javax.osx; and allow interested operating system vendors to innovate there. The operating system does matter.
3. Commit to longer-term VM improvements necessary to allow compilation of C# into extended JVM bytecode (not CIL)
4. Commit to VM improvements necessary to make Jython/JRuby work well
5. Stay on top of Linux distribution integration, make sure packagers aren’t carrying patches (this includes JSR 277 work)
6. Together with the above, branding as OpenVM or something similar to express willingness to be more than just the old “Java” which was JVM+Java language+Swing

Jonathan, you are leading the development of a Free Software, high-quality, multi-language VM runtime with an extensive class library, called OpenJDK.
Miguel, you are leading the development of a Free Software, high-quality, multi-language VM runtime with an extensive class library, called Mono.

How about a merge? We’ll call the new project “OpenVM”, for convenience in this letter.

Let’s jump right in to the advantages for the projects:

Advantages for Mono

In one word – control. Miguel, your original goal with Mono was to bring a modern and Free Software development stack to GNOME and Linux. In many respects, you and the Mono community have been succeesful, helping spur the creation useful applications for the Free desktop, as well as getting Mono deployed in interesting applications like Second Life. However, you are largely not in control of your destiny. You’re stuck implementing a clone of what Microsoft creates, and besides the fact that cloning something is much less fun for your engineers, you can’t help but be behind.

By helping to create the OpenVM project, you will regain control. In an OpenVM effort, drawing on the common shared work of several corporations (Sun, Novell, Red Hat, Google, and IBM, to name a few), your engineers get to help design the future of Free Software. You will instantly remove all hesitation that the Free Software community has about your work, and have been the a key part of not one but two cornerstone projects for Free Software (GNOME, and Mono->OpenVM).

Advantages for OpenJDK

Jonathan, you have said you want to take the J out of JVM. By stepping up and adding Mono technology like a high quality C# compiler to this OpenVM effort, in the short term you will regain the eroding market share of the JVM on Windows by allowing interoperability between the growing C# code base and existing Java code. In the longer term, developer attraction to OpenVM will let you accelerate improvements to Java, and reverse gains in C# market share.
Moreover, the community agile languages such as Ruby and Python are nearly certain to join an OpenVM effort. Your company will again be at the core of the stack for the vast majority of the computing industry, from the Free Software community to the proprietary applications.

From the Free Software side, turning Mono from a Microsoft technology clone into a part of a truly Free project would eliminate the increasing use of .NET in the community.

Finally, leveraging the Mono team would bring a number of excellent engineers who know the Free desktop very well, having created high-quality bindings for GNOME, and Free applications that many people use.

Advantages for the Free Software community

The Free Software community has long been split between developers using Free and agile languages like Ruby and Python, the the Mono-based community, and a huge community of developers in the world who used in the formerly-proprietary Java in Free projects like Apache. A combined OpenJDK and Mono would dramatically further the merge of all three of these communities, increasing the control the Free Software community has over the stack and reducing duplication of HTTP libraries, database access libraries, etc.

Details

Obviously, there would be many details to work out in such an effort, like how the class libraries could be merged. My intuition is that initially OpenVM would have both JDK and .NET “personalities”. Over time, the Mono .NET class library would be rebased on top of an evolved JDK class library, and eventually the .NET personality could be relegated to a separate “OpenVM-.NET emulation” project as most applications are ported to use the OpenVM JDK-based class library.

But the details are just that – where there’s a will there is a way. So the open question is – who will register the domain name first?