I see from this history meme that a lot of people are still using prehistoric shells! Low numbers like 303 or even 24 total times for top execution? Now, my shell knows how to store data properly, so for example it doesn’t toss away your entire session if you don’t exit it cleanly, and has scaled so far with near-instant searches for a 20000+ history size.

My code:

import os,sys,sqlite3
c = sqlite3.connect(os.path.expanduser('~/.hotwire/state/history.sqlite'))
cmds = map(lambda x: x[0].split(' ', 1)[0], c.execute('SELECT cmd FROM Commands'))
stats = {}
for cmd in cmds:
count = stats.get(cmd, 0)
stats[cmd] = count+1
cmds = stats.keys()
cmds.sort(lambda a,b: cmp(stats[b],stats[a]))
for cmd in cmds:
print "%s: %s" % (cmd, stats[cmd])

Output:

cd: 2280
ls: 2003
ed: 1823
git: 1362
python: 1187
make: 817
svn: 751
sudo: 669
py: 543
cat: 442
rpm: 380
rm: 370
less: 366
cvs: 306

Mailing lists

Benjamin discovers that yes, pretty much no one except hardcore hackers use mailing lists (my thoughts on them here). I don’t see the disconnect between normal users and developers as “scaling” – it’s not a good thing. I’ll try to look at Fedora Forum more…must train self to ignore the flashing “NEW” icon.

New journal

I spent a little bit of free time a few days ago to make the Firefox Journal extension work again. As mentioned before, you’ll need to create an AMO account to install because it’s still in the “sandbox”.

Now it needs some CSS love for sure. But, it works and I find it pretty useful for a few tasks, such as “go back to that new bug I was looking at 10 minutes ago”.

Speaking of extensions…Password Maker is awesome. I’ve now redone most of my web passwords using it.

Account system

I’ve been thinking lately about how to improve the online.gnome.org account system. If you’re curious, my proposal is linked from here

Languages

• Clojure definitely shows that there is a lot of room left for innovation in programming languages. I’d only briefly heard of transactional memory before; this is something to learn.
• ECMAScript 4 looks like a massive improvement over the old JavaScript. I, for one, will not miss the old bizarre prototype system.

A good post by one of the PHP authors on dynamic vs static languages (though he phrases it as Ruby/Python/PHP vs Java). I think my feelings on this lately are that if I were going to work on a website frontend process, it makes total sense to use Ruby or Python. But there’s a lot more to the world than website frontends. For example, I don’t think I would choose a dynamic language to write Hadoop or HDFS. The dynamic vs static debate is of course a very old one, but what is new is the dramatic progress of dynamic languages on VMs designed for static languages, which I was playing with before.

What I think people in these debates greatly underestimate though is the importance and power of the Free Software ecosystem, which could also be called the “default factor”. /usr/bin/ruby has always been a simple yum/apt-get install away on Free operating systems if it wasn’t already there; no license clickthrough web pages. That matters.

It’s the community, stupid

Speaking of technology though, a lot of us sysadmins, programmers and free software hackers can get easily get caught up in the implementation details of our systems. Now for sure, some things are just technically better than others; for example, the JVM is years ahead of CRuby’s green threads and basic bytecode interpreter. But what ultimately matters is the people behind the technology. It’s easy to get confused because the technology names are staring us in the face; when you want to run your program, you type /usr/bin/ruby, and not /usr/bin/a-language-from-matz-and-and-now-a-global-community.

A key example where I see this happen often is in “packaging systems”; many people confuse the program apt-get, which is just an overgrown wget, with all the hard work that goes into packaging and maintaining software. In the beginning, people had to go through and make sure it worked. For certain types of software, it is actually hard and there’s no getting around it; an example is JNA which deals with some lower-level JVM access. In that case, a lot of the work was done by two Fedora community members. And we even sent the patch for our work upstream, so that all Free operating systems can benefit.

Anyways, that’s enough for a Friday pre-lunch update…

“What it lacks is an extensive screen reader like GNOME’s Orca”: Do you really think our time is best spent spending time reimplementing Orca just so we have something in Qt? Maybe someday someone will, but right now we have better things to do. This kind of mentality of “my toolkit or DIE!” is a disease. It makes us waste more effort and time when it isn’t specifically needed. In this case, just use Orca with KDE4 apps.
— Aaron Segio.

Agreed! — Colin, who is currently using the official Last.fm client written in Qt because, hey it worked, and it’s Free Software.

Very good post I saw while scanning Planet KDE. Which reminds me – besides keeping up with what’s happening in KDE via their planet, other recent additions to my feed list are Stevey’s Blog Rants and the Hadoop blog.

The previous post spawned a lot of discussion, a lot of which was surprisingly technical and on-topic. But after talking with some people I realized that OpenJDK can do a lot just on its own. Here’s my wishlist:

OpenJDK->OpenVM plan

1. Split up source distribution into OpenVM core, place things like Swing into separate source project
2. OS-specific integration in core; e.g. javax.unix namespace (e.g. Unix domain sockets), javax.windows (similar to python-win32), javax.osx; and allow interested operating system vendors to innovate there. The operating system does matter.
3. Commit to longer-term VM improvements necessary to allow compilation of C# into extended JVM bytecode (not CIL)
4. Commit to VM improvements necessary to make Jython/JRuby work well
5. Stay on top of Linux distribution integration, make sure packagers aren’t carrying patches (this includes JSR 277 work)
6. Together with the above, branding as OpenVM or something similar to express willingness to be more than just the old “Java” which was JVM+Java language+Swing

Jonathan, you are leading the development of a Free Software, high-quality, multi-language VM runtime with an extensive class library, called OpenJDK.
Miguel, you are leading the development of a Free Software, high-quality, multi-language VM runtime with an extensive class library, called Mono.

How about a merge? We’ll call the new project “OpenVM”, for convenience in this letter.

Let’s jump right in to the advantages for the projects:

Advantages for Mono

In one word – control. Miguel, your original goal with Mono was to bring a modern and Free Software development stack to GNOME and Linux. In many respects, you and the Mono community have been succeesful, helping spur the creation useful applications for the Free desktop, as well as getting Mono deployed in interesting applications like Second Life. However, you are largely not in control of your destiny. You’re stuck implementing a clone of what Microsoft creates, and besides the fact that cloning something is much less fun for your engineers, you can’t help but be behind.

By helping to create the OpenVM project, you will regain control. In an OpenVM effort, drawing on the common shared work of several corporations (Sun, Novell, Red Hat, Google, and IBM, to name a few), your engineers get to help design the future of Free Software. You will instantly remove all hesitation that the Free Software community has about your work, and have been the a key part of not one but two cornerstone projects for Free Software (GNOME, and Mono->OpenVM).

Advantages for OpenJDK

Jonathan, you have said you want to take the J out of JVM. By stepping up and adding Mono technology like a high quality C# compiler to this OpenVM effort, in the short term you will regain the eroding market share of the JVM on Windows by allowing interoperability between the growing C# code base and existing Java code. In the longer term, developer attraction to OpenVM will let you accelerate improvements to Java, and reverse gains in C# market share.
Moreover, the community agile languages such as Ruby and Python are nearly certain to join an OpenVM effort. Your company will again be at the core of the stack for the vast majority of the computing industry, from the Free Software community to the proprietary applications.

From the Free Software side, turning Mono from a Microsoft technology clone into a part of a truly Free project would eliminate the increasing use of .NET in the community.

Finally, leveraging the Mono team would bring a number of excellent engineers who know the Free desktop very well, having created high-quality bindings for GNOME, and Free applications that many people use.

Advantages for the Free Software community

The Free Software community has long been split between developers using Free and agile languages like Ruby and Python, the the Mono-based community, and a huge community of developers in the world who used in the formerly-proprietary Java in Free projects like Apache. A combined OpenJDK and Mono would dramatically further the merge of all three of these communities, increasing the control the Free Software community has over the stack and reducing duplication of HTTP libraries, database access libraries, etc.

Details

Obviously, there would be many details to work out in such an effort, like how the class libraries could be merged. My intuition is that initially OpenVM would have both JDK and .NET “personalities”. Over time, the Mono .NET class library would be rebased on top of an evolved JDK class library, and eventually the .NET personality could be relegated to a separate “OpenVM-.NET emulation” project as most applications are ported to use the OpenVM JDK-based class library.

But the details are just that – where there’s a will there is a way. So the open question is – who will register the domain name first?

The previous entry was cut slightly short because I had to head out to dinner. What I was trying to finish was a bit of code to experiment with JRuby, Jython, Rhino, and the new JSR 223 scripting framework.

My first step was to build JRuby from trunk and the Jython 2.5 branch, as well as JSR223 trunk. After a small patch, I got some code working:

package org.verbum;

import javax.script.ScriptContext;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
import javax.script.SimpleScriptContext;

public class LangFun {
public static class Hello {
public void sayHello() {
System.out.println("Hello world!");
}
}
public static void main(String[] args) throws ScriptException {
ScriptEngineManager mgr = new ScriptEngineManager();

Hello greeting = new Hello();

greeting.sayHello();

ScriptContext context = new SimpleScriptContext();
context.setAttribute("greeting", greeting, ScriptContext.ENGINE_SCOPE);          

ScriptEngine javaScript = mgr.getEngineByExtension("js");
javaScript.setContext(context);
javaScript.eval("greeting.sayHello()");

ScriptEngine ruby = mgr.getEngineByExtension("rb");
ruby.setContext(context);
ruby.eval("$greeting.sayHello");

ScriptEngine python = mgr.getEngineByExtension("py");
python.setContext(context);
python.eval("greeting.sayHello()");        
}
}

To run this, you’ll need to link to jruby.jar, jython.jar, rhino.jar, as well as the respective engines from JSR223: jruby-engine.jar, jython-engine.jar, js-engine.jar.
The idea behind this code is pretty simple – we first create a Java object, with a single method. Then we use the JSR223 interface to instantiate an engine for each of the languages, hook up a context object which maps the variable greeting into the global namespace for each language, then call their respective eval methods. The result is what you’d expect:

Hello world!
Hello world!
Hello world!
Hello world!

Pretty cool! Of course, I’m not really stressing the system here with these simple scripts; but given the dramatic progress (really, look at that graph!) of projects like JRuby, the future approaching very quickly.

“Ok, but…”, you might say, “what’s this useful for besides sharing libraries?” The general answer is that the single process, shared memory model is fundamentally more powerful than the multi-process, communicating over bytestream pipes model. You can just do more, with fewer hacks. An example is software like Reinteract – before, it could only support Python. Now, not only could you add an input language chooser to Reinteract; you could actually pass the results from a Python computation increasingly seamlessly into Groovy, Java, or whatever. Personally, I specifically want this for Hotwire. Right now we only support Python well, because the project is based on CPython.

Many new languages for the Free Software community

There was a comment on the last entry mentioning Scala. I’ve only looked at it very briefly. But this is just one of many languages that are now truly, finally part of the Free Software community. The Scripting project has a list of the engines written. But that list is far from complete, because some don’t have the engine glue written yet, and because other languages like Scala are more designed as Java replacements that run on the JVM, rather than “scripting”.

And of course through all of this, venerable Java isn’t standing still – it will likely gain closures. And remember – all of this will soon be available (if it isn’t already) via yum install or apt-get install, etc., all of it entirely Free Software, and increasingly integrated with the operating system and your favorite libraries.

A large underwater earthquake ends up creating an effect called a tsunami. The event can be detected by sensors around the world, but the resulting tsunami isn’t immediately visible; if you’re in the surrounding ocean you’ll notice it, but it’s only when it hits land that you really notice the effect.

On May 8, 2007, there was an effect a lot like an underwater earthquake in the software world. What are we talking about? The complete release of OpenJDK, of course. Since that time, we’ve mostly been in the underwater propagation stage. A lot has been happening behind the scenes such as removing proprietary bits, fixing OS integration, etc. But now, I think we’re close to moving into the stage where the ocean recedes, so you can see the first visible effects.

The original OpenJDK release was a snapshot of the in-development version 7, so it was not quite suitable as a drop-in replacement for software that was targeting JDK 6. But recently in February, Sun released the sources to the stable version of the JDK, version 6. Thanks to the combined work of the OpenJDK team and the IcedTea project, this is now suitable to effectively work as a drop-in replacement for the earlier proprietary JDK releases.

On Fedora 8, this command worked for me:

sudo yum --enablerepo=development install java-1.6.0-openjdk

It pulled in new versions of a few things like zlib, but no big deal. In any case I think you’ll likely see OpenJDK 6 pushed as an update for Fedora 8 too.

Why is this so important?

It’s pretty hard to underestimate the transformative impact Java and the JVM have had on the software industry in the last decade or so. Now, Java is a very well designed language, but what I think is equally (if not more) important is the JVM. The JVM was really pretty far ahead of its time; the optimizing JIT compiler, the class structure, the threading model, concurrent generational garbage collection, etc.

There has been quite a lot of innovation on top of that platform. Ok, that sounds a bit buzzword-y. How about this: People have written a lot of awesome software for the JVM.

The examples seem endless; but let’s mention some:

• Processing – a combination language and mini-IDE designed for creating visual effects
• NASA’s World Wind
• Groovy, which is along with Boo in my opinion one of the top contenders for the next-generation of programming languages
• MIT SIMILE projects like Welkin.
• Hibernate – Best-in-class object-relational mapping
• Eclipse
• Hadoop – Free Map/Reduce implementation

Impact ahead

But before OpenJDK, most of these projects effectively did not exist to the core of Free Software community. Even though all of these projects are themselves Free Software, to run it you had to download and install proprietary JDK. I think for most of us, we might as well have been required to download a Windows or OS X VM. At least that’s the way I felt. It wasn’t very integrated with the operating system. But most importantly, it was a blob which we didn’t ultimately control, and we were right to avoid the Java Trap.

But, those were pre-earthquake times. The integration of the formerly-separate Java/JVM world with the Free Software community is ramping up very quickly. For example, Fedora is close to landing all of the dependencies of JRuby. I don’t think anyone has started on things like Processing or World Wind yet – it could be you!

A shared Free Software runtime

I want to talk specifically about efforts like JRuby, and the newly-invigorated Jython. In an earlier blog entry, we looked at the fragmentation in the Free Software community. Every free language has its own runtime and libraries; and until now, building on the JVM wasn’t an option if you wanted many contributors from the Free Software world. OpenJDK is finally changing that. Now, you can write a library using Java, it can be sensibly integrated with Free operating systems like Fedora, and can be consumed by anything on the JVM, which includes Python and Ruby, as well as new languages like Groovy. We’d actually be sharing more than just the OS kernel and C library.

Crucially, this is a platform we now control, because it’s Free Software. And the original upstream is not just giving some standards or source code drops, they are actively helping us. In fact, “they” are “us” now!

Part 2 of this entry

I’ve just uploaded a new version of the Firefox Journal to Firefox Addons. It’s in the “sandbox” right now, which means you need to create an AMO account and enable the sandbox to download it from there.

In this new release, because the awesomebar does search so well, this version of the journal removes the search interface; it’s now focused on helping you get back to sites you recently visited quickly, as well as providing an automatically-generated list of your 5 most-frequently visited web sites.

Here’s an updated screenshot. For more links and background information on the journal, see the home page.