Neil’s point-free blog

Alex Blewitt posted a pair of articles on his personal blog and on the InfoQ news site, about the “death” of JSR 294. It turns out that the obituary may have been premature, with the other Alex (Buckley, spec lead of the JSR) clarifying that the JSR was tagged as “inactive” merely for process reasons within the JCP, and that Sun had not in fact abandoned it (yet).

Nevertheless, even if not actually dead yet, the buzzards are circling. I for one will not mourn it, if and when it does die. Here, I explain why.

JSR 294 aims to add declarations to the Java Language Specification (JLS) to support modularity. The idea is for runtime modularity to be provided by OSGi and/or Jigsaw, and these module systems would be able to take advantage of declarations in Java source files to work out things like the contents of a module, its version and its dependencies on other modules. Ideally the declarations would also enable interoperability, so that (for example) you could use a Jigsaw module in an OSGi runtime or an OSGi module in a Jigsaw runtime. Then library authors would not have to compile, test and build separate deployment artefacts for each module system. Unfortunately JSR 294 in its current state (PDF warning) falls short of both these goals.

Let’s take a look at an example; here is how a module declaration might look if we are using Jigsaw (this example provided by Alex Buckley):

Rather than explain immediately what this means, we will dissect its anatomy and see how else we might have written it. There are three parts: the new “module” keyword; the module identity which consists of the string “M1 @ 1.0“; and the directives section which is everything between the braces.

Here’s another example using a hypothetical module system called MentalModules. The following will be legal Java code in Java 7, if JSR 294 is passed in its current proposed form:

My apologies about the highlighting… my blog’s syntax highlighter couldn’t cope with such nonsense; neither will the one in your IDE. Anyway, how can this rubbish be valid Java? In fact the second example means just as much to the JLS as the first one does: nothing at all. Aside from some very light syntax rules (e.g., no commas) the contents of the directives section are entirely unspecified by the JLS. They are a little like annotations, except they do not need to be imported; instead, they must be understood by plug-ins into the Java compiler. This means there will need to be a compiler plug-in for Jigsaw and one for OSGi, and one for MentalModules too, if anybody feels like writing one. Library authors will have to decide early which module system to compile for, and under this system it will be hard to support multiple module systems.

Now take a look at the version strings. Alex’s examples look sensible enough — “2.0″ is a useful version string — but the ones in the second example don’t look so nice. In case you hadn’t spotted them, here again are all the legal versions used in the example:

• you_call_this_a_version?? (including the question marks)
• @
• π
• Believe_It_Or_Not_This_Is_a_Version_String_Also

Under JSR 294 versions are nothing but strings, again with some extremely light syntax rules. This supports Jigsaw’s anything-goes approach to versioning, and some commenters have found such liberal versioning attractive since it can accommodate, unmodified, all of the crazy versioning schemes used in the wild. But the problem is this makes it extremely hard for the importers/requirers of a module to depend on a range of versions rather than on a point version, so every tiny bugfix release of a module will invalidate all the modules that depend on it. There is not even a consistent way to calculate which of two versions is “higher”.

Let’s get back to the more sensible-looking example provided by Alex. Unfortunately even this declaration is as useful to OSGi as my deliberately nonsensical one. While the require directive could correspond to OSGi’s Require-Bundle, the provides and permits directives are meaningless to OSGi. Likewise any OSGi-specific directives we might think of to use with JSR 294 will probably be useless to Jigsaw.

So JSR 294 does not help to enable interoperability of module systems. It does not define any shared semantics for module versioning or dependencies. It merely provides a dustbin into which module systems can dump all their arbitrary and proprietary directives. For these reasons, I find myself unable to muster a tear at the thought of its demise.

Well… maybe not ban them exactly. Yes, the title of this post is deliberately provocative and overstates my argument, for which I apologise. Nevertheless it’s true that, in many cases, activators would be better written as Declarative Services (DS) components.

Compare the following two code blocks. First an activator:

Next a DS component:

Notice the difference? The DS one doesn’t depend on OSGi APIs at all, so it’s easier to test and it’s reusable in non-OSGi containers. It can receive configuration data easily via a Map instance, although we can omit that parameter if we don’t need it. Also whereas a bundle can have only one activator, it can have as many active components as we like, which saves having to multiplex several loosely-related concerns into a single class.

Now, you may have heard that DS requires you to write an XML file that is pretty ugly. Never mind, let’s just get our build tool to generate it from our source code. Bnd can do that, here’s an example… let’s say we also want to change the names of the lifecycle methods because the default names (activate and deactivate) are too boring:

Something else we often need to do from an activator is invoke a service. This is quite hard because we don’t know if the service will be available when our bundle starts. The solution to this when writing activators is to implement a ServiceListener or ServiceTracker so we can be called back later when the service we want becomes available. Unfortunately it all ends up in very complex programmatic listener code. With DS we can do this declaratively by saying we need at least one instance of the service, so please don’t activate us until such a service instance is available. Here’s the code for that:

Couldn’t get much easier, could it? We can even add an optional attribute to that @Reference annotation to say that we would like to use the service if it is available, but we’ll go ahead and continue working without it if not.

Finally a common task that activators do is publish a service. Although that’s not particularly difficult to do programmatically, in DS it’s even simpler:

Now this is where DS gets really cool. When the bundle containing the above component is activated, DS will register the service for us in the service registry but will not actually instantiate the class. Only when a consumer bundle comes along and really uses the service does DS create the implementation object. Thus we can create large numbers of components as services essentially for free, and only the ones that are actually used will be loaded. In contrast, when we use activators we generally must eagerly create an implementation backing a service before we know whether any consumers even want to use the service… so we may end up creating many service objects that are never used, and this wastes memory and CPU time.

So most of the time we can get by without bundle activators. When do we still need to use them? Generally if we want to do something like writing an extender bundle — which manipulates the state and behaviour of other bundles — we need access to the OSGi APIs as provided by a bundle activator. We cannot write extenders as POJOs, but we can use them to enable POJO-based programming in other bundles. In other words, extenders are “glue” code, not business logic. For the majority of your application, stick to DS components and POJOs.

On the 1st and 2nd December I will be delivering a 2-day OSGi training course near Amsterdam. This will be a pretty intense couple of days with lots of hands-on practice, which should really get you up to speed with the essentials of OSGi. Best of all, it is priced very competitively at just €900 in total.

The hosts for the course are Industrial TSI, a great Netherlands-based open source company who are very active in the OSGi and Eclipse communities. Wim Jongman has even offered an evening tour of Amsterdam for delegates! But if you prefer to spend the evening discussing whiteboard patterns and extender models over a few drinks in a quiet pub, then that would be cool too.

Click here for more details and registration. Note that the course is part of the Eclipse training series but my approach is to keep strictly to standard OSGi concepts that may be applied across all the OSGi frameworks. You will need to be a competent Java developer, as there will be no time for hand-holding, and all my exercises have optional additional challenges for the high-flyers; see how many of them can you crack…

One of the great advantages of the Declarative Services (DS) approach to OSGi development is laziness. If you listened to the podcast interview with Chris Aniszczyk on EclipseZone recently, you might think that laziness in DS is a special, new-fangled feature in Equinox 3.5 that requires funky bundle activation states. Frankly it sounds complex and scary. But in fact DS laziness can be quite simple and clean if we eschew a little bit of Eclipse/PDE baggage.

First let’s just explain what we’re talking about here: OSGi services are used to decouple the producers of functionality from consumers. For example consider a Translator interface. A “producer” may offer an instance of this interface to translate English strings into Klingon, by publishing it as a service. A “consumer” needs to know how to say “revenge is a dish best served cold” in Klingon, so it queries the service registry to find an instance of the Translator interface and then invokes the translate method. This model is wonderful because the producer knows nothing about the consumer and the consumer knows nothing about the producer: they are perfectly decoupled. But in one way it is too decoupled: the producer doesn’t know whether any consumers even exist!

Suppose that setting up a full English to Klingon translation service takes some time and memory; we wouldn’t want to do it unless we were sure that at least one consumer exists. At the very least we should defer creating it until an interested consumer comes along, so that we don’t unnecessarily delay the startup of our application. Imagine if our application was constructed out of hundreds of services, each of which has a non-negligible cost to create, but only a fraction of which may actually be used by a real consumer.

Typically though, services themselves are not very expensive at all. Most are simple stateless Java objects with a very low marginal instantiation cost. But in the traditional, eager approach to service registration we must at least create a classloader for the bundle in order to load the service implementation class and instantiate the service object, and in large systems the cost of all those classloaders may be quite high.

Enter DS. If we declare our Translator service in an XML file, then the DS runtime bundle is able to register an entry in the service registry before actually creating the implementation class for the service. Only when a consumer comes along wishing to use the service does the implementation class actually get loaded. Handily, this all happens completely transparently to both the consumer and the producer.

However there’s a little wrinkle which, at first glance, makes this seem rather pointless: DS will ignore our bundle and all the lovely XML declarations it contains unless it is in the ACTIVE state! This is very important, by the way, because ACTIVE state gives us explicit control (contrast with the Extension Registry in Eclipse which looks at RESOLVED state, and there is no way to explicitly move a bundle in and out of RESOLVED state). However, activating the bundle means running its activator, which means loading the bundle and creating a classloader and all the other stuff that we just said was expensive. This is the point where some people get confused and start looking to the fancy-schmancy “lazy activation policy” stuff introduced in OSGi R4.2. But wait a minute… what if the bundle has no activator?

The key point here is that bundles can still be active even without an activator class. This may seem silly; what is the purpose in activating a bundle if it has no activator? Well in that case, the ACTIVE state is merely a flag that tells DS to look at the bundle. Now activation is free because there is no need for the framework to create a classloader, as there is no activator class to load.

Therefore, the simplest approach by far to developing services with DS and benefiting from laziness is to build bundles without activators. As far as I know, this works with all OSGi R4.x implementations.

So what’s all the lazy activation policy stuff in OSGi R4.2 about? Well I believe this is because Eclipse wants to have its cake and eat it. The problem is most legacy Eclipse plug-ins do have activators, and furthermore the “New Plug-in” wizard in PDE offers to generate an activator class for you by default. All these activators barely do anything useful and they get in the way of using DS lazily. Eclipse wanted to have activators and delayed classloader creation, so the OSGi spec was enhanced (i.e., made more complicated) in R4.2 to allow bundles to be “lazily started”. This means they enter a kind of limbo where they hang in the STARTING state — a state that normally only exists briefly during the execution of the activator. In this state the bundle classloader has not yet been created and the activator has not yet been executed, but DS can see the XML service declarations and can register the services. As soon as a consumer requests the service, then the classloader is created, the activator is executed and the service instantiated. Yikes, bundle lifecycle is difficult enough to explain already!

There are some nice enhancements in DS in OSGi 4.2 though. The most noticeable is that it is now much easier to write DS components as POJOs, because the activate and deactivate methods are no longer required to take a ComponentContext parameter, and they can therefore be written without any dependency on the OSGi APIs. Also they can receive configuration data from the Configuration Admin service via a Map parameter, and they can be made dependent on configuration data being available.

Sadly, the laziness of DS has its limits, no matter whether we use the no-activator approach or the lazy activation approach. In a DS component we declare references to other services; for example the Translator service may refer to a Dictionary service to look-up the meanings of individual words. DS will see this declaration and supply the Dictionary service to our Translator service implementation via the bind method, e.g. a setDictionary method. This is enough to count as “using” the Dictionary service, which will cause the implementation class for the Dictionary to be loaded and so on, even if our Translator doesn’t really need to use the Dictionary until much later on. There is a way around this problem: instead of asking DS to supply the Dictionary object, we can ask it to supply a ServiceReference object, which we can use later to get the Dictionary when we really need it from the ComponentContext. Unfortunately this pollutes our code with OSGi API calls: our components are no longer POJOs. The only OSGi component model that I know of that avoids this problem is iPOJO, which in this scenario would inject a proxy Dictionary object into the Translator wait until the translator actually accesses the dictionary field before fetching the dictionary service object (thanks to Clement for correcting me on how this works).

Anyway, I enjoyed the interview with Chris, and the DS component editor introduced in Galileo should be very handy — if only it worked outside of PDE projects!

Are you interested in Domain Specific Languages (DSLs), Scala, finance or JavaFX? Are you in or near London?

On Monday 29th June, SkillsMatter will be hosting the third Eclipse DemoCamp to be held in London. DemoCamps are fun, informal events for Eclipse developers and users to talk about what they are working on, with a strong focus on working demos and real life applications. This time we have a number of very exciting speakers:

• Heiko Behrens will be demoing Xtext. This is an amazing tool for building DSLs — it allows you to define a simple grammar in EBNF notation, and from this it will generate a parser, Abstract Syntax Tree (AST) meta-model, and a fully-featured Eclipse editor with auto-completion, syntax highlighting, content outline, hyperlink navigation.

• Miles Sabin will show his work on the Scala IDE for Eclipse. Many people (including me) believe that Scala will be the next big language for the JVM, and it’s important to have tooling for it that is just as good as the tooling available for Java. Miles is doing a great job in making that a reality.

• Richard Gomes will be talking about JQuantLib, an open source library for quantitative financial calculations in Java. Richard has been working on integrating JQuantLib with OSGi and (hopefully!) contributing it to the Eclipse Finance Platform project, and he will talk about the benefits and challenges involved.

• Dan Pollitt will demo NatTable, a high performance SWT data grid control that can be used to display huge quantities of rapidly changing data, and is used by a number of banks for trading applications.

• I will demo the JavaFX plug-in for Eclipse, which I have been working on for Sun Microsystems.

Full details and registration instructions are on the Eclipse Wiki site. Hope to see you there!

For the last month and a half I have been almost completely incommunicado, only emerging occasionally to ask strange questions on the #eclipse IRC channel. I have been working on a project that was due to be announced at JavaOne, so now JavaOne is here I can finally talk about what I’ve been doing!

In late April, I was asked (through a friend of a friend) if I could help Sun Microsystems to improve their JavaFX tooling for Eclipse. Sun has had an Eclipse plug-in for JavaFX development for a quite a while at Project Kenai, their forge site, but it only had basic editing and launching capabilities, and was nowhere near as usable as the JavaFX tooling in NetBeans. It lacked many of the features that developers demand from their tools these days (we are a demanding bunch!). I believe this was simply a problem of a lack of resources.

To my great delight, I was given a free rein to enhance the plug-in however I deemed necessary. Unfortunately Rome was not built in a day, and fully-featured language IDEs cannot be built in 6 weeks. So I had to choose where to focus my efforts, and I have mostly chosen to focus on the raw usability of the code editor. To this end, I have added auto-indentation and brace insertion, auto-completion, continuous feedback of errors, block folding, and integration with the Eclipse Outline view. You should also see pop-ups of the JavaFX API documentation, both during completion and when hovering the mouse over an identifier.

Aside from the editor, an important feature in this release is debugging. It’s an interesting experience to debug JavaFX Script code, because it is a much more declarative language than Java so the program counter tends to skip around a lot. Nevertheless one can write imperative blocks, and debugging into these could be invaluable.

We also have a large number of templates supplied with the plug-in, to kick-start your JavaFX development in Eclipse. Also I’m hoping soon to provide an import wizard for NetBeans projects so you can easily import any of the other samples and templates provided on the JavaFX site.

There is much more work to do, and we are hoping to attract community involvement and contributions to improve the plug-in. So how can you contribute? This leads to the other part of the announcement: Sun intends to contribute this work as an official Eclipse project, and we working together now on a project proposal for submission to the EMO. Look out for the proposal soon when it hits eclipse.org.

In the meantime, you can download the plug-in binaries from javafx.com. They are provided through a simple update site which you can add to your existing Ganymede installation (and probably Galileo too, but this has not been fully tested yet). Please visit the Getting Started page for installation instructions.

Incidentally, NetBeans users may be concerned that this project signals some kind of shift within Sun away from NetBeans. Nothing could be further from the truth — as far as I can tell, Sun’s commitment to NetBeans is just as strong as it ever was. The decision to invest in Eclipse tooling for JavaFX is a pragmatic one: JavaFX should be accessible to all developers, no matter what their preferred IDE. I would also like to thank the NetBeans developers at Sun who provided me with a lot of help and also built a very sound foundation for features that I have built on top.

This Monday at EclipseCon I delivered a tutorial along with my colleague Heiko Seeberger on component models used in OSGi service programming. Our aim was to explain and compare three of the more popular component modes: Declarative Services (DS) , Spring Dynamic Modules (DM), and Apache iPOJO. The slides follow at the bottom of the post as an embedded SlideShare presentation. The example code is available on GitHub at http://github.com/njbartlett/eclipsecon09_components/. Inside the project you can also find the slides in PDF format. Both the slides and code are released under the Eclipse Public Licence.

This week at EclipseCon I have had the pleasure of meeting Alex Buckley, Sun’s Java language supremo and spec lead for JSR 294. For those of you who do not memorise JSR numbers, 294 was originally known as “superpackages”, was killed off for a while, and is now back in the form of enhancements to the Java language to support modularity. As an OSGi supporter I have no problem in principle with JSR 294, indeed it may make the use of OSGi easier, especially for beginners learning the Java language.

However, authors of libraries and applications should not have to choose which of two module systems to support when they release their software to the world. Another Sun project named Jigsaw aims to build an alternative module system to OSGi, one which will be built into OpenJDK and available for library and application developers to use. For a large number of Java developers, the convenience of using Jigsaw is likely to trump any arguments about the technical superiority of OSGi, as OSGi must be downloaded separately. Mark Reinhold announced in his blog that Jigsaw will be fully supported by Sun, although it is not an official Java standard backed by a JSR.

Time for some commercial reality. Yes, Sun is the “mothership” (as Alex puts it) for Java right now, but Sun is unlikely still to exist in its current form by this time next year. Whether it is eaten whole by IBM or torn in half and shared between HP and Oracle, or any of the other rumoured outcomes, a promise from Sun to support Jigsaw is completely worthless. This is especially true because Jigsaw is in direct opposition to the commercial needs of all the potential acquirers. If IBM gets Java, it will kill Jigsaw, I guarantee it. Ditto Oracle. By choosing to build Jigsaw outside of a JSR and in opposition to an established industry standard, Sun has exposed its customers to substantial commercial risk.

I don’t like making the argument against Jigsaw this way. I would much rather debate the relative technical merits versus OSGi, but I feel I have already done so and the OSGi community continues to do so. However Sun intends to use its commercial power as the owners of the predominant Java distribution to force through Jigsaw irrespective of its technical merits, therefore I believe it is not inappropriate to counter with commercial arguments in addition to technical ones.

So as a Java developer interested in modularity, you now have a choice. Do you choose OSGi, an industry standard supported by an alliance of over 30 companies, two JSR numbers and over ten years’ experience? Or do you choose Jigsaw, a new proprietary platform supported by one vendor on the brink of acquisition/bankruptcy? Your job may depend on the answer.

On the 14th April I will begin teaching the first in a series of OSGi training courses, in association with SkillsMatter in London. This is a really in-depth course that explores the fundamentals of building modular applications with OSGi. Then after getting to grips with modularity we will tackle component-oriented development with OSGi’s dynamic services.

In my opinion OSGi is going to be extremely important for the future of the Java platform, and this course will teach a highly pragmatic approach to adopting it. This includes a review of build tools, testing techniques, and how to use OSGi in “legacy” applications.

The first run of the course is heavily discounted — if you book early you can get 50% off the full price. If you can’t make the April run then subsequent runs will happen in June and September, and for those not in the UK there are plans to teach the course in several other European cities this year also.

Please see the SkillsMatter course page for full details and booking information.

S3Install is a very simple runtime OSGi bundle for provisioning an Amazon EC2 node. It is modelled on the FileInstall bundle by Peter Kriens, but instead of monitoring a local directory it monitors an S3 “bucket”. So, you can drop a bundle JAR into S3 and the OSGi runtime will install and run it. Drop in an updated version and the runtime will update to that version on the fly. Delete the JAR from S3 and the corresponding bundle will be uninstalled from the runtime.

Although simplistic, this approach should scale very well. Whether you have one EC2 node or thousands, you can roll out new and updated application code simply by uploading a file with your favourite S3 client. Assuming all the nodes are running an OSGi framework plus S3Install, they will automatically update themselves.

This is very much “Poor Man’s Provisioning” at the moment. The Rolls Royce of provisioning is probably Newton from Paremus, but I think it makes sense to start with a simple approach and add complexity only when needed.

The code along with some instructions are at GitHub, please try it out and let me know if it works for you and what features could be added. Features I’m already planning to add include support for Configuration Admin (like FileInstall), and support for E4 JavaScript bundles.