Release engineer vs dependency management: Basing RCP Product and headless build from plugins to features

Blog
1

A year and a half after the first draft of Bonita Studio, we decided to move our product build from plugins to features, in order to better handle internationalization, and with the hope to better factorize the build of our 2 products: BOS and BOS-SP (which is a set of extensions on top of BOS).

Our build still uses the highly customized headless PDE-build wrapped in a master script responsible for packaging and testing the application as we deliver it. Here are the steps we followed:

Create the feature from the product definition

This was not a trivial task, since there is no tool to transform a product definition to a feature. The solution we chose was to use Run Configuration to create the feature. This is an intermediate step that is useful to turn a product definition into a feature:
  • Right-click on your *.product, and Run as Eclipse application
  • The Run Configuration matching the definition of your .product (and then the set of plugins that it contains) is created
  • Create a new feature, and use the magic Create from Run Configuration button
  • Remove the launchers (org.eclipse.equinox.launcher*) from this feature
This method is inspired from blog posts written by Manuel Selva.

Redefine your *.product

From there, you can redefine your .product to include only 2 features: the one you just created, and the org.eclipse.rcp feature.

Quickly validate it

You can make a first validation of your product. The steps are easy:
  • Create a launch configuration for your product by clicking on the Run As Eclipse application or the Synchronize entry

    Launch product from product editor

  • Open the launch configuration wizard, and click on the validate button

    The validate button will give you some hints to fix deps

  • It tells you about some static missing dependencies. Correct your feature according to its advices.
  • Be patient and keep retrying when you get errors. I think I needed more than a dozen iterations before getting the happy button telling me all is well.

Add the feature to your map

This step is only mandatory if you have a headless build that uses map files. If you usually build using UI, just skip it.

Happy HEADLESS Halloween !

Everything is in the title: you just created a new feature, and you’ll need it at build time. Simply add an entry for the feature to your map:

[cc]
!** Features
feature@org.bonitasoft.studio=SVN,url=http://svn.bonitasoft.org/,tag=bonita-studio,path=XXX_TAG_XXX/releng/org.bonitasoft.studio-feature
[/cc]

Try, see and troubleshoot

Once you've done all that, try to build your product. Using the UI entry should be enough.

The export product wizard

Fixing startup

After that, you can give a first try to your bundled product. Run it. It may fail to start. In this case, analyze the startup log that you will find in YourProduct/config/[timestamp].log. You'll probably see some lines such as these: [cc] !ENTRY org.eclipse.osgi 2 0 2010-10-21 09:17:50.299 !MESSAGE One or more bundles are not resolved because the following root constraints are not resolved: !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.299 !MESSAGE Bundle reference:file:plugins/org.eclipse.jdt.junit.runtime_3.4.200.v20100526-0800.jar was not resolved. !SUBENTRY 2 org.eclipse.jdt.junit.runtime 2 0 2010-10-21 09:17:50.299 !MESSAGE Missing required bundle org.junit_3.8.2. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.299 !MESSAGE Bundle reference:file:plugins/javax.servlet.jsp_2.0.0.v200806031607.jar was not resolved. !SUBENTRY 2 javax.servlet.jsp 2 0 2010-10-21 09:17:50.299 !MESSAGE Missing imported package javax.servlet.http_2.4.0. !SUBENTRY 2 javax.servlet.jsp 2 0 2010-10-21 09:17:50.299 !MESSAGE Missing imported package javax.servlet.resources_2.4.0. !SUBENTRY 2 javax.servlet.jsp 2 0 2010-10-21 09:17:50.299 !MESSAGE Missing imported package javax.servlet_2.4.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.299 !MESSAGE Bundle reference:file:plugins/org.bonitasoft.studio.common_1.0.0.20101021-0701.jar was not resolved. !SUBENTRY 2 org.bonitasoft.studio.common 2 0 2010-10-21 09:17:50.299 !MESSAGE Missing required bundle org.codehaus.groovy_1.7.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.299 !MESSAGE Bundle reference:file:plugins/org.mortbay.jetty.util_6.1.23.v201004211559.jar was not resolved. !SUBENTRY 2 org.mortbay.jetty.util 2 0 2010-10-21 09:17:50.299 !MESSAGE Missing imported package javax.servlet.http_0.0.0. !SUBENTRY 2 org.mortbay.jetty.util 2 0 2010-10-21 09:17:50.299 !MESSAGE Missing imported package javax.servlet_0.0.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.299 !MESSAGE Bundle reference:file:plugins/org.bonitasoft.studio.simulation_1.0.0.20101021-0701.jar was not resolved. !SUBENTRY 2 org.bonitasoft.studio.simulation 2 0 2010-10-21 09:17:50.299 !MESSAGE Missing required bundle org.codehaus.groovy_1.7.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.300 !MESSAGE Bundle reference:file:plugins/org.bonitasoft.studio.console.libs_1.0.0.20101021-0701 was not resolved. !SUBENTRY 2 org.bonitasoft.studio.console.libs 2 0 2010-10-21 09:17:50.300 !MESSAGE Missing required bundle javax.servlet_0.0.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.300 !MESSAGE Bundle reference:file:plugins/org.eclipse.pde.core_3.6.0.v20100601.jar was not resolved. !SUBENTRY 2 org.eclipse.pde.core 2 0 2010-10-21 09:17:50.300 !MESSAGE Missing required bundle org.eclipse.equinox.p2.touchpoint.eclipse_[2.0.0,3.0.0). !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.300 !MESSAGE Bundle reference:file:plugins/org.codehaus.groovy.eclipse.core_2.0.2.e36-special20100412-1500-e36-special.jar was not resolved. !SUBENTRY 2 org.codehaus.groovy.eclipse.core 2 0 2010-10-21 09:17:50.300 !MESSAGE Missing required bundle org.codehaus.groovy_0.0.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.300 !MESSAGE Bundle reference:file:plugins/org.eclipse.pde.build_3.6.0.v20100603 was not resolved. !SUBENTRY 2 org.eclipse.pde.build 2 0 2010-10-21 09:17:50.300 !MESSAGE Missing required bundle org.eclipse.equinox.p2.director.app_1.0.200. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.300 !MESSAGE Bundle reference:file:plugins/org.mortbay.jetty.server_6.1.23.v201004211559.jar was not resolved. !SUBENTRY 2 org.mortbay.jetty.server 2 0 2010-10-21 09:17:50.300 !MESSAGE Missing imported package javax.servlet.http_2.5.0. !SUBENTRY 2 org.mortbay.jetty.server 2 0 2010-10-21 09:17:50.300 !MESSAGE Missing imported package javax.servlet_2.5.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.300 !MESSAGE Bundle reference:file:plugins/org.codehaus.groovy.eclipse.refactoring_2.0.2.e36-special20100412-1500-e36-special.jar was not resolved. !SUBENTRY 2 org.codehaus.groovy.eclipse.refactoring 2 0 2010-10-21 09:17:50.300 !MESSAGE Missing required bundle org.codehaus.groovy_0.0.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2010-10-21 09:17:50.300 !MESSAGE Bundle reference:file:plugins/org.eclipse.ocl_3.0.0.v201005061704.jar was not resolved. !SUBENTRY 2 org.eclipse.ocl 2 0 2010-10-21 09:17:50.300 !MESSAGE Missing required bundle lpg.runtime.java_[2.0.17,3.0.0). ... [/cc] This simply tells you the missing bundles in your feature. Then, for each Missing required bundle entry, you will need to add it to your feature, take a look at its dependency tree, add some its dependencies if you missed them, and... Retry !

Fixing runtime

It can easily happen that one of your plugins does not start without crashing the whole product. In such case, you silently lose the features provided by the code of your plugin. Then you can use the OSGi console to get insight about what's going wrong.

In this example, the log tells me that it cannot load the test plugin, but I don’t really know why. So I can try the following:
[cc]
mistria@mistri-laptop:~/BonitaStudio-20101021$ ./BonitaStudio -console
[… Some more or less useful stuff …]
osgi> start org.bonitasoft.studio.tests
org.osgi.framework.BundleException: The bundle “org.bonitasoft.studio.tests_1.0.0.20101021-2140 [1449]” could not be resolved. Reason: Missing Constraint: Require-Bundle: org.bonitasoft.studio.repository.test; bundle-version=“0.0.0”
at org.eclipse.osgi.framework.internal.core.AbstractBundle.getResolverError(AbstractBundle.java:1317)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.getResolutionFailureException(AbstractBundle.java:1301)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:319)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:284)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:276)
at org.eclipse.osgi.framework.internal.core.FrameworkCommandProvider._start(FrameworkCommandProvider.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.osgi.framework.internal.core.FrameworkCommandInterpreter.execute(FrameworkCommandInterpreter.java:155)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.docommand(FrameworkConsole.java:156)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.runConsole(FrameworkConsole.java:141)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.run(FrameworkConsole.java:105)
at java.lang.Thread.run(Thread.java:595)

osgi> start org.bonitasoft.studio.repository.test
org.osgi.framework.BundleException: The bundle “org.bonitasoft.studio.repository.test_1.0.0.20101021-2140 [1340]” could not be resolved. Reason: Missing Constraint: Require-Bundle: org.bonitasoft.studio.util.tests; bundle-version=“1.0.0”
at org.eclipse.osgi.framework.internal.core.AbstractBundle.getResolverError(AbstractBundle.java:1317)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.getResolutionFailureException(AbstractBundle.java:1301)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:319)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:284)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:276)
at org.eclipse.osgi.framework.internal.core.FrameworkCommandProvider._start(FrameworkCommandProvider.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.osgi.framework.internal.core.FrameworkCommandInterpreter.execute(FrameworkCommandInterpreter.java:155)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.docommand(FrameworkConsole.java:156)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.runConsole(FrameworkConsole.java:141)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.run(FrameworkConsole.java:105)
at java.lang.Thread.run(Thread.java:595)

osgi> start org.bonitasoft.studio.util.tests
org.osgi.framework.BundleException: The bundle “org.bonitasoft.studio.util.tests_1.0.0.20101021-2140 [1371]” could not be resolved. Reason: Missing Constraint: Require-Bundle: org.eclipse.swtbot.eclipse.finder; bundle-version=“2.0.0”
at org.eclipse.osgi.framework.internal.core.AbstractBundle.getResolverError(AbstractBundle.java:1317)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.getResolutionFailureException(AbstractBundle.java:1301)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:319)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:284)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:276)
at org.eclipse.osgi.framework.internal.core.FrameworkCommandProvider._start(FrameworkCommandProvider.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.osgi.framework.internal.core.FrameworkCommandInterpreter.execute(FrameworkCommandInterpreter.java:155)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.docommand(FrameworkConsole.java:156)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.runConsole(FrameworkConsole.java:141)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.run(FrameworkConsole.java:105)
at java.lang.Thread.run(Thread.java:595)
org.osgi.framework.BundleException: The bundle “org.eclipse.swtbot.eclipse.finder_2.0.0.568-dev-e36 [1301]” could not be resolved. Reason: Missing Constraint: Require-Bundle: org.eclipse.swtbot.swt.finder; bundle-version=“2.0.0”
at org.eclipse.osgi.framework.internal.core.AbstractBundle.getResolverError(AbstractBundle.java:1317)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.getResolutionFailureException(AbstractBundle.java:1301)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:319)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:284)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:276)
at org.eclipse.osgi.framework.internal.core.FrameworkCommandProvider._start(FrameworkCommandProvider.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.osgi.framework.internal.core.FrameworkCommandInterpreter.execute(FrameworkCommandInterpreter.java:155)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.docommand(FrameworkConsole.java:156)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.runConsole(FrameworkConsole.java:141)
at org.eclipse.osgi.framework.internal.core.FrameworkConsole.run(FrameworkConsole.java:105)
at java.lang.Thread.run(Thread.java:595)

[… And so on, until …]

osgi> start org.hamcrest.library
Cannot find bundle org.hamcrest.library
[/cc]
Thanks to the console, I can figure out that the missing bundle is org.hamcrest library. I just add it to my feature, and everything works better. Using the console is a general method to resolve dependencies issues. The console is often your best friend!

Conclusion

As you can see, switching from a plugins-based product to a feature-based is not immediate. Features do not provide the fantasically useful Add Required Plug-ins button that you can find on a plugin-based product, and so this move will probably lead to errors in dependency management.

The "Add Required Plug-ins" button is only available for plugin-based products

That’s why, once you have your feature, you need to tweak it to get your product working, since you can easily miss a bundle. Moreover, the bundles you ship may have a different version after a feature-based build. You’ll need to test it well to ensure that this move did not break some features of your product because of bundle versioning. I opened bugs 328323 and 319085 to ask for improvments on this topic.

As always, automated non-regression tests are your very best friend for such a move.

Our feature-based build has now been working for a few days, and I can already tell I am quite happy with it. The build of our 2 products is easier to understand and maintain, and now looks like a simple composition of features. Also, adding a language does not require us to modify the .product any more, and it helps us to get closer to the ability to provide “language packs” as extensions of the product.

Next step, moving to p2…but that will be another story!

By the way...

Aurélien and I will be at Eclipse Summit in Ludwigsburg. Aurélien will have the opportunity to present to the Eclipse community how we leverage the Modeling projects in our product. If you want to have a chat with us, about anything, feel free to drop us an email to plan a meeting, or grab us during the event!