Managing upstream integrations and releases

The AWS SDK for PHP Drupal integration project, that I maintain, provides releases that correspond to each release made by Amazon. Having releases that correspond to an upstream source is something that you see in Linux packaging systems and offers some interesting bonuses. In Linux distribution model individuals do not monitor upstream for changes, and packages are rebuilt and increment version regardless of changes to the packaging scripts themselves. Given that Drupal distributions mimic Linux distributions in many ways and even individual site builds I think the reasons behind making corresponding releases are interesting and worth consideration.

To clarify what releases corresponding to upstream releases means consider the following from the AWS SDK for PHP project page.

Since Drupal.org does not allow for three-part version numbers this project follows the AWS SDK for PHP 1.x development line and the release version mapping is as follows. The mapping shows the Drupal release to the Amazon release. Please ensure you are using a Drupal module and Amazon SDK pair that match the version number mapping. For example, use the 4.1 Drupal module with the 1.4.1 Amazon SDK.

  • 4.x -> 1.4.x
  • 3.x -> 1.3.x
  • 2.x -> 1.2.x

Making a release whenever upstream makes a release means that some releases may end up with little or no changes to the actual Drupal module. The AWS SDK module provides a Drush Make file that is updated each release to point to the matching upstream release, but that is the only change for some of the releases.

Making arbitrary releases may sound foolish at first, but consider some of the advantages.

  • Drupal site maintainers and distributions developers can receive update notifications from the standard Drupal core update system instead of having to monitor upstream for releases. Making it easier to watch for updates encourages keeping up-to-date with upstream changes which has implied benefits.
  • Any incompatibilities with a specific version of a third-party system or upstream library do not need to be kept track of since the integration project is always used with a specific upstream release.
  • New features or configuration options from an upstream source may be added immediately to the integration module without worry about incompatibility with older releases of the upstream source. Meaning if a configuration variable changes name or a new option is added it can be exposed through a Drupal UI without worry if people are using the appropriate upstream version (since they are always intended to work with a matching release).
  • When building a site or distribution using Drush Make specific versions of upstream libraries can be easily included using the corresponding Drupal project with a proper make file. If one wants AWS SDK 2.6 simply add projects[awssdk] = 2.6 instead of having to include the Drupal project and override any third party generic make script.
  • Handling of updates, when necessary, is much simplier since you always know the library version that was used with the previous release.

The general pattern that seems to emmerge is that upstream sources integrate better into Drupal workflow, tools, and infrastructure when matching releases are made.

It does not seem that any other (or few if they exist) third-party integrations or library integrations on drupal.org follow this workflow. Making corresponding releases may not be appropriate in all situations, but I think it warrants consideration and I am interested to hear thoughts on the subject.

Comments

What if you are dealing with upstream code that is dysfunctional in the sense that it doesn't have tagged releases - breaking changes can occur at any time with no warning?

Any tips as to how to handle this better? Should a new release be made on d.o. every time a breaking change was introduced upstream? How would people now which was the one to coordinate with the upstream library since it doesn't have tagged releases?

For dysfunctional upstream sources I am not sure there is a solid general answer.

Should a new release be made on d.o. every time a breaking change was introduced upstream?

Making a release every commit or interesting commits doesn't sound like a good idea. You might almost make your own idea of releases from them. You could otherwise attempt to host a clone on github/gitorious or such with tags or just get the source to be sensible.

Do you have a particular upstream source you are referencing? Curious.

I was referring to the Salesforce.com Toolkit for PHP - https://github.com/developerforce/Force.com-Toolkit-for-PHP, which is a requirement for the Salesforce API module, which I co-maintain. That module, in turn, is a requirement for the Salesforce/Ubercart Integration module which I maintain.

We've had a lot of issues in the past year where the toolkit has changed the return values of its methods without tagging a new release, or really notifying potential users of the change.

I have made contact with at least one of the maintainers now though, so potentially this situation will change in the future. I will suggest to them that they start tagging releases on Github from now on. That way, we could at least say that the Salesforce API module on d.o. was compatible with a specific tagged release of their toolkit.

Also, though, I'm hoping that they have fixed most of their outstanding issues, so things might calm down on their end now.

On the bright side, at least I'm not one of the developers trying to keep up-to-date with the Facebook APIs... :)

Getting upsteam to tag would solve your problem. Lets hope they go along with it. :)