Part 2: Breathing new life into the testbot

The purpose of this post is to describe the solution that, after careful consideration, seems best suited to alleviating the situation described in the previous post. Other solutions may exist that we have not considered and that will effectively solve the problem. We are open to discussing alternatives and welcome constructive comments on our proposal. At the same time, we discourage negative comments that do not offer a positive alternative. As it is clear the current situation needs improvement, simply dismissing our proposal without offering a better alternative is not useful.

Proposal

ReviewDriven (RD) is a distributed quality assurance platform built to provide a simple yet powerful interface that makes it easy to apply the best practices of continuous integration, test driven development, and automated quality reviews to your development life cycle. The ReviewDriven stack provides a completely rebuilt system designed to take advantage of Drupal 7 and contributed modules that will allow Drupal.org, Drupal development shops, and site owners to take advantage of automated quality assurance. In Drupal terms, RD is the next generation of the testbot (qa.drupal.org).

We would like to see DO and other interested parties take advantage of automated QA tools. Towards that end, we propose that DO engage RD to assume the role of the testbot and provide those same services to the Drupal community.

Advantages

One of the limitations of the current system and one of the primary concerns we addressed with RD is the lack of control over the testing workflow. For example, the current workflow settings apply globally instead of on a more granular basis. In contrast, the RD platform will allow Drupal.org full control to define the workflow and settings to be used with each review. The integration between the testbot (RD) and Drupal.org will continue to be maintained as an open source module which will allow anyone to contribute ideas and changes to the QA workflow on Drupal.org. Since the ReviewDriven stack provides a versioned API, the Drupal.org integration may be maintained and updated independent of RD and on it's own schedule. This approach leaves all the control and flexibility in the hands of the Drupal community and shifts the burden of the testbot to RD.

Other challenges faced by the current system were also taken into consideration when building ReviewDriven. The ReviewDriven stack is extremely flexible which in itself solves a number of the current issues and opens up a variety of new options. This opens up the possibility of reviews for things like Coder and code coverage.

The RD stack as a whole is much more maintainable since it is built on as many contributed modules as possible. This keeps the actual codebase much smaller then the current system (PIFR). Depending on contributed modules has led us to suggest and contribute a number of improvements to those modules, and to create other contributed modules. This also implies the system is easily maintainable by the Drupal community in the event we open source the code (since the majority of the code is already maintained by others). The RD architecture is analogous to any other Drupal site (sponsored by a good citizen) in that we are maintaining the code specific to our site while contributing back to the community modifications to existing modules and new features designed as generic modules.

Putting QA in context

Using a house as illustration for building a site on Drupal

Our proposal hinges on the fact that the testbot (and most of Drupal.org) provides a direct benefit to everyone but, just like roads and other infrastructure, the cost needs to be shared. Core and contributed modules provide a direct benefit in an indirect way by reducing the amount of and time spent writing custom code, ensuring the base system works as intended, and allowing you to leverage with confidence that code base while building your site. Core and contributed modules represent the sure foundation upon which you build your house.

There has been plenty of discussion about the important role the testing infrastructure and testing as a whole played in Drupal 7 development. The benefit of QA is also evident by the fact that a number of very large Drupal sites launched on Drupal 7 before it was officially released. The stability and dependability offered by quality assurance testing is something everyone wants.

Drupal.org is one of very few open source projects, much less projects in general, to adopt quality assurance and testing. The Drupal core development process requires new features to have tests and bug fixes to include tests. This workflow is encouraged for contributed projects and has been adopted by many of the more used projects among others. Not only does this help ensure the stability and quality of Drupal and its contributed projects, but in turn serves as a selling point and differentiator for Drupal adoption. Given that QA is both an adoption point and a vital tool for improving Drupal, does it not follow that it makes sense to provide funding for a full-time effort towards its improvement?

Just as many Linux developers are full-time employees paid to work on improving Linux, we seek to work full-time on improving the Drupal ecosystem through quality assurance. We are not the first to be funded full-time to work on Drupal or to be paid to improve Drupal.org. A perfect example is Angie Byron (webchick) who was hired by Acquia to work full time on Drupal. Just as Linux was started by hobbyists and grew into a profession, so too Drupal appears to have outgown the ability to be maintained entirely by volunteers.

Funding

We see two separate areas that need funding. The first focuses on taking advantage of the ReviewDriven platform by updating the Drupal.org integration with the new testbot (RD). The second area is the ongoing fee for use of the platform (which includes infrastructure costs). RD will use the ongoing fees to improve and maintain the platform (like any other business).

Harnessing the flexibility provided by ReviewDriven will require a large overhaul of the current Drupal.org QA integration module (PIFT). We envision Drupal taking advantage of the granular settings supported by RD to provide per project, per release, per issue, and per patch settings to control the reviews made. Granular settings will ensure that the various workflows, coding standards, and environments that exist in "contrib" can be handled properly. Many projects have different requirements or adhere to different standards between their various releases. The integration with the new testbot would remain open source as Drupal.org integration and can be funded just like any other Drupal.org project.

We would also like to see a QA status advertised on each project page, possibly even some sort of ranking based on a number of quality assurance metrics. These metrics would help people select between similarly featured modules, advertise that we do QA, and help motivate developers to adopt QA. We have many other ideas for improvements and anticipate suggestions from the community.

The ongoing service also requires ongoing funding to handle infrastructure costs, feature improvements, updates for Drupal core changes, and requests from the Drupal community would ensure that things do not stagnate. We have a vision for new features that would significantly improve the Drupal ecosystem, some of which we have discussed with a few community members.

We envision either the Drupal Association or a group of businesses and other organizations with an interest in Drupal to hire RD as the logical successor to the current testbot. Our business will be to develop and maintain the testbot for use by Drupal.org and other organizations. The same approach can then be applied to other critical peices of infrastructure such as the improvement of Drupal.org and its maintenance. We would like to pioneer this effort for Drupal to further enhance the process and tools available to Drupal contributors and the community.

Further details about the specifics of the arrangement, details of the improvements, and plans for the future can be found in our formal proposal and addendum.

What to expect during the transition

With both the short-term upgrades and improvements to the Drupal.org integration and the ongoing RD services funded, we see the transition to RD taking shape as follows.

The first stage of the transition will require the update of Drupal.org's integration with the testbot to provide basic connectivity with ReviewDriven. Supporting RD will require a number of changes, both user-facing and behind the scenes. In addition, just using the ReviewDriven platform will enable a number of features and workflows.

Once the initial integration is ready to be tested in production, we suggest both ReviewDriven and the predecessor be run in parallel. Running both systems in parallel will provide the community with a preview of what is to come and an opportunity for feedback. Results from the two systems can be compared to provide a final round of human checks and give people time to adjust to the new system. After the completion of the parallel phase the old testbot will be deactivated and the new system will be given priority.

The second phase involves the larger changes necessary to take advantage of ReviewDriven's features and flexibility. We will start discussions and work on this phase as the initial integration stabilizes.

Some of the exciting features we will expose to DO in one or both of the stages include:

Code coverage example
Example of code coverage during a test run
green = executed, red = not executed, gray = ignored (or non-executable)
  • Much improved turnaround time with the ability to scale as the test suite grows
  • Code coverage reports from test runs [picture]
  • Testing of sandboxes (both core forks and modules)
  • Support for the developer application process
  • Drush make scripts for retrieving third party libraries.
  • Drush make files in lieu of parsing the project dependencies
  • Execution of arbitrary commands during various stages of the worker processing
  • Automatic enabling of issue retesting
  • Completely automated site reviews (reviews run against a configured site)
  • Reroll of a patch (using git --rebase) on one issue after a commit on another issue (for example, this large core change)
  • Display of quality metrics
  • Visible branch test results
  • Forcing a patch to run when a branch is broken (in order to fix the branch)
  • Determine the disruption to other patches that would be caused by another patch (e.g. the patch to move all core files)
  • Run Selenium tests
  • Provide separation between the testbot and the Drupal installer by writing a special script maintained by the community

Conclusion

We look forward to feedback about our proposal and encourage you to voice your opinion. Please be sure to be constructive. In case its not obvious, we are extremely passionate about doing this. So let's make this happen.

Comments

It'd be one thing if you wanted to ask the Drupal Association to hire yourself as Jimmy the test guy to run qa.d.o. It's an entirely different, off-putting thing, to brand a for-profit entity that, oh by the way, is also available for hire to other Drupal shops. It seems somewhere you lost your way or didn't entirely capture the culture Drupal has created for co-op style project management amongst Drupal shops.

Perhaps if you just market reviewdriven.com as a platform available for others to use for driving testing, for shops who don't want to manage their own Jenkins setup, and then share that same platform, hosted by OSU on Drupal equipment, that'd be a better fit. If that model doesn't work, then perhaps what you've discovered is that if someone cares enough to do automated reviews, they're probably also capable of running their own Jenkins server. In that case, perhaps you're better of just joining the staff as a new employee of the DA.

I am happy to see you are confident enough in your views that you did not including your identity. This is not an idea that someone just came up with one night and decided to run with. The many possibilities and angles have been given months of consideration.

I am not sure if you actually read the entire post, but one of the primary ideas was to bring QA to _MORE_ than just d.o. Sure the top 10 shops will setup Jenkins and all end up doing the same thing, but the thousands of smaller shops simply do not have that kinda of time to waste. By allowing those shops to effectively use QA the quality of Drupal development as a whole can be raised instead of only the code hosted on drupal.org.

Another primary point was that the testbot does not fit the standard model and requires a different one. Joining the DA staff would solve the problem for d.o, but we still end up with 99% of Drupal shops not implementing QA of any kind.

As the stated we are open to other alternatives that are actually practical at providing for both audiences, but as of yet I have not heard anything new or useful. This seems like a win in that we raise the overall standard in the Drupal world instead of just Drupal.org and at the same time Drupal shops are sponsoring improvement of Drupal.org, just like the standard model you referenced.

One problem with this is that probably only the top10 shops, which setup jenkins for their own, actually use testing a lot. Most of the small shops don't want to have the "costs" of writing tests.

"The integration between the testbot (RD) and Drupal.org will continue to be maintained as an open source module which will allow anyone to contribute ideas and changes to the QA workflow on Drupal.org."

I have to agree with anonymous above. Relying on a branded, for-profit company running propietary software is not where I want to see Drupal in the future. It's requirement that all code contributed to D.O be GPL 2.0 compatible - it would feel really hypocrytical if we started running all our tests on a closed platform.

So here's my alternative - open source not just the integration module, but the whole thing. Let D.O run its own version, or the Drupal Association decide to pay reviewdriven.com to maintain the hardware that the FOSS runs on. You can advertise on your module page, "Maintained by ReviewDriven", and on your site mention the fact that you power D.O's testing.

But as long as you're setting up a vender-lock-in scenario, I'm not going to agree.

We had a long discussion about this topic in #drupal-contribute the other day which raised these, as well as many other points.

So I 100% support Jimmy and Jim pursuing the ability to be paid to do what they love. They've made tremendous investments in Drupal's platform and the automated testing framework has rocket-powered the core development team, and obviously these tools will see more and better attention if the Berrys are allowed to focus on them and not also trying to wrangle client work. I also think there's a lot of room in the Drupal ecosystem for a service that makes this kind of work easy, because it certainly is not.

At the same time though, it's a lot to ask for the Drupal community to move away from a fully free, GPLed testing stack (regardless of its current level of bugginess) to a closed-source, subscription-based commercial vendor, regardless of how much we might know/trust/respect said vendor. Heck, we don't even run Mollom on Drupal.org for this very reason, and that's a proprietary service by our freaking BDFL. :) For all the same reasons that we all don't use closed-source operating systems, version control systems, CMSes, browsers, etc. I don't think it's a good investment of the community's money to use a closed-source testing framework, either.

And while I strongly disagree with the wording of some people in the comments here, I think what some are having a visceral reaction to here is that the Drupal community has already invested thousands of dollars into these tools over a number of years, in the form of Google Summer of Code and other types of sponsorship. And now it seems like the latest people to work significantly on those tools are taking that investment, closing it up and putting improvements to it behind a "secret sauce" paywall, and then issuing an ultimatum that it's either pay us to use this new thing, or suffer. I know that's not the intent; the intent is to make the important work that you're doing sustainable, but the approach and proposed solution I think is leaving a bad taste in peoples' mouths.

The way forward, IMO, is just as threewestwinds outlines: open source all of your secret sauce, and build ReviewDriven's business on your subscription-based PaaS offering that runs the open source tools in a fast and easy way, and contract out for customizations, support, and so on to the platform. If you did this, it would be a lot easier of a sell for the DA to pony up for an initial investment in moving to better GPL tools on Drupal.org, and then Review Driven would have a nice initial flagship customer to put in its marketing brochures.

I agree.

On one hand: Jimmy is free to decide what to do with his time, and the Drupal Association is free to continue improving the existing testing bot and infrastructure if they want to.

I'm sure it won't be a simple decision for the DA to choose what to do (hire someone? outsource it to ReviewDriven?) but in any case, it's time to evaluate what to do. I agree testing is *critical* to keep a sane development of Core; it should be given the proper attention regardless.

On one hand, I know that you are really good at building QA platforms and that it'd probably be really solid.

On the other hand, I don't think we should be using a proprietary web service to automate testing of GPL code. Seems a little odd to me. I don't want to shoot down your idea here, but I don't think you're going to get a lot of buy in.

Whatever the outcome, don't stop working nor contributing to the Drupal FOSS environment. The best solution proposed here is indeed to GPL and distribute the whole stack — as the mantainer and expert on the technology, no one will thrive on your work before you do.

You can try scraping core issues and submitting your reviews there, but then you'll have to work with the data you can get (from git, issues and what else). In a smaller scale, it is what klausi is doing for project reviews, see http://drupal.org/sandbox/klausi/1320008 and the project application issue queue.