reviewdriven

Drupal testbot command line tool

Are you a developer familiar with the patch submission workflow?

  • load the issue page in your browser
  • add some helpful text describing the patch
  • create a patch file
  • upload it to the issue
  • change the issue status to 'Needs review'
  • wait 20 minutes (at best) to find out if the tests pass or fail with your patch

If the tests fail, then you get to do it all over again, including making possible revisions to the issue summary.

Ever wanted to run the tests while iterating on a change, but are reluctant due to:

  • insufficient local machine resources or configuration to run tests efficiently
  • inconsistent results compared to the official testbot
  • disruption of your 'creative coding moments' due to the patch submission workflow
  • long test runtime on qa.d.o

Well you may be in luck.

What if you could type "drush testbot" to have the changes in your working branch submitted and tested as a patch against Drupal? And be able to view the results in five minutes or less? If this sounds interesting, then install the "drush testbot" command file (see instructions) and take it for a spin.

Suggested usage

If your changes only affects a single module (or a few), then an assessment of your changes can often be had by simply running the tests for that module (or those modules). Once you know that your changes do not break the immediate context, then the entire test suite can be run before posting the patch on the issue queue.

You can do so by including a list of test classes to run (see examples); in so doing you might reduce your response time to a matter of seconds.

Drupal 8 examples

Run the command from the working directory with your code changes. The default test environment for Drupal 8 is mysql 5.5 and php 5.4. In lieu of setting properties on the command line, add them to a '.testbot' file in the repository root directory (i.e. the same directory that contains the .git directory).

Description Command
All defaults (without .testbot file)
drush testbot
Using properties from .testbot file
drush testbot
Single test class
drush testbot --properties='{"classes":["Drupal\taxonomy\Tests\TermTest"]}'
Alternate environment and multiple test classes
drush testbot --database='mongodb-2.6' --php-version='php-5.5'
--properties='{"classes":["Drupal\node\Tests\NodeQueryAlterTest",
"Drupal\node\Tests\NodeRevisionsTest"]}'

Drupal 7 examples

The default test environment for Drupal 7 is mysql 5.5 and php 5.3. The command syntax matches that shown above with the addition of the 'branch:7.x' parameter to the properties. An example of testing two classes is:

drush testbot --properties='{"branch":"7.x","classes":["NodeQueryAlter","NodeRevisionsTestCase"]}'

Instructions

Point your browser to https://github.com/reviewdriven/testbot and see the README file for installation instructions.

Caveat

If our test queue is empty when you submit a patch (and do not specify a list of classes to test), the actual response time for a Drupal 8 patch will be under ten minutes based on existing test suite and available hardware at the time of this writing. If the queue is full, we may not run your patch and will let you know.

Final thoughts

If you have questions, would like to offer constructive comments or suggestions, or want to let us know that you found the tool useful, please post in the github issue queue.

The testbot is provided as a complimentary service to the Drupal community. No financial assistance is received from the Drupal Association (to defray even the hardware costs). Unlike the current testbot which runs on the most powerful compute instances offered by Amazon Web Services, these tests are run on hardware with roughly 25% the processing power yet return results in under half the time.

Code coverage reports

Today, I made public a whole bunch of commits I made while improving the Code coverage project for use with ReviewDriven. You may remember the screenshot I included in Part 2: Breathing new life into the testbot which is where this is coming from. Thanks to the improvements the module now actually works and is much more efficient, polished, accurate, and much more. With that in mind I am proud to announce the first stable release of the project targeted at Drupal 7.14.

Using the module you can get coverage reports for individual page executions or any combination of tests. Code coverage reports can be extremely useful in determining areas of code that are not tested at all or provide a snapshot of the code involved in generating a page.

Pages

Code coverage can be recorded by adding ?code_coverage=true to the end of a page URL. After the page has completed execution a linked will be placed at the bottom of the HTML which will display outside the page style.

Coverage page

The link will open the coverage report generated for that page request. The report will include all the files that where loaded during the execution of the page.

Tests

Similarly, a link is provided for the code coverage recorded during a test run.

Coverage test

Reports

The report includes two parts: 1) the summary or index, and 2) the line-by-line coverage information. The links described above point to the summary of the coverage information.

Coverage summary

The links in the summary point to the line-by-line coverage information overlayed on the corresponding code. The colors indicate the following.

  • green = executed
  • red = not executed
  • gray = ignored (or non-executable)

Coverage example

Filters

The coverage scope may be filtered to focus on improving coverage for a particular module/file/directory. Reducing the scope will also improve the coverage recording performance which may be useful when when dealing with large tests.

Coverage filter

Future

I have already integrated the Code coverage project with Conduit (the open source ReviewDriven platform) which will be replacing the current system running qa.drupal.org. The plan is to get the new platform up and running in parallel with the current system at which time regular coverage runs against core (and contrib projects) can be made publicly available.

Reflections on Drupal Quality Assurance

We recently launched ReviewDriven, a distributed quality assurance platform, which is the culmination of months of work and knowledge gained working with Drupal and its community over the past 3+ years. I look forward to feedback from the community regarding ReviewDriven, and being able to fund further development of this service and, at the same time, improving the Drupal quality assurance ecosystem.

Since ReviewDriven is a major event in my life and Drupal career it caused me to reflect on how I got to this point. From my humble beginnings in the Google Highly Open Participation Contest (GHOP), where my testing roots were planted, I received encouragement and mentoring which inspired me to continue working with open source. Previously, I had been interested in contributing to open source, but had never found a place to plug in. Since my start with Drupal I have contributed in a variety of ways to openSUSE, the Linux kernel, and KDE.

After my initial introduction to Drupal, I was received with enthusiasm and the community helped me get to Drupalcon Boston 2008. I took part in the GHOP and SimpleTest presentations. During the coding sprint after the conference, I was approached by Kieran Lal who offered to help me get to a testing sprint in Paris. Again the Drupal community along with some help from Google made it possible for me to attend. Not only was it my first time out of the United States, but I got to spend a few days working closely with some of Drupal's best. During the sprint Drupal took a major step towards realizing automated testing with the introduction of SimpleTest (or rather a fork) into Drupal core.

In the months after the sprint I pushed hard to maintain, add to, and improve the tests in core. At the time patches were committed without much thought given to the tests so keeping the tests passing was a full time job. After discussions with Kieran I ended up taking over the testing.drupal.org (now qa.drupal.org) effort. After a radical redesign and plenty of work we managed to deploy testing.drupal.org and enable integration with the issue queue once we finally got all the tests passing. With the integration also came the adoption of the current "tests always pass" ideology and requirement to include tests with patches which has revolutionized Drupal development. The system even caught some interesting drupal.org bugs.

Again thanks to community support I was able to attend Drupalcon DC and give a presentation with Kieran on the testing saga. The conference was a lot of fun in general and gave me a chance to meet all the people who I had been working with fairly regularly. Later that year I was accepted to Google Summer of Code (GSoC) for the second time and I worked part-time for Acquia as an intern over the same summer. After an exciting summer, with help from the community, I attended Drupalcon Paris 2009 where I gave another presentation on SimpleTest and the automated testing system with Kieran. After a productive Drupalcon we deployed the second version of the automated testing system and continued to improve the system.

Before the end of the year I was hired full-time by Examiner to lead their quality assurance effort. The opportunity provided me with first-hand experience on how quality assurance can fit into an enterprise Drupal development workflow. Additionally, I was able to spend a portion of my time improving the automated testing system so that we could enable partial testing of contributed projects. Examiner required a slightly different approach to testing which formed the basis for SimpleTest 7.x-2.x. Examiner sponsored the development team to attend Druaplcon San Francisco 2010 during which I gave a talk at the Core developer summit on Quality Assurance in Drupal 8, a SimpleTest presentation, and a productive BoF on testing.

In addition to the specifics mentioned I have been blessed to work with and learn from many skilled Drupal developers, and to contribute to Drupal core and contrib which has further refined my skills. My Drupal career has been a great learning experience in addition to being fun and exciting. I look forward to continued involvement with and support from the Drupal community!

Subscribe to RSS - reviewdriven