Most of you have probably seen the drupal.org front page post concerning the deployment of the automated testing system version 2.1 and the subsequent beta phase of contrib project testing. This post will provide additional details regarding the recent deployment and result of that deployment.

Contributed project testing Without a doubt the biggest new feature provided by the 2.1 update was the addition of the ability to test contributed projects. The reason this took so long to support, was not due to lack of support from the testing master server or testing clients, but rather in the determination of module dependencies. In order to test contrib projects that have that have dependencies on other contrib projects the dependency relationships must be determined so the dependencies can be checked out by the testing clients.

The beta status will continue until the dependecy parsing component is complete and able to determine dependencies with 100% accuracy. I have documented what needs to be done in order to complete the code and will begin working on it, but any help would be appreciated.

The biggest issue that needs to be decided, in part by the community, is the way in which Drupal 7 style dependency information can be provided in Drupal 6 module .info files. Once we have a consensus that does not break compatibility with Drupal 6, core then the system needs to be updated to support Drupal 7.x style dependency restrictions in both 7.x and 6.x compatible projects.

On the positive side of things, we were able to track down issues with contrib testing after enabling reviews on a small number of contrib projects. With the help of the project maintainers reporting issues, I was able to fix the problems relatively quickly. Now that we have deployed the changes I have gone ahead and enabled testing on 31 projects (included Drupal core) that had requested to be included in the beta phase. It is great to see this much interest and enthusiasm by the Drupal community in embracing testing. The following is a list of the 31 projects that are currently taking advantage of the automated quality assurance system.

Brief look at of ATS 2.2

I wrapped a few other bug fixes along with the contrib testing bugs into a new release, which was deployed this morning. You can get a feel for what the contrib testing bugs were that we squashed in the initial testing phase below.

PIFR 6.x-2.2, 2010-01-27

  • Bugs:
    • #695350: Provide ‘last’ field in pifr.retrieve() and correct query.
    • Events should only be triggered when trigger modules is available.
    • Remove notices when test record is saved before client record.
    • #695278: Test list should be generated from root directory.
    • #695278: Module ‘tests’ directory should be searched.
    • #696044: Patches are not being applied properly to contrib projects.
    • #696194: Cannot preview client test information.
    • Do not add SimpleTest as dependency if it has already been added.

Coder review

In an effort to expand Drupal quality assurance beyond straight forward testing, I have written the initial integration for coder and coder_tough_love. Since Drupal core does not currently “pass” the review I have not enabled it on the main Drupal core tests, but have instead provided a stub test to demonstrate the functionality and provide and public set of results to work against.

It is my goal that Drupal 7/8 core and coder/coder_tough_love will be cleaned up to the point where there are no false positives from coder and Drupal itself is clean so that the review can be enabled on all Drupal core patches and commits.

I will queue the stub test for review every few days, or upon request in IRC.

Notification e-mails

Another new feature provided in the 2.1 update was the addition of a trigger/actions implementation that allows e-mails to be sent from qa.drupal.org upon a test result. The system is not designed to send e-mails to specific project maintainers and such, but instead to notify test client maintainers of a client confirmation failure (client no longer working) and the core maintainers upon a bad commit that break Drupal core.

I would like to extend this system to provide general test e-mails for project maintainers and other interested parties, but that work needs to be done on the drupal.org side of things.

Realizing the system’s power

Something that may not be apparent to everyone is that the pifr/pift system can be used for more then just reviewing patches and commits. Not only can we extend the system to perform performance analysis, test coverage analysis, textual analysis, and automatic tests/security reviews, but we can use the system as a generic platform for performing distributed operations on intelligent triggers.

For example, we could use the system as a distributed platform for generating the API documentation seen on api.drupal.org. The system would automatically rebuild the documentation on commit and due to its distributed nature might even allow for the expansion of api.drupal.org for contrib projects (need to talk to infrastructure).

An advanced plugin API is provided for implementing additional functionality for the system. I have created an easy to extend Drupal specific implementation that should make it easy to build Drupal specific additions. Please contact me in IRC/email or the issue queue if you are interested in developing a plugin.

The future and beyond

I have identified items that I would like to have completed for the next installment of the system, created issues for them, and provided a summary below. I would appreciate any help from the community in completing them.

Project Issue File Review (PIFR) (qa.drupal.org & clients)

Project Issue File Test (PIFT) (drupal.org)

Project Info (drupal.org)