Boombatower Blog

As you may have seen in my previous post I have been getting back into the swing of things after some time away. I have a number of projects that I need to attend to, but as always not enough time to work on them all. Bellow you will find a list of my most important projects that need attention. If you have time available and an interest in helping out I would appreciate any extra hands. A number of the items do not have links since I have yet to spec them out completely, but I will update this post as I fill them in.

• Parse project .info files: present module list and dependency information - needed in order for contrib testing to work 100% of the time and to move it out of beta so anyone can enable testing of their project.
• PIFT 2.3 (drupal.org side of qa.drupal.org) - In order to make the most of the qa framework a number of improvements need to be made, mainly options and information display on drupal.org so everyone can take advantage of all the features the system already provides.
• PIFR 2.3 (qa.drupal.org and clients) - A fair amount of testing of the current development code base, a few more bug fixes, and features are needed to round out the next release.
• A few remaining cleanup items for SimpleTest in Drupal 7 core.
• SimpleTest 7.x-2.x - The branch will be used for continued development during Drupal 7 life cycle and already contains a number of improvements over the base Drupal 7 testing system. It was designed so that it can coincide with the Drupal cores framework and allow for tests to use either framework (one framework per module). The switch module, which no longer works (need to look into), allows you to switch between the contrib and core versions with a single click. The switch module also could be used by other projects attempting to replace core modules and can do so with a small amount of abstraction. With the next PIFT/PIFR release I plan to support projects that wish to use the 7.x-2.x API for testing.
• Code coverage - Reports for all tests (patches or at least core/contrib commits).
• Refactor/Rewrite SimpleTest for Drupal 8 - I have an extremely detailed roadmap in my head (and some paper notes) and a small code base that needs to be made public.
• PIFR 3.x - Same as above, lots of plans, but not very much publicly available yet.

Feel free to read through the issues, information, and code and let me know if you need any help getting started. Thanks!

Most of you have probably seen the drupal.org front page post concerning the deployment of the automated testing system version 2.1 and the subsequent beta phase of contrib project testing. This post will provide additional details regarding the recent deployment and result of that deployment.

Contributed project testing
Without a doubt the biggest new feature provided by the 2.1 update was the addition of the ability to test contributed projects. The reason this took so long to support, was not due to lack of support from the testing master server or testing clients, but rather in the determination of module dependencies. In order to test contrib projects that have that have dependencies on other contrib projects the dependency relationships must be determined so the dependencies can be checked out by the testing clients.

The beta status will continue until the dependecy parsing component is complete and able to determine dependencies with 100% accuracy. I have documented what needs to be done in order to complete the code and will begin working on it, but any help would be appreciated.

The biggest issue that needs to be decided, in part by the community, is the way in which Drupal 7 style dependency information can be provided in Drupal 6 module .info files. Once we have a consensus that does not break compatibility with Drupal 6, core then the system needs to be updated to support Drupal 7.x style dependency restrictions in both 7.x and 6.x compatible projects.

On the positive side of things, we were able to track down issues with contrib testing after enabling reviews on a small number of contrib projects. With the help of the project maintainers reporting issues, I was able to fix the problems relatively quickly. Now that we have deployed the changes I have gone ahead and enabled testing on 31 projects (included Drupal core) that had requested to be included in the beta phase. It is great to see this much interest and enthusiasm by the Drupal community in embracing testing. The following is a list of the 31 projects that are currently taking advantage of the automated quality assurance system.

Brief look at of ATS 2.2
I wrapped a few other bug fixes along with the contrib testing bugs into a new release, which was deployed this morning. You can get a feel for what the contrib testing bugs were that we squashed in the initial testing phase below.

PIFR 6.x-2.2, 2010-01-27
------------------------
- Bugs:
   * #695350: Provide 'last' field in pifr.retrieve() and correct query.
   * Events should only be triggered when trigger modules is available.
   * Remove notices when test record is saved before client record.
   * #695278: Test list should be generated from root directory.
   * #695278: Module 'tests' directory should be searched.
   * #696044: Patches are not being applied properly to contrib projects.
   * #696194: Cannot preview client test information.
   * Do not add SimpleTest as dependency if it has already been added.

Coder review
In an effort to expand Drupal quality assurance beyond straight forward testing, I have written the initial integration for coder and coder_tough_love. Since Drupal core does not currently "pass" the review I have not enabled it on the main Drupal core tests, but have instead provided a stub test to demonstrate the functionality and provide and public set of results to work against.

It is my goal that Drupal 7/8 core and coder/coder_tough_love will be cleaned up to the point where there are no false positives from coder and Drupal itself is clean so that the review can be enabled on all Drupal core patches and commits.

I will queue the stub test for review every few days, or upon request in IRC.

Notification e-mails
Another new feature provided in the 2.1 update was the addition of a trigger/actions implementation that allows e-mails to be sent from qa.drupal.org upon a test result. The system is not designed to send e-mails to specific project maintainers and such, but instead to notify test client maintainers of a client confirmation failure (client no longer working) and the core maintainers upon a bad commit that break Drupal core.

I would like to extend this system to provide general test e-mails for project maintainers and other interested parties, but that work needs to be done on the drupal.org side of things.

Realizing the system's power
Something that may not be apparent to everyone is that the pifr/pift system can be used for more then just reviewing patches and commits. Not only can we extend the system to perform performance analysis, test coverage analysis, textual analysis, and automatic tests/security reviews, but we can use the system as a generic platform for performing distributed operations on intelligent triggers.

For example, we could use the system as a distributed platform for generating the API documentation seen on api.drupal.org. The system would automatically rebuild the documentation on commit and due to its distributed nature might even allow for the expansion of api.drupal.org for contrib projects (need to talk to infrastructure).

An advanced plugin API is provided for implementing additional functionality for the system. I have created an easy to extend Drupal specific implementation that should make it easy to build Drupal specific additions. Please contact me in IRC/email or the issue queue if you are interested in developing a plugin.

The future and beyond
I have identified items that I would like to have completed for the next installment of the system, created issues for them, and provided a summary below. I would appreciate any help from the community in completing them.

Project Issue File Review (PIFR) (qa.drupal.org & clients)

• Display "postponed" tests in queue graph somehow
• Standardize and clarify plugin arguments
• Check dependencies recursively when postponing test due to branch failure
• Restrict pifr_server_test_get_since() to the project client making the request
• Listing projects and current status
• Change admin/pifr/server/* to admin/pifr/client/*
• Change 'pifr administer' permission to 'pifr administrator'
• Provide 7.x coder style checks
• Provide mechanism for limiting environment (db) based on environment arguments

Project Issue File Test (PIFT) (drupal.org)

• Provide additional options for testing
• Retest branches automatically on time and commit trigger
• Provide e-mail notification of test results
• Integrate branch tests with project page/releases
• Per project branch filter
• Ensure/Refactor cron_retest() query to only re-test the last file on an issue
• When tests fail set Issue tag to "Failed Testing"

Project Info (drupal.org)

• Parse project .info files: present module list and dependency information

You may or may not have noticed the launch of the second iteration of the Drupal automated testing system last Thursday. I am very excited to have the system up and running and by the possibilities it opens up.

In order to help others get a feel for some of the new features that it provides and how the features can be used I created a screencast covering some of visual changes and features provided to end-users.

This Summer I will be working part-time as an intern for Acquia. I am very excited to be working with Acquia and having the chance to spend more time improving things that I have interest in. To clarify I will be working on projects that benefit the entire Drupal community. The items I will be working on are improvements to projects I have either started or that I am heavily involved with.

During the discussion of the internship I came up with the following goals that were then prioritized by Dries.

Primary goals

• Finalize testing of contributed modules and Drupal 6.x projects/core.
• Add executive summary of test results on project page.
• Extend the SimpleTest framework so we can test the installer and update/upgrade system.
• Improve and organize SimpleTest documentation
• Work on general enhancement of Drupal 7 SimpleTest.

Secondary goals

• Provide on-demand patch testing environment for human review of patches.
• Finish refactoring of SimpleTest to allow for a clean implementation of "configuration" testing.
• Analyze current test quality and code coverage, and foster work in areas requiring attention.

I will post updates on some of the more interesting items as they are accomplished. Additionally, I would like to give a special thanks to Kieran Lal for his mentoring and help in finding me sponsorship.

FOLLOW UP:
To clarify I will still be participating in Google Summer of Code 2009, which was explicit in my agreement with Acquia.
Follow up post by Dries.

I decided to pull gather some statistics about the automated testing system. These statistics were collected on Wednesday, May 6, 2009 at 4:00 AM GMT. Automatic generation of these statistics along with analysis is a feature I have in mind for ATS 2.0. I appreciate donations to the chipin (right), as this project requires a lot of development time.

From the data you can see that the test slaves have been running tests for the equivalent of 200 days. The system has been running for 192 days and not all the data was included since some of it is inaccurate. That means the system has saved 200 days of developer's time! It is clear that the ATS is a vital part of test driven development. Additionally the time that would have been spent fixing regressions and new bugs has been drastically lowered.

One item you may notice is the maximum test exception count of 813,795. The patch that caused that many exceptions proved that our system is scalable! The patch is much appreciated. :)

Saved the current test result breakdown.

The average test run length for all the active test slaves can be seen below. This data is only looking at the latest test run for each patch in the system.

* Excludes test runs that do not pass initial checks and fail before running test suite.

Over the next few weeks I plan to make a number of posts about the new features provided by ATS 2.0 and the benefits to the community. Currently, the system is in the final stages of deployment, but is not yet active. Please be aware that these features will be available once ATS 2.0 has been deployed. I appreciate donations to the chipin (right), as this project requires a lot of development time.

Server management

One of the major restraints holding back the expansion of the system has been the need to manually oversee the array of testing servers. The new system contains a number of enhancements to make it not only easier to manage the network, but also automates the task of adding new clients.

Upon request to enable client a set of error cases are sent to the client with expected results.

The most important addition that makes all this possible is the automatic client testing. Clients are automatically tested to ensure they are functioning properly. This is done through a set of tests that are sent to each test client with an expected result. The results the client sends back and compared with the expected result and that information is used to determine if the client is functioning properly. Clients are tested on a regular basis to ensure that they continue functioning as expected.

Another helpful change has been re-working the underlying architecture to use a pull based protocol instead of a push based protocol. This alleviates the issues caused when a client is unreachable for a period of time, or is removed without notice.

Public server queue

Very simple screen that allows users to add a test client to the network.

Another improvement that will facility a larger testing network is the public server queue. Allowing anyone to add a server to the network is possible since the clients are automatically tested as described above.

The interface has been designed so that users may control the set of machines that they have added to the network. The system automatically assigns the client a key that must be stored on the client and is used for authentication. The process of adding a client to the master list is very simple and should provide an easy way for users to donate servers.

If the system detects any issues with the client down the line, such as becoming out of date, it will notify the server administrator of the problem and disable the test client. The system will continually re-test the client and re-enable it automatically if it passes inspection. Alternatively, the server administrator may request the client to be tests immediately after fixing the issue.

Multiple database support

The new system has been abstracted to allow for the support of PostgreSQL and SQLite in addition to MySQL. This is vital to ensure that the Drupal 7 properly supports all three databases. Just as patches are not committed until they pass all the tests, patches will not be committed until after passing all the test on all three databases (5 environments with the database variations).

During the last several months I put a substantial amount of work into improving the Automated Testing System. Future posts will describe the exciting new features and the benefits to the community. If interested, a brief overview of some of the requirements can be found in the PIFR and PIFT issue queues.

For additional background, the original thoughts can be found at the following links:

• The Future of Automated Patch Testing
• Testbed design change and development
• Automated Testing System Development Clarification

Final steps
There a number of steps that need to be completed before the Drupal community can reap the benefits of the new system.

1. Security review of rewritten Project Issue File Test (PIFT) module that integrates with the project module on drupal.org.
2. Someone familiar with SQLite, possibly one of the D7 maintainers, needs to write a PIFR DB driver to implement the required methods. MySQL and PostgresSQL have already been completed and can be used as examples. The driver is relatively simple, but will require manually connecting to SQLite since PIFR runs in D6 which does not support SQLite.
3. Update testing client setup/installation script where necessary.
4. Deploy current development system to project.drupal.org and the create a parallel testing client network.
5. Freeze the current test client network and extract the test ID map for use in drupal.org upgrade.
6. Upgrade and finalize test client network and test server (testing.drupal.org). Possibly move testing.drupal.org under the drupal.org infrastructure.
7. Confirm upgraded testing network is functional.
8. Plan for approximately 15 minutes of downtime on drupal.org.
9. Update PIFT code and run data update using extracted test ID map from #4 on drupal.org during downtime.
10. Watch deployed system closely and solicit community feedback and bug reports.
11. Request additional hardware to use as community test clients (to allow for future expansion into testing contributed modules).

Future
Once the second generation framework is in place and running smoothly I will begin work on finishing the last pieces required to allow for testing of contributed modules (D6 and D7) and Drupal 6 core. I will be writing more on the new features and UX improvements to be looking for in during the upcoming deployment.

I will be presenting Saving webchick time - the saga along with Kieran Lal at Drupalcon DC 2009. To quote the session abstract:

One of the major enhancements made to the Drupal development cycle has been the addition of a fully automated testing bot, built on the testing framework in Drupal 7.

This session will focus on automated testing as it relates to Drupal 7: its history and direction, the automated testing bot: what has gone into it and where the future leads, and most importantly what is the end gain to the Drupal community.

The framework has gone through a rather long and interesting history with a number of road-blocks and challenges that have been overcome. The session will tell the story of the framework and the benefits it provides to the community through the enhanced Drupal 7 development work-flow. Although the session will go into some technical details it will overall tell the story of the framework and where we plan to take it. The presentation should be interesting to most and provide a great time to throw out any comments or concerns.

After the presentation I will stick around to discuss our recent launch of the Boombatower Testing Service (BTS). If you have any questions, comments, or concerns please drop by.

After my most recent post concerning the Testbed design change and development I received a number of comments that lead me to believe that the post was misunderstood. The comments suggest that the readers believe my primary focus of development will be changing from a push to a pull model. The post seems to have been misleading in that regard.

The change to a pull architecture is a very minor change in terms of coding, only 2 functions are even effected, but in terms of managing the network it helps out a lot. The majority of the development, that I am raising $3000 for, will be focused on implementing the ideas described in The Future of Automated Patch Testing. The feature additions will:

• Give more control to server administrators (donated servers)
• Make it easier to manage the automated testing framework
• Allow for testing of multiple environment configurations
• Open up testing of Drupal 6 core and contrib code.

I hope this has cleared up any misconceptions with the automated testing system as it stands and my plans for its feature.

I appreciate any donations.