Boombatower Blog

Many of you may be familiar with my previous post Diaries of a Core Developer and that sadly nothing has changed. We had a lot of good discussion in the comments, but as far as I understand nothing has been done to improve the development workflow.

Drupal QA has and is currently having issues due to a complex problem with the core design of SimpleTest. I documented this problem some time ago, and made several attempts to refactor SimpleTest (before and after). The patches were quite large since they did a proper overhaul of the system. I was told to break them into smaller pieces so they would be easier to review, but after discovering how intertwined the code was and not receiving any other real feedback I was burned out after a week of working on the patch (and the lack of time others were willing to spend).

Just as the database layer and other large changes had large patches and lots of follow-up patches, even SimpleTest itself was introduced in a large patch, so does this refactoring. I briefly described what needs to be done and some of the benefits in the original issue. You can find more details on the refactoring in one of my later consolidation issues in which I attempted to explain the design and work through the issue.

As I explained in the issue there are some hackish ways we can work around the problem for the time being. I am happy to implement those fixes, given that I am not wasting my time. I still believe an overall refactoring is in order to fix a number of problems including proper isolation of test processes and a clean way of gather errors.

Before I commit a large amount of time, again, to refactoring the system I would like confirmation that someone will actually review the patch, those involved are fine with the changes, and that we will see this through to be committed (preferably some assurance from Dries of webchick). Of course this leads back the fact that I have no authority as a maintainer to make this crucial decision and that the core maintainers are bogged down reviewing ALL patches, instead of distributing the load to the sub-maintainers and core maintainers reviewing sub-maintainers patches and overall changes.

Considering:

• the last official backport was April 23 2009
• a number of very cool features have been added in core in addition to incremental changes
• people have been asking
• Drupal 7 is in "code slush"

it seems appropriate to perform another backport. I finished the bulk of the backport during Drupalcon Paris and have been tweaking and fixing bugs from feedback since then.

In order to ensure that all the new features from Drupal 7 were available it was necessary to create a patch against Drupal 6 core which needs to be applied before installation, as described in INSTALL.txt.

Please update and report any issues in the queue and have fun with the new features and proper error reporting!

Recently, I have made two major improvements to debugging in Drupal 7, the addition of a general debug function and a verbose mode for SimpleTest. The two additions make it much easier to debug problems quickly through the use of a consistent method. Take a look at what chx said via twitter:

Writing #drupal code? Check the new function debug(). Writing #drupal tests? Check $this->verbose(). And debug() works too. AWESOME!

General debug function
The general debug function can be used at any point after Drupal is bootstrapped, although the limitation may be removed in the future. The function provides a very simple wrapper to dump data through the use of var_export() and print_r(). When used normally it will display data based on the "Logging and errors" settings provided in Drupal 7 core. If using a dev version the debug information will be displayed using drupal_set_message() as shown in the screenshot below.

The exciting part about the new debug() function is that it also works during testing. The debug() function can be placed inside the test itself or in any other part of Drupal and it will be picked up and displayed in the test results as shown below.

SimpleTest verbose mode
Another exciting new debugging tool that is extremely useful when writing tests is the new verbose mode for Drupal 7 SimpleTest. The verbose mode can be enabled on the SimpleTest settings page.

Once enabled SimpleTest will automatically record the page as it was seen by the SimpleTest browser after each drupalGet() and drupalPost() call. A link is then placed in the test results that will display the page the browser saw and some meta data related to the request.

Page 1

Page 2

Manual verbose
In addition to the automatic message provided by SimpleTest custom verbose data may be dumped using DrupalWebTestCase->verbose() which can be used in a test as shown.

If the data to be dumped in not available in the test, but in the code being tested a function is provided that may be accessed by including the DrupalWebTestCase as shown below.

Summary
By adding these debugging tools to Drupal 7 the developer experience involved in writing a test has been greatly improved. These methods can still be improved and as such please feel free to file issues in the Drupal 7 SimpleTest issue queue. Also note that this work was sponsored by Acquia as part of my Summer Internship.

This Summer I will be working part-time as an intern for Acquia. I am very excited to be working with Acquia and having the chance to spend more time improving things that I have interest in. To clarify I will be working on projects that benefit the entire Drupal community. The items I will be working on are improvements to projects I have either started or that I am heavily involved with.

During the discussion of the internship I came up with the following goals that were then prioritized by Dries.

Primary goals

• Finalize testing of contributed modules and Drupal 6.x projects/core.
• Add executive summary of test results on project page.
• Extend the SimpleTest framework so we can test the installer and update/upgrade system.
• Improve and organize SimpleTest documentation
• Work on general enhancement of Drupal 7 SimpleTest.

Secondary goals

• Provide on-demand patch testing environment for human review of patches.
• Finish refactoring of SimpleTest to allow for a clean implementation of "configuration" testing.
• Analyze current test quality and code coverage, and foster work in areas requiring attention.

I will post updates on some of the more interesting items as they are accomplished. Additionally, I would like to give a special thanks to Kieran Lal for his mentoring and help in finding me sponsorship.

FOLLOW UP:
To clarify I will still be participating in Google Summer of Code 2009, which was explicit in my agreement with Acquia.
Follow up post by Dries.

I maintain a backport of Drupal 7 SimpleTest for use with Drupal 6 as the SimpleTest module 2.x branch. A fair amount of work goes into maintaining the code even though it is a backport.

Tonight I finished up a fresh backport of Drupal 7 and made a release along with a number of other changes. Please update to SimpleTest 6.x-2.8 and report any issues you find.

Note to developers

Since this release includes a fresh backport it also contains an API change that you need to be aware of. All getInfo() methods need to be changed to static before using this release. If you are a module developer or maintain tests internally please make sure you update them.

Old

New

I will be presenting Saving webchick time - the saga along with Kieran Lal at Drupalcon DC 2009. To quote the session abstract:

One of the major enhancements made to the Drupal development cycle has been the addition of a fully automated testing bot, built on the testing framework in Drupal 7.

This session will focus on automated testing as it relates to Drupal 7: its history and direction, the automated testing bot: what has gone into it and where the future leads, and most importantly what is the end gain to the Drupal community.

The framework has gone through a rather long and interesting history with a number of road-blocks and challenges that have been overcome. The session will tell the story of the framework and the benefits it provides to the community through the enhanced Drupal 7 development work-flow. Although the session will go into some technical details it will overall tell the story of the framework and where we plan to take it. The presentation should be interesting to most and provide a great time to throw out any comments or concerns.

After the presentation I will stick around to discuss our recent launch of the Boombatower Testing Service (BTS). If you have any questions, comments, or concerns please drop by.

I am sure by now most, if not all, are aware of the drupal.org upgrade to Drupal 6 and may have noticed some of the changes. After the upgrade was completed I went ahead and posted my updated hook_test() patch to Create hook_test(): move SimpleTest getInfo() out of test cases. Upon posting I noticed that the file was uploaded to:

http://drupal.org/files/

instead of

http://drupal.org/files/issues/

That may not sounds like much of an issue, but it has a number of trickle effects.

• Files are not renamed properly. Meaning that files with the same name may exist, but in different directories.
• Inconsistent data would be sent to testing.drupal.org that would have caused issues.
• Confusing urls.

I talked with Chad "hunmonk" Phillips about it in IRC and discovered that is was due to the new File API. Drupal.org was put in maintenance mode, as many of you probably noticed, and Chad dove into the code. After discussing for a while a "fix" was created.

The moral of the story...yet another bug found due to SimpleTest (the patch I was posting was related to SimpleTest). That of course is not to say it would not have been found soon enough, but SimpleTest indirectly found it first. :)

Situation
The SimpleTest browser that is included in the Drupal 7 core is very powerful and I have found myself hacking in order to use it outside of SimpleTest on a number of occasions, as have others. In addition the DrupalWebTestCase is becoming rather bloated. There is an issue to create pluggable backends for drupal_http_request which could be cleanly done by abstracting the SimpleTest browser. The change would create a clean and more powerful API as well as open up a number of possibilities by having a browser built into Drupal core and thus is something worth working on.

Solution
A while back I did some work towards abstracting the SimpleTest browser, not only out of DrupalWebTestCase, but improving/cleaning the code and writing it with a pluggable backend layer. The code is quite far along and just needs a bit more refinement before it can replace the existing SimpleTest browser. The next step would be to replace drupal_http_request or make it a wrapper. After that, areas of core that parse HTML and such can be cleaned up and simplified using the browser.

Possibilities
With a browser in core there are a number of powerful tools that can very easily be added. For instance install and update scripts that can trigger remote servers!! Think of having a single script on a development machine that runs the update script on all specified sites!

Installing via a script can be very usefull not only for the obvious, but also for modules like the Usability Testing Suite and Project Issue File Review that have to make separate installations of Drupal and currently must maintain their own install scripts. Having an abstracted installation script is necessary for testing.drupal.org to be able to function with patches that make changes to the installer.

I plan to work on this further, but have been busy with other projects lately and my recent video card experience. I would like to hear some feedback from the community and possibly recruit some help.

Related issues

• Pluggable architecture for drupal_http_request()
• Abstract SimpleTest browser in to its own object
• Create auto-install script

After my most recent post concerning the Testbed design change and development I received a number of comments that lead me to believe that the post was misunderstood. The comments suggest that the readers believe my primary focus of development will be changing from a push to a pull model. The post seems to have been misleading in that regard.

The change to a pull architecture is a very minor change in terms of coding, only 2 functions are even effected, but in terms of managing the network it helps out a lot. The majority of the development, that I am raising $3000 for, will be focused on implementing the ideas described in The Future of Automated Patch Testing. The feature additions will:

• Give more control to server administrators (donated servers)
• Make it easier to manage the automated testing framework
• Allow for testing of multiple environment configurations
• Open up testing of Drupal 6 core and contrib code.

I hope this has cleared up any misconceptions with the automated testing system as it stands and my plans for its feature.

I appreciate any donations.

Please see clarification.

The next goal for the testing system (centered at testing.drupal.org) is to test Drupal 6, Drupal 6 contrib, and eventually Drupal 7 contrib. This will require:

• Additional testing computers to be added to the current fleet of 4-6 servers which handles the Drupal 7 core load
• Re-architecting the automated testing system as described below
• The addition of new features described in my previous post, The Future of Automated Patch Testing

In order for me to be able to devote significant resources in the short term to get this completed it would very helpful to have funding. I would like to raise $3,000 for the development of the new system.

Before describing the details of the new system, lets look at how the current system works.

Current system

The automated testing system has three stages, or sections, to it. The sections are illustrated in the figure to the right, but I'll give an overview of what they do.

1. The project server that manages all the issues related to projects. - drupal.org
2. The testing master server which distributes the testing load and aggregates the results to the appropriate project server. - http:testing.drupal.org
3. The testing server that performs the patch review and reports back to the master server. - network of servers

New architecture

The automated testing system is classified as a push architecture. That means that the testing master server pushes (ie. sends) patches to each of the testing servers. The new architecture will be based on a client/server model where the test client pull (ie. request) patches to test from the server.

In addition to changing the basic architecture of the system the large feature list will also be implemented. After all is said and done the new system will provide a more powerful tool to the Drupal community.

Reasoning

The motivation for making the change is due to the difficulties we have run into while maintaining the system for the last few weeks.

• Adding a testing server requires the entire system to be put on "hold".
• There is not an automated way to confirm that a testing server is functioning properly.
• When a testing server becomes "messed up" or "un-reachable" the system does not automatically react.
• Server administrators cannot simply remove their server from the fleet; they must first inform us that they are doing so.

All this manual intervention adds up to a large amount of time that Chad "hunmonk" Phillips and I have to spend to keep the system running smoothly. As we look to add more servers our time commitment will increase to the point where it is not feasible to maintain the system, thus the changes are required.