After several days of battling the server and the enormous XML file generated by exporting the assertion data a solution was found. Instead of dumping all the results the framework only stores summary results. Just enough to tell you where the tests failed, but at the same time preserving the server.

Intended workflow

After a developer notices that there patch unexpectedly failed they will run the failed tests on their dev environment to find out what went wrong and work from there. This keeps the debugging work in the developers environment instead of on testing.drupal.org.

Patch criteria

One of the remaining issues pertains to what patches should be tested. The following represents what I believe we need from reading the issues and pondering. Some of these are already implemented, but listed for clarity.

  • File must have patch or diff extension.
  • Issue is for Drupal 7.
  • Issue is marked as patch (code needs review) or patch (reviewed & tested by the community).
  • Issue is not in documentation component.
  • Do not test flag. Something in filename like D6_*.patch if in Drupal 7 issue, like a port. And possible NOTEST_*.patch or a component? This is the part that needs to be figured out.

If the patch meets the criteria then it will be eligible to be tested and at some point sent for testing.

Patch sending event

Patches need to be sent out frequently to ensure they are tested as quickly as possible. To do this drupal.org will be setup to send files to be tested every 5 minutes.

In addition once daily, every other day, or some similar pattern the entire issue queue should be searched for patches that match the above mentioned criteria, regardless if they have already been tested, and send them all for testing. This will ensure that “patch rot” or patches that no longer apply after other changes are committed are automatically detected.

Reporting back

The most exciting and useful feature of this entire endeavor is being saved until last. Once we have all the things mentioned above in place and testing.drupal.org running smoothly then reporting back will be turned on.

What reporting will do is after a patch has been tested the results will be sent back to drupal.org. Code running on drupal.org will then check the issue related to the file results sent back to see if there is a newer patch. If there is the results will be ignored. If not then the issue status may be changed to reflect the result of testing.

In addition I envision having the results for each file listed on their comment in a summarized form with a link to the full results on testing.drupal.org.

Roadmap

In order to make this a reality a standard needs to be decided on for the patch criteria, either confirming/appending to the above, or something new. Once that is complete then the code needs to be written and placed on drupal.org.

After that is complete testing.drupal.org should be monitored to ensure it works properly and consistently. In the meantime work will begin on the code related to reporting back to drupal.org.

Community help

It would be very much appreciated if the community would check the results of testing on their patches to Drupal 7 core. This can be done be visiting the following page.

http://testing.drupal.org/pifr/node/1/[node_id]

Make sure that the testing results work and that your patch is tested if it meets the criteria. Please check even if your patch should not be tested, to ensure that it isn’t.

Thanks for your help and post to the queue if you find problems.