drupal

Testbed design change and development

Please see clarification.

The next goal for the testing system (centered at testing.drupal.org) is to test Drupal 6, Drupal 6 contrib, and eventually Drupal 7 contrib. This will require:

  • Additional testing computers to be added to the current fleet of 4-6 servers which handles the Drupal 7 core load
  • Re-architecting the automated testing system as described below
  • The addition of new features described in my previous post, The Future of Automated Patch Testing

In order for me to be able to devote significant resources in the short term to get this completed it would very helpful to have funding. I would like to raise $3,000 for the development of the new system.

Before describing the details of the new system, lets look at how the current system works.

Current system

System flow

The automated testing system has three stages, or sections, to it. The sections are illustrated in the figure to the right, but I'll give an overview of what they do.

  1. The project server that manages all the issues related to projects. - drupal.org
  2. The testing master server which distributes the testing load and aggregates the results to the appropriate project server. - http:testing.drupal.org
  3. The testing server that performs the patch review and reports back to the master server. - network of servers

New architecture

The automated testing system is classified as a push architecture. That means that the testing master server pushes (ie. sends) patches to each of the testing servers. The new architecture will be based on a client/server model where the test client pull (ie. request) patches to test from the server.

In addition to changing the basic architecture of the system the large feature list will also be implemented. After all is said and done the new system will provide a more powerful tool to the Drupal community.

Reasoning

The motivation for making the change is due to the difficulties we have run into while maintaining the system for the last few weeks.

  • Adding a testing server requires the entire system to be put on "hold".
  • There is not an automated way to confirm that a testing server is functioning properly.
  • When a testing server becomes "messed up" or "un-reachable" the system does not automatically react.
  • Server administrators cannot simply remove their server from the fleet; they must first inform us that they are doing so.

All this manual intervention adds up to a large amount of time that Chad "hunmonk" Phillips and I have to spend to keep the system running smoothly. As we look to add more servers our time commitment will increase to the point where it is not feasible to maintain the system, thus the changes are required.

The Future of Automated Patch Testing

Over two years of development has lead to testing.drupal.org becoming a reality. The testbed has been active for almost two months with virtually no issues related to the automated testing framework. That is not to say the bot has not needed to be disabled, but instead that the issues were unrelated to the automated testing framework itself.

The stability of the framework has lead to the addition of over 6 testing servers to the network, with more in the works. Increasing the number of testing servers also means an increased load required to manage the testing network. A fair amount of labor is needed to keep the testing system running smoothly and to watch for Drupal 7 core bug introductions.

Having the testing framework in place has saved patch reviewers, specifically the Drupal 7 core maintainers: Dries and webchick, countless hours that would otherwise have been spent running tests. The automated framework also ensure that tests are run against every patch before it is committed. Ensuring tests are always run has lead to a relatively stable Drupal 7 core that receives updates frequently.

Based on the overwhelming positive feedback and personal evaluation of the framework a number of improvements have been made since its deployment. The enhancements have, of course, lead to further ideas which will make the automated testing system much more powerful than it already is.

Server management

A number of additions have been made to make it easier to manage the testing fleet. The major issue that remains is that of confirming that a testing slave returns the proper results. The process currently requires manual testing and confirmation of the results. The process could be streamlined by adding a facility that would send a number of patches with known results to the testing slave in question. The results returned would then be compared to the expected results.

The system could be used at multiple stages during the testing process. Initially when a testing slave is added to the network it would be automatically confirmed or denied. Once confirmed the server would begin receiving patches for testing. Periodically the server would then be re-tested to ensure that it is still functioning. If not the system would automatically disable the server and notify the related server administrator.

Having this sort of functionality opens up some exciting possibilities as described bellow.

Public server queue

Once the automated server management framework is in place the process of adding servers to the fleet could be exposed to the public. A server donor could create an account on testing.drupal.org, enter their server information, and check back for the results of the automated testing. If errors occur the server administrator would be given common solutions in addition to the debugging facilities already provided by the system.

If the server passes inspection an administrator of testing.drupal.org would be notified. The administrator would then confirm the server donation and add the server to the fleet. Once in the fleet the server would be tested regularly like all the other servers. If the donor wishes to remove their server from the fleet they would request removal of their server on testing.drupal.org. The framework would let any tests finish running and then remove the server automatically.

A system of this kind would provide a powerful and easy way to increase the testing fleet with minimal burden to the testing.drupal.org administrators. Having a larger fleet has a number of benefits that will be discussed further.

Automated core error detection

Automatically testing an un-patched Drupal HEAD checkout after each commit and confirming that all tests pass would ensure that any human mistakes made during the commit process do not cause false test results. In addition to testing the core itself the detection algorithm would also disable the testing framework if drupal.org is unavailable. Currently when drupal.org goes down the testing framework continues to run which causes errors due to the patches not being accessible. Having this sort of system in place would be a great time saver for administrators and ensure that the results are always accurate.

There is currently code in place for this, but it needs expanding and testing.

Multiple database testing

Drupal 7 currently supports three databases and there are plans to support more. Testing patches on each of the databases is crucial to ensure that no database is neglected. Creating such as system would require a few minimal changes to the testing framework to store results related to a particular database, send a patch to be tested on each particular database, and display the results in a logical manor on drupal.org.

Patch reviewing environment

In addition to performing patch testing the framework could also be used to lower the barrier required to review a patch. Instead of having to apply a patch to a local development environment a reviewer would simply be required to press a button on testing.drupal.org after which he/she would be logged into an automatically setup environment with the patch applied.

This sort of system would save reviewers time and would make it much easier for non-developers to review patches, especially for usability issues.

Code coverage reports

Drupal 7 strives to have full test coverage. What that means is that the tests check almost every part of the Drupal core to ensure that every works as intended. It is rather difficult to gage the degree to which core is covered without the use of a code coverage reporting utility. Setting up a utility of that kind is no small task and getting results requires large amounts of CPU time.

The testing framework could be extended to automatically provide code coverage reports on a nightly basis. The reports can then be used, as they have been already, to come up with a plan for writing additional tests to fill the gaps.

Performance ranking

Since the tests are very CPU intensive having a good idea of the performance of a particular testing slave would be useful for ordering which servers are sent patches first. Ensuring that patches are always sent the fastest available testing server will ensure the quickest turn-around of results. The testing framework could automatically collect performance data and use an average to rank the testing server.

Standard VM

Creating a standard virtual machine would have a number of benefits: 1) eliminate most configuration issues, 2) provide consistent results, 3) make the processing of setting up a testing slave easier, and 4) make it possible for one testing server to test patches on different databases. Several virtual machines are currently in the works, but a standard one has yet to be agreed upon.

Benefits

Drupal is somewhat unique in having an automated system like the one in place. The system has already proven to be a beneficial tool and with the addition of these enhancements it will become a more integral part of Drupal development and quality assurance. Maintaining the system will be much easier, reviewing core patches will be simpler, and the testing fleet can be increased in size much more easily.

With a larger testing fleet the testing framework can be expanded to test contributed modules. In addition the framework can be modified to support testing of Drupal 6 and thus enable it to test the large number of contributed modules that have tests written. Having such a powerful tool available to contrib developers will hopefully motivate more developers to write tests for their modules and in doing so increase the stability of contributed code base.

The automated testing framework is just beginning its life-cycle and has already proven its worth, with enhancements like the ones discussed above the framework can continue to provide new tools to the Drupal community.

Keep core clean: issues will be marked CNW by testing framework

The final updates to the testing framework have been placed on drupal.org. The updates will allow issues to be marked as code needs work when the latest eligible patch fails testing.

After a bit of work fixing the latest bugs in core in order to make all tests pass, the testing system has been reset and should now be reporting back to drupal.org again, but this time marking issues as CNW!

Current Drupal 7 issue queue status:

267 Pending bugs (D7)
241 Critical issues (D7)
1116 Patch queue (D7)
388 Patches to review (D7) 

I hope to see some changes.

On the other side this means if a bug is committed to core that causes the tests to fail all patches tested will be marked as CNW.

Make sure you wait for testing results on a patch before you mark it as ready to be committed.

Testing results on drupal.org!

Today testing.drupal.org began sending results back to drupal.org! The results are displayed under the file they relate to. The reporting will not trigger a change in issue status if the testing fails, as is the future plan. That will be turned on when the automated testing framework has proven to be stable.

The following is an example of what a pass result look like on drupal.org. (http://drupal.org/node/180379#comment-739254)
pass example

The following is an example of what a fail result look like on drupal.org. (http://drupal.org/node/200185#comment-752032)
fail example

Testing.drupal.org also provides detailed stats on patch results and status. An example of patch result stats is as follows.
stats example

Early success
Before the actual reporting was turned on the automated testing framework discovered two core issues. One was a partial commit that caused a PHP syntax error. The other was a bug with the installer. I hope to see the results make patch reviewing much faster and easier.

Your help
Please report any test result anomalies in the PIFR queue.

Testing.drupal.org is running!

After several days of battling the server and the enormous XML file generated by exporting the assertion data a solution was found. Instead of dumping all the results the framework only stores summary results. Just enough to tell you where the tests failed, but at the same time preserving the server.

Intended workflow
After a developer notices that there patch unexpectedly failed they will run the failed tests on their dev environment to find out what went wrong and work from there. This keeps the debugging work in the developers environment instead of on testing.drupal.org.

Patch criteria
One of the remaining issues pertains to what patches should be tested. The following represents what I believe we need from reading the issues and pondering. Some of these are already implemented, but listed for clarity.

  • File must have patch or diff extension.
  • Issue is for Drupal 7.
  • Issue is marked as patch (code needs review) or patch (reviewed & tested by the community).
  • Issue is not in documentation component.
  • Do not test flag. Something in filename like D6_*.patch if in Drupal 7 issue, like a port. And possible NOTEST_*.patch or a component? This is the part that needs to be figured out.

If the patch meets the criteria then it will be eligible to be tested and at some point sent for testing.

Patch sending event
Patches need to be sent out frequently to ensure they are tested as quickly as possible. To do this drupal.org will be setup to send files to be tested every 5 minutes.

In addition once daily, every other day, or some similar pattern the entire issue queue should be searched for patches that match the above mentioned criteria, regardless if they have already been tested, and send them all for testing. This will ensure that "patch rot" or patches that no longer apply after other changes are committed are automatically detected.

Reporting back
The most exciting and useful feature of this entire endeavor is being saved until last. Once we have all the things mentioned above in place and testing.drupal.org running smoothly then reporting back will be turned on.

What reporting will do is after a patch has been tested the results will be sent back to drupal.org. Code running on drupal.org will then check the issue related to the file results sent back to see if there is a newer patch. If there is the results will be ignored. If not then the issue status may be changed to reflect the result of testing.

In addition I envision having the results for each file listed on their comment in a summarized form with a link to the full results on testing.drupal.org.

Roadmap
In order to make this a reality a standard needs to be decided on for the patch criteria, either confirming/appending to the above, or something new. Once that is complete then the code needs to be written and placed on drupal.org.

After that is complete testing.drupal.org should be monitored to ensure it works properly and consistently. In the meantime work will begin on the code related to reporting back to drupal.org.

Community help
It would be very much appreciated if the community would check the results of testing on their patches to Drupal 7 core. This can be done be visiting the following page.

http://testing.drupal.org/pifr/node/1/[node_id]

Make sure that the testing results work and that your patch is tested if it meets the criteria. Please check even if your patch should not be tested, to ensure that it isn't.

Thanks for your help and post to the queue if you find problems.

Final pieces to automated patch testing

I recently had a talk with hunmonk in IRC about what needs to be done to get drupal.org to start sending patches to the new automated testing framework. After almost an hour of discussion and thought we came to the following agreement. (summarized from IRC log)

Initial plan

  • PIFT which is already in place on drupal.org will continue to be used to scan the issue queue and send patches to testing.drupal.org. (May need criteria updated.)
  • PIFT client will then be ported to Drupal 6.x and benefit from the addition of a hook as apposed to the current mechanism.
  • PIFR will be updated to implement the new PIFT hook in the ported Drupal 6.x client.

Future plan

  • Reporting back to drupal.org will be discussed a bit further as there are several ways and opinions on how it should work both technically and from a workflow standpoint.
  • PIFR will send summary of testing results back to drupal.org through PIFT.
  • PIFT will display the results on drupal.org.

Possible future

  • PIFT may be further abstracted to allow for general distributed information sending between Drupal sites.
  • Reporting logic may be moved out of PIFT and into another module.

Nomenclature:

Automated patch testing (testing.drupal.org) realized

The idea of automated testing using SimpleTest has been around almost as long as the SimpleTest module itself. A movement to accomplishing this has been centered at testing.drupal.org and the following projects:

Testing.drupal.org

The goal is to have all patches against Drupal 7 core, over 1000 currently, automatically tested to ensure that they apply and pass the suite of tests which consists of over 5500 individual assertsions for about 70% code coverage. Completing this goal has been long in the making by a number of individuals, who have been focused on Drupal 4.7, 5, and 6, but now appears to be realized. Being heavily involved in SimpleTest development, I have taken on the task of finishing up the automated testing framework to be compatible with Drupal 7 and SimpleTest core.

Hurdles
The automated testing framework was designed to support Drupal 4-7 through a 5.x module. This approach worked well, but with the recent explosion in SimpleTest 7.x development the workflow and design has been changed.

Changes from previous SimpleTest version to 7.x:

  • The SimpleTest PHP library is no longer in use.
  • The XML reporter is no longer available, and is necessary for transferring results from test slave to testing master server.
  • The Shell script to run tests does not work properly in that it returns different results than the web interface.
  • The Drupal 7.x SimpleTest framework does not support META refresh and thus cannot run batch API pages necessary for running the tests and installing.
  • The Drupal 7 installer does not auto detect JavaScript support and does not work without JavaScript. (issue)
  • The XML output, over 2.7MB, is too large to transfer between servers and will only continue to grow with more tests.

Goalworkflow
The desired workflow is described below.

Steps

  1. A file is attached to an issue at drupal.org.
  2. The system determines if the file is a patch/diff and fits the proper criteria.
  3. The file and issue information is sent to testing.drupal.org.
  4. The file is added to the testing queue.
  5. When a testing slave is available the next file in the queue is sent to be tested.
  6. Once the slave has completed the testing the result is sent back to testing.drupal.org.
  7. The result is then store and then the next file in the queue is sent to the slave.

This will provide a very consistent way to test files that will allow for scaling in the event that additional servers are necessary to deal with the patch load.

Solution
The previous framework is described at http://testing.drupal.org. The new framework is based heavily off the old framework, but runs in Drupal 6, has been re-written, and is better suited for running tests against Drupal 7.

The re-written code resides as Project Issue File Review. The single module can act as any combination of the three roles: project server, testing master server, and testing slave. That means that the current single server that testing.drupal.org has available can act as both the testing master server and a testing slave. This setup can encompass internal business testing on a single server as well.

The module addresses the hurdles in several ways.

  • A patch is applied to Drupal 7 HEAD that converts the output from the standard HTML interface to an XML dump.
  • Instead of using the shell script the code uses SimpleTest with a patch that adds support for meta refresh.
  • When installing Drupal 7 HEAD the batch API is forced to use META refresh.
  • The XML output is gzipped to allow for it to transfered between slave and testing master server.

Advantages
Some may wonder why I re-wrote the existing code, there are several reasons.

  • Drupal.org is being redesigned in which it will also be ported to Drupal 6 so the testing framework might as well lead the way and not become a Drupal 5 dependency.
  • Solution that is tailored towards the SimpleTest 7.x workflow.
    • The previous framework kept each test installation around. Since SimpleTest no longer works on the active database all that would be seen is a base Drupal installation and is thus useless to keep around.
    • Removes SimpleTest library dependency.
    • Removes SimpleTest module download dependency during Drupal HEAD installation as it is now in core.
    • Code logic built around the community's desires for testing.drupal.org. (explained further bellow)
  • File results are displayed in relation to the node they originated from which makes for easy relation between drupal.org and testing.drupal.org. As apposed to the previous system that assigned a random ID to each file tested. This setup still allows for multiple project servers with the same issue node IDs.
  • Files are sent for review immediately after being posted. (further explained below)
  • I much prefer working in Drupal 6 and when going through the trouble of porting why not do a bit or clean-up/re-write.

What to expect
Summary of files per nodefile summary
To view a list of all the files reviewed for an individual node you would enter a URL like: http://testing.drupal.org/pifr/view/1/9 where 1 is the project server ID, could be replaced by textual "drupal.org", and 9 is the issue node ID. All the files that testing.drupal.org is aware of will be displayed along with their current status and testing result. The possible status values are: Queued, Sent to slave #, and Reviewed. This allows developers to check on a files testing status at any time. Although not displayed the comment ID is also stored for future auto commenting back on drupal.org issues.

Detailed test results
file summaryUpon clicking "View results" the detailed assertions will be displayed from the tests run. Currently this consists of a dump by test case. In the future this will hopefully look more the like the standard SimpleTest interface in Drupal 7. That will be handled in core when the test results layer is abstracted. I have already created a patch to begin this with the separation of test selection and test results. Once completed I will attempt to further cleanup the results display to work nicely for testing.drupal.org and any other modules that wish to integrate with SimpleTest. That will also allow this results page to be easily updated with the latest improvements to the SimpleTest results display.

Status summary
file summaryIn addition to checking the status on individual files the overall status of the queue and testing slaves can be viewed by entering the URL: http://testing.drupal.org/pifr/status. The operations will only be visible to administrators. This will give an a general feel for the load being put on the testing infrastructure and allow for easy reset of a slave server when they crashes.

What is left
There are several things left to be done, some of which require the community's involvement.

  • Drupal 7 HEAD does not pass all tests. That means that when testing.drupal.org is turned on all patches will fail. That defeats the purpose of testing.drupal.org to some degree. Anyone can help to fixing the tests or issues related to the failures. Sadly it used to have 100% passes.
  • Add meta refresh support to SimpleTest patch needs to make it in.
  • Rework the SimpleTest results interface. The clean-up of backend code needs to complete and committed. The abstratction of the testing results code is dependent on this clean-up.
  • Testing results code needs to be abstracted once the clean-up of backend code gets committed.
  • Determine the proper criteria and workflow for choosing what patches to test and how to report back. (more detail below)
  • Code review would be nice, but lets focus on critical issues and worry about minors ones once testing.drupal.org is up and running.

Patch criteria and workflow
This is the one area that needs to be flushed out. As many may have noticed when testing.drupal.org was turned on to simply test if patch applied to the current development branch, refereed to as HEAD, it created a number of issues. The following are issues that need to be resolved.

  • When should patches be sent to testing.drupal.org from drupal.org: right away or wait for cron?
  • If newer patch is posted when test results are returned should the auto comment still be sent?
  • Should the workflow be amended to have something like patch(tested) between patch(needs review) and patch(review & tested by the community).
  • What should comment text contain? Link to testing results, message, both, etc?

One advantage with sending patches right when they are posted is that the logic about when to comment back can be put on testing.drupal.org as opposed to having a shared more complex logic on both drupal.org and testing.drupal.org. Since all that is sent is the node ID, comment ID, and file URL it shouldn't be a load on drupal.org.

Comments are appreciated, but keep in mind that the code is still being developed and there are a few things I will add before attempting to install on testing.drupal.org. Part of which is a Drupal 5 piece of code to send files, I might just modify the previous framework for that.

SimpleTest 7.x core for Drupal 6

For those of you who don't follow Drupal 7 development, the SimpleTest module was committed to core. I have been maintaining the both core and contrib versions of the module and decided to back-port the enhancements made in core to contrib.

There are now two versions of SimpleTest available for Drupal 6.

  • 1.x: Old code base that requires SimpleTest PHP library.
  • 2.x: New code, with countless enhancements, back-ported from core.

Because of the back-ported 2.x version you can write your tests in the Drupal 7 format that is the future. It is highly recommenced that you convert tests to this new format as Drupal 7 will not support it and Drupal 5 and 6 may not support the old format much longer. Converting will also help because the documentation available is being converted to the new format and many new features are available.

If you are interested in starting to write tests or getting up to speed on recent development I have create a getting started document that will explains the different modules available and give you a brief history or changes.

If anyone is interested in helping me back-port the changes to 5.x that would be great. Better yet just get it working and submit a patch.

Usability Testing Suite Alpha 2

I have released the second alpha of the Usability Testing Suite. The API is fully functional and there are two plug-ins implementing the API. The reason the module is still marked as alpha is due to the user flow issues and clean-up items.

For more detailed information on the status of the project please read the alpha 2 release notes.

I would like to get feedback on what people think of the module, as it is intended to be used to enhance Drupal's usability.

Make sure you read the INSTALL.txt file before installing.

Tags:

Core patches need to be tested!

It has come to our, SimpleTest module developers, attention that the tests are breaking at a rapid rate. Most of the tests that currently have issues passed when they were originally committed to the core. The list of tests that passed was used during the cleanup stages, but hasn't been changed since SimpleTest was added to core and stands as a record.

I previously posted "Core patches need to update tests as well" in preparation for SimpleTest becoming part of core. Sadly it doesn't appear it was heeded.

After a long process of trial and error rolling back commits, webchick found the patch that broke the translation test, and possibly others. This would have saved everyone involved a lot of time had those who reviewed and wrote the patch simply run the tests on the patch, or better yet if testing.drupal.org were finished and implemented on drupal.org so that it would happen automatically.

Regardless it is rather annoying to spend several weeks cleaning up the tests to see them broken by un-tested commits. Before marking a patch as "ready to be committed" ensure that someone has run the suite of tests on it. If that hasn't been done I would hope that the patch isn't committed.

Pages

Subscribe to RSS - drupal