Automated Testing System 2.0 - New Features - Part 1

Over the next few weeks I plan to make a number of posts about the new features provided by ATS 2.0 and the benefits to the community. Currently, the system is in the final stages of deployment, but is not yet active. Please be aware that these features will be available once ATS 2.0 has been deployed. I appreciate donations to the chipin (right), as this project requires a lot of development time.

Server management

One of the major restraints holding back the expansion of the system has been the need to manually oversee the array of testing servers. The new system contains a number of enhancements to make it not only easier to manage the network, but also automates the task of adding new clients.

Client enable process
Upon request to enable client a set of error cases are sent to the client with expected results.

The most important addition that makes all this possible is the automatic client testing. Clients are automatically tested to ensure they are functioning properly. This is done through a set of tests that are sent to each test client with an expected result. The results the client sends back and compared with the expected result and that information is used to determine if the client is functioning properly. Clients are tested on a regular basis to ensure that they continue functioning as expected.

Another helpful change has been re-working the underlying architecture to use a pull based protocol instead of a push based protocol. This alleviates the issues caused when a client is unreachable for a period of time, or is removed without notice.

Public server queue

Add client
Very simple screen that allows users to add a test client to the network.

Another improvement that will facility a larger testing network is the public server queue. Allowing anyone to add a server to the network is possible since the clients are automatically tested as described above.

The interface has been designed so that users may control the set of machines that they have added to the network. The system automatically assigns the client a key that must be stored on the client and is used for authentication. The process of adding a client to the master list is very simple and should provide an easy way for users to donate servers.

If the system detects any issues with the client down the line, such as becoming out of date, it will notify the server administrator of the problem and disable the test client. The system will continually re-test the client and re-enable it automatically if it passes inspection. Alternatively, the server administrator may request the client to be tests immediately after fixing the issue.

Multiple database support

The new system has been abstracted to allow for the support of PostgreSQL and SQLite in addition to MySQL. This is vital to ensure that the Drupal 7 properly supports all three databases. Just as patches are not committed until they pass all the tests, patches will not be committed until after passing all the test on all three databases (5 environments with the database variations).

Comments

This is fantastic because now people have a brand new way to contribute to Drupal! They can provision a testing server and add it to the queue. Wonderful.

The next frontier is a set of performance benchmarks that also do performance testing and measure increases or decreases in performance of any patch.

Great work!

Performance of Drupal with a certain patch should be taken with a certain grain of salt, of course. That being that future patches can increase performance of certain subsystems. For example, the first Fields patches decreased performance significantly.

Thanks for the happy update, Jimmy.

Economist.com is working on automated performance testing of HEAD Drupal with pretty graphs and such. We are donating hardware and expertise to get this going for the community. netaustin is leading this effort.

Jimmy, thanks a lot for your work in getting Drupal tested automatically by ATS.

If ATS 2.0 provides an API for server managament (adding new clients, asking what kind of clients are needed the most); the process of adding new servers can be automated completely. In my imagination, I see amazon EC2 instances being added whenever there is a need for extra testing power.

I will for sure keep an eye on the progress of this project, and see if I can jump in to help building a framework that makes server management easier.