Git bisect saves the day

I have been working on a private project for quite some time using git. Yesterday, I noticed that one of the views on the site was taking around 10 seconds to generate instead of less than a second like it used to. I scanned through the recent commits, scratched my head, and messed with a bunch of stuff, but to no avail. I reverted to a commit from over a month ago just for kicks and sure enough the view rendered quickly again. So I decided it was time to learn how to use git bisect.

From git documentation:

Find by binary search the change that introduced a bug

I had read in passing the general idea behind git bisect and that you could use a script that returned pass or fail to automate the process. After reading through the man page I confirmed that a script may be used and simply needs to return exits status code 0 for pass and 1 for fail. So I figured I could write a script that manually executes the view and check to see if the amount of time required was over a certain threshold. Interestingly it seems the view executes much faster using the script than inside a normal page request. Thus the threshold used is much lower then one might expect.

I created two files since I wanted to use drush php-script and follow the documentation’s recommendation by placing the scripts outside the repository. The first script is a wrapper that simply changes to the directory in which Drupal is installed and then executes the drush command.

git_bisect.sh

#!/bin/bash
cd /path/to/drupal
drush php-script ~/check_view.php

check_view.php

drupal_flush_all_caches();

$view = views_get_view('MY_CUSTOM_VIEW');
$view->set_arguments(array(3)); // Test data.
$start = microtime(TRUE);
$view->execute();
$stop = microtime(TRUE);

// 0: pass, 1: fail
$diff = $stop - $start;
$status = $diff > 0.3 ? 1 : 0; // Threshold.

var_dump($diff);
var_dump($status);

// If exit(0) is called drush still views it as abnormal shutdown and sets code
// to non-zero so only call when we want abnormal shutdown.
if ($status != 0) {
  exit(1);
}

My case was a bit more complex since code beyond a certain point was incompatible since I had to backup a related module to work with old revisions.

078d60ad18b73ec356436a7ea30528c95c9c4844 (bad)
3f1cfca83821a6b2d694cf228e5d8af3db20922f (good)

I ran the following inside the repository directory.

git bisect start 078d60ad18b73ec356436a7ea30528c95c9c4844 3f1cfca83821a6b2d694cf228e5d8af3db20922f --
git bisect run ~/git_bisect.sh

I ended up with the following result (– indicates where I scrubbed data for privacy).

running /home/boombatower/git_bisect.sh
float(0.43191289901733)
int(1)
Drush command terminated abnormally due to an unrecoverable error.
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[50dcca7e9cec514c2bcc24156cd8b4622eb2cd3e] -- message --
running /home/boombatower/git_bisect.sh
float(0.53287100791931)
int(1)
Drush command terminated abnormally due to an unrecoverable error.
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[37e793d693a75a55470e6a92f5e3f30649ee2214] -- message --
running /home/boombatower/git_bisect.sh
float(0.18644404411316)
int(0)
Bisecting: 1 revision left to test after this (roughly 1 step)
[77ddb40b26fe2436cd7a15549109ffa9095d6995] -- message --
running /home/boombatower/git_bisect.sh
float(0.51487994194031)
int(1)
Drush command terminated abnormally due to an unrecoverable error.
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[08d99c98f4b9d837775db47770bc125727d93dc6] -- message --
running /home/boombatower/git_bisect.sh
float(0.1781919002533)
int(0)
77ddb40b26fe2436cd7a15549109ffa9095d6995 is the first bad commit
commit 77ddb40b26fe2436cd7a15549109ffa9095d6995
Author: --
Date:   --

    -- message --

:040000 040000 a4d3a8cb990d0eff7a7dd87c941a3f12b768feaf 10ab97d60a9b03a56862828e0f9f66cf7f4ef6b4 M      --
:100644 100644 7254a3027edfb35e10b948a9dcd994a9fbdd44a3 0a8ef37931373929328fe6458bf1f595549d265a M      --
bisect run success

Sure enough the “first bad commit” was indeed the commit that caused the performance issue. Very cool!

Drush, Drush Make, other Drupal packages, and development setup for openSUSE

I have recently added drush and drush make packages to my openSUSE repository. For more information or to report bugs on the packages please visit their respective project pages: drush and drush_make.

To install the packages you can use the one-click installers provided by the build service or manually add my repository and install the packages as shown bellow.

su
zypper ar http://download.opensuse.org/repositories/home:/boombatower/openSUSE_11.[2 or 3]/ home:boombatower
zypper in drush drush_make

Also note my existing Drupal packages: drupal-dev and drupal-vhosts, as well as the LAMP Drupal one-click pattern. The latter package (drupal-vhosts) is very useful in setting up a multi-drupal version, multi-subdomain work environment.

To use simply install and run the command to point the virtual hosts to the directory containing your Drupal code.

su
zypper in drupal-vhosts
drupal-vhost /path/to/main/software/directory

Either edit the hosts file directory or use YaST -> Network Services -> Hostnames to add an entry for every Drupal version you wish to run (package currently supports 6, 7, and 8). The relevant lines from my /etc/hosts file are as follows.

127.0.0.1       d7x.loc 
127.0.0.1       d6x.loc 

For my setup I use /home/boombatower/software for all my code with Drupal cores in drupal-7 and drupal-6 directories respectively. If you want to have subdomains for your sites just add more entries to /etc/hosts and use the respective Drupal sites directories.

Personally, I then create symbolic links to all my modules so that the code resides in the root of the software directory, but can be used by any respective site. This makes the paths to modules and what not much shorter and easier to reference from multiple specific sub-sites and what not. For example to link pathauto to the all modules directory for Drupal 7 I would execute the following.

ln -s ~/software/pathauto ~/software/drupal-7/sites/all/modules

Or from within the sites/all/modules directory as I tend to do.

ln -s ~/software/pathauto .

Also note, to enable mod_rewrite and get clean URLs to work simply go to YaST -> System -> /etc/sysconfig Editor then Network -> WWW -> Apache 2 -> APACHE_MODULES and add rewrite to the end of the line. You can do so manually of course as well.

In order for the virtual host changes and apache module addition to take effect you will need to restart apache and for the /etc/hosts changes you need to restart the network which you can do with the following commands run as root.

rcapache2 restart
rcnetwork restart

The end result of all this work is beautiful URLs like: http://d7x.loc/node/1, http://foo.d7x.loc/user, and http://d6x.loc/.

I also create a similar structure within MySQL. First, I set an easy to remember MySQL root password since there really no reason for it not to be easy to remember and it is helpful when having to enter it a lot.

mysqladmin -u root password EASY_TO_REMEMBER_PASSWORD

Next setup a drupal user in MySQL and give the user all permissions to d7x* and d6x* named databases which allows us to use a single user for all our drupal sites (much easier to remember login info) without having to update privileges all the time. I name my databases the same as virtual hosts, so for d7x.loc I would have d7x as the database name and for foo.d7x.loc I would have d7x-foo.

CREATE USER 'drupal'@'localhost' IDENTIFIED BY  'EASY_TO_REMEMBER_PASSWORD';
GRANT USAGE ON * . * TO  'drupal'@'localhost' IDENTIFIED BY  'SAME_EASY_TO_REMEMBER_PASSWORD' ;

GRANT ALL PRIVILEGES ON  `d7x%` . * TO  'drupal'@'localhost';
GRANT ALL PRIVILEGES ON  `d6x%` . * TO  'drupal'@'localhost';

Anytime you want to add a database for a new site simply run the following.

CREATE DATABASE  `DATABASE_NAME` ;

Enjoy your fancy development environment!

General quality assurance roadmap and request for help

As you may have seen in my previous post I have been getting back into the swing of things after some time away. I have a number of projects that I need to attend to, but as always not enough time to work on them all. Bellow you will find a list of my most important projects that need attention. If you have time available and an interest in helping out I would appreciate any extra hands. A number of the items do not have links since I have yet to spec them out completely, but I will update this post as I fill them in.

• Parse project .info files: present module list and dependency information - needed in order for contrib testing to work 100% of the time and to move it out of beta so anyone can enable testing of their project.
• PIFT 2.3 (drupal.org side of qa.drupal.org) - In order to make the most of the qa framework a number of improvements need to be made, mainly options and information display on drupal.org so everyone can take advantage of all the features the system already provides.
• PIFR 2.3 (qa.drupal.org and clients) - A fair amount of testing of the current development code base, a few more bug fixes, and features are needed to round out the next release.
• A few remaining cleanup items for SimpleTest in Drupal 7 core.
• SimpleTest 7.x-2.x - The branch will be used for continued development during Drupal 7 life cycle and already contains a number of improvements over the base Drupal 7 testing system. It was designed so that it can coincide with the Drupal cores framework and allow for tests to use either framework (one framework per module). The switch module, which no longer works (need to look into), allows you to switch between the contrib and core versions with a single click. The switch module also could be used by other projects attempting to replace core modules and can do so with a small amount of abstraction. With the next PIFT/PIFR release I plan to support projects that wish to use the 7.x-2.x API for testing.
• Code coverage - Reports for all tests (patches or at least core/contrib commits).
• Refactor/Rewrite SimpleTest for Drupal 8 - I have an extremely detailed roadmap in my head (and some paper notes) and a small code base that needs to be made public.
• PIFR 3.x - Same as above, lots of plans, but not very much publicly available yet.

Feel free to read through the issues, information, and code and let me know if you need any help getting started. Thanks!

Drupalcon SF - Quality assurance thoughts

Preface Before I get to the actual body of this post I would like to give an explanation for my somewhat distant behavior the last month or so since Druaplcon SF and the reason for this post being so long in the making. I have been going through some life changes and issues that have required most of my attention and left me with little to time for the Drupal community. I have resolved the issues that were consuming my time and I am looking forward to picking up where I left off. Hopefully, you will see a lot more activity from me in the near future.

Summary We had an educational discussion during the quality assurance break-out session at the Core Developer Summit. During the session we discussed the following topics.

• JavaScript testing for Drupal
• Site-builder testing tools
• Drupal core performance tests
• Ensuring its easy to start testing

I was charged with leading the discussion and taking notes. The following are my notes of the conversation that took place during the session.

• JavaScript testing for Drupal
  • Use testswarm to crowd source JavaScript testing.
  • Either, test HEAD/branches only and do so against tagged versions or wait for a single browser result to come back and do on patches.
  • Determine list of browsers/configurations we official support and that must pass JavaScript tests.
  • Look into themes breaking JavaScript, possibly run core JavaScript tests against contributed themes.
  • Selenium seems to have limited run-ability.
• Site-builder testing tools
  • Provide base set of tests to ensure a Drupal server is well.
  • Provides the ability to run tests against non-Drupal sites which can be useful when porting sites, working with sites that are not entirely written in Drupal, and for checking third-party integrations.
  • Maintain site-builder tools in 7.x-2.x branch of SimpleTest in contrib.
  • Possibly provide a slimmed down version of SimpleTest for use outside of Drupal.
• Drupal core performance tests
  • Does not have to be complicated, bug simply provide a consistent benchmark.
  • Something like loading several URLs a number of times on the same server and configuration.
  • Have a scripted setup containing lots of content on server.
  • Provides another use-case for extracting the SimpleTest browser for use in core and elsewhere.
  • Simple graph of performance over time.
  • Possible initial performance suite
    • View several anonymous pages
    • Login
    • Create a node
    • Make a comment
    • View several administration screens
    • Load modules page (historically one of the slowest)
    • Logout
• Ensuring its easy to start testing
  • Use Selenium IDE combined with simpletest_selenium to make it easy to create basic tests.
  • Submit native Selenium IDE output with bug reports to make it simple for developers to re-create bug and check if bug still exists.
  • Could also be used by experienced developers to create a good basis for a test.

Thoughts

After letting everything digest I have a number of thoughts regarding the discussion and ideas that were presented as well as a few additional pieces of information. First of all I want to share my thoughts on JavaScript testing, as I am not sure I was able to properly present this idea in person.

I look at JavaScript testing the same way I look at the current PHP based testing we do. We assume a number of things work and are tested by other organizations. As such we do not duplicate those testing efforts ourselves which is a wise decision. What I mean by that is we assume the PHP language to work as expected, the SQL language and database engine to work, and a number of other components to function. No where in our testing system do we attempt to ensure that the PHP language constructs behave as they should. We should treat jQuery as the language that it is and assume, just as with PHP, that the language functions properly in all supported environments.

The implications of the above may not be clear. What the above implies is that we do not spend time ensuring that our components and JavaScript interactions function in all browsers, environments, and operating systems. Instead we leave that job to the folks at jQuery whom already do extensive testing. Drupal should focus on ensuring that the widgets/components that core provides, such as the form API autocomplete and the ctools framework function properly. This means that we can use a tool like Crowbar or Webkit to interpret the JavaScript/jQuery and run our tests in that manor.

Attempting to test our JavaScript implementation on the infinite number of environments that exist adds a large amount of complexity to our work-flow and, as far as I can tell, very little gain. Unless someone can come up with a solid reason why we need to run our JavaScript tests on lots of environments I do not feel the idea is worth any more consideration. Oddly enough the proponents of it seemed willing to delude to waiting for one environment to return before reporting the results on drupal.org. It seems we have a lot of interest around the idea with little concern given to the implementation and cost vs benefit.

I propose we evaluate JavaScript testing frameworks with this in mind. We also need to be aware that we do not need to re-test the whole of Drupal using a JavaScript testing framework. On the contrary we need to ensure that our components and interactions work in a generic form and leave the actual testing of the final operations such as submitting a form to the already existing PHP tests. Maintaining two suites of tests that cover the same ground would be a big mistake that I hope we do not make.

To solidify this point further let’s compare the popular JavaScript testing framework Selenium to our current PHP testing framework. After you boil down the features and purpose of the two systems you discover that they both focus on the same key ability, that being to submit forms and perform actions as a user would. The area that Selenium allows us to test that our current system does not is in regards to JavaScript interpretation, while our current system allows us to test the PHP API directly, interact with the database, and even perform unit testing. So in order to give ourselves a fully rounded test framework we simply need to fill in the small bit that our current system does not give us.

More specifically, we need to be able to test our JavaScript behaviors and components built on top of jQuery. Something more along the lines of qUnit seems appropriate since it focuses on doing just that. We will most likely develop some wrapper code for Drupal specific things, but the library provides us with a much closer starting point. There is already a patch that takes us most of the way.

Plans

The site-builder tools discussed will be maintained in the SimpleTest 7.x-2.x branch and hopefully committed to Drupal 8 once development has begun. I will continue to work on improving the tools provided to site-builders in regards to testing in the 2.x branch and will also provide back-ports of these tools to the SimpleTest 6.x-2.x branch. Since these tools are relatively new I appreciate feedback.

• Remote server testing
• Database clone testing

Drupalcon SF - Core developer summit - Quality Assurance in Drupal 8