Drupal quality assurance requires a decision

Many of you may be familiar with my previous post Diaries of a Core Developer and that sadly nothing has changed. We had a lot of good discussion in the comments, but as far as I understand nothing has been done to improve the development workflow.

Drupal QA has and is currently having issues due to a complex problem with the core design of SimpleTest. I documented this problem some time ago, and made several attempts to refactor SimpleTest (before and after). The patches were quite large since they did a proper overhaul of the system. I was told to break them into smaller pieces so they would be easier to review, but after discovering how intertwined the code was and not receiving any other real feedback I was burned out after a week of working on the patch (and the lack of time others were willing to spend).

Just as the database layer and other large changes had large patches and lots of follow-up patches, even SimpleTest itself was introduced in a large patch, so does this refactoring. I briefly described what needs to be done and some of the benefits in the original issue. You can find more details on the refactoring in one of my later consolidation issues in which I attempted to explain the design and work through the issue.

As I explained in the issue there are some hackish ways we can work around the problem for the time being. I am happy to implement those fixes, given that I am not wasting my time. I still believe an overall refactoring is in order to fix a number of problems including proper isolation of test processes and a clean way of gather errors.

Before I commit a large amount of time, again, to refactoring the system I would like confirmation that someone will actually review the patch, those involved are fine with the changes, and that we will see this through to be committed (preferably some assurance from Dries of webchick). Of course this leads back the fact that I have no authority as a maintainer to make this crucial decision and that the core maintainers are bogged down reviewing ALL patches, instead of distributing the load to the sub-maintainers and core maintainers reviewing sub-maintainers patches and overall changes.

Comments

Please stop complaining about core maintainers. Can yow show a single issue, where a good patch was RTBC but rejected solely by the core maintainer?
Drupal needs to be developed and reviewed by the community, not core maintainers.. Look at core maintainers as just members of the community, they do not have much to decide about your issue.

steps to proceed:
1. be sure what you are proposing is a good thing and belongs to core. if you are unsure, ask others from the community
2. make sure that someone from the community who understands your patch will make a full review
3. be able to defend your changes if someone from the community challenge them
4. if you have done 1+2+3 and your patch is not committed, you can complain..

By no means am I complaining about core maintainers. Read the text, I simply state that the job is bigger then they are. The maintainers do an excellent job, but due to recurring issues like this and a growing developer community we cannot maintain such an unbalanced ratio.

Jimmy,

You haven't heard the gossips then that if any contribution does not fall in Acquia Roadmap, it will not get attention. Talk to some large development shops both on east coast and west coast and you'll know why they do not contribute anymore.

All the best to you and hope you will find a way where you are not frustrated.

Tapei

I am not sure where you are coming from with that comment. Acquia has sponsored a number of community efforts and even sponsored me to work on improving the testing and quality assurance system which benefits the entire community.

The design decisions are still made through community discussions. I have been a part of issues where Dries has gone with community will over his own reasons. In my opinion that dispels any thoughts of an Acuqia roadmap controlling what gets in, not to mention the lack of evidence to even make that claim.

My issue is more over delegation of responsibilities and the timeline over which things occur. I am not having an issue with my ideas being denied or anything of the sort.

I've been working on testing and automated testing for Drupal since 2007. I believe it to be one of the core pillars that's going to allow Drupal to continue to accelerate development and keep up with the need for stability as Drupal grows to be used on bigger and bigger sites.

During this time we've really struggled to build a strong core testing framework team. Moshe started with Simpletest. We had some good mentors, like Thomas Ilsche. Rok Zlender who did a great job getting the first generation of the automated core testing platform started. When Dries layed out the need for testing in core http://buytaert.net/embracing-test-driven-development there was a surge of excitement and many people rallied to help. Karoly, Angie, and Jimmy worked closely together and recruited and trained dozens of contributors while writing a Simple test based framework to meet Drupal's needs.

We've done a good job as community writing tests. Today there are almost 15,000 tests for Drupal 7. Over the last 6 months we struggled to get enough system administrators to maintain and debug the testing infrastructure. We recruited a lot of system administrators and many of them helped us find bugs, write documentation, and provide UX feedback. Small iterative improvements helped. But one of the biggest changes of this effort is that we recruited an testing infrastructure co-maintainer to help Damien keep the testing infrastructure stable. David Norman, with the support of Classic Graphics has helped remove that frustrating single point of failure.

At Acquia we hired Jimmy to be an intern over the summer, to help provide the support and resources to improve the testing framework. This included time for for getting the third generation of the framework released as well as refactoring long standing issues. NowPublic/Examiner has stepped in to support Jimmy's work and that's really great too. But Jimmy, as talented as he is, needs a co-maintainer. Someone who can help take the burden off his shoulders and bring a different perspective and patch reviews that the culture of core development requires.

There are three issues here:
1) Get PIFR 2.0 launched and testing Drupal 6 and Drupal 7 contributed modules not just Drupal 7 core.
2) Refactor significant problems in the testing framework.
3) Address workflow issues in Drupal core development.

This reminds me of a quote from Hillary Clinton when asked why Health care reform failed in the first Clinton administration. She said that they "Failed to have a strategy".

It is my hope, that when more developers running production Drupal 6 sites start using the simple test framework for their contributed modules, it will bring a larger audience of developers and they will quickly discover the flaws and need for refactoring that Jimmy has identified. With that wider awareness I hope a co-maintainer emerges. Right now, many developers are beginning to port their contributed module to Drupal 7. If they are doing so without tests, then it's going to be much harder to capture their mindshare and momentum later.

In my opinion, the strategy right now should be to get D6 and D7 contributed modules using the simpletest framework in core and the automated patch testing. That will recruit more contributors, and give the base audience simpletest framework reviewers needed. If the strategy continues to take on all three issues simultaneously, people will get confused, even if each issue is important in it's own right.

Kieran

I'm happy to review any patch that fixes the design and implementation of our test framework, and I'm sure other people in the community will be happy to as well.

What Pasqualle said.

You are passing the blame to the core maintainers, the core development/commit process, and a number of other places. But the reality is, the reason you're frustrated is because there is a lack of a cohesive "team" of developers/reviewers around the testing frramework: it's basically just you. And passing the responsibility onto to core maintainers to know and understand the testing system inside and out enough to review all of your changes is indeed a totally losing strategy, since we have purview over the entirety of core and need to remain generalists (not specialists) in order to do our jobs of giving objective, final QA on any patch that makes it to RTBC.

Other subsystems that have built up such a group (accessibility, ux, database system, render system, javascript, etc.) tend to get patches in relatively easily, since core maintainers can read the reviews, tell that thorough thought has been put into things, do our final QA check thing, and then we're good to go. When we need to transition from that role to "let me understand the testing system well enough to evaluate this change on all of its merits" it completely obliterates our ability to remain objective evaluators, and also takes forever. If you are trying to utilize core maintainers this way, You Are Doing It Wrong.

Focus your time/energy less on writing frustrated blog posts and more on documenting the testing system's innards, mentoring new contributors on how it works, and rallying reviewers for your patches, and you will have a much easier time of things.

That said, of course, I will happily review any patches that make it across my desk, and am having trouble understanding why the tone of this post implies Dries or I ever haven't.

1) Pasqualle was responding to "re gossip".
2) I am really am unsure what to think at this point. You say the blame needs to be placed on the lack of a development team around SimpleTest. I have two issues with that thought: 1) the database layer has a team, yet as referenced in my previous post Larry had all kinds of issues getting a patch in due to his lack of authority (as one of my primary concerns), 2) if there is no team then we need an altered workflow for SimpleTest for the time being. Do we really want to completely hinder SimpleTest development because we do not have enough people interested and knowledgeable in its development? Why bother maintaining it at all? It just completely burns out anyone trying to get a patch through, as I can completely attest to. Mentally and physically burned out.

1) No, I was referring to Pasqualle's initial post at http://blog.boombatower.com/drupal-quality-assurance-requires-a-decision..., which is absolutely bang-on. I would absolutely never call "bullshit" on anything you write. Give me a little more credit than that, please. :)

2) I am genuinely sorry that you are feeling burnt out, and for any part that I may have played in that feeling. But code that goes into Drupal core absolutely has to be peer reviewed. And code that is itself is intended to uphold the quality of the rest of the code in core needs particularly careful review. So, no... I definitely do not support an alternate workflow for testing framework-related code, and in fact I think that adopting a wild west, contrib-style workflow around the testing framework would actually be the worst thing that we could possibly do.

Why? Because it would mean that you continue to be the only person who understands what's going on in there, and that in turn means more pressure on you personally to find and fix bugs, improve the UI, add additional functionality, etc. which is only going to make matters worse. What we need to get out of the current burnout-inducing situation are more people who understand this section of code, who can team up together to provide patches, and solid reviews to each others' patches. Then, you can start acting in an actual maintainer capacity: diving in on particularly tweaky issues, eyeballing others' changes and and providing guidance, and giving the final thumbs-up when things are good to go.

So instead of focusing on these other tangential issues, focus on building your team. Fixes will go in faster, you'll have less work to do, and the work you do have will be a lot more fun when you're not feeling so alone.