Drupal 8 thoughts: configuration management and improved installation

I have been doing a lot of work related to easing the process of building a site from scratch on an individual machine and thus dabbling with configuration management and related topics. Since configuration management is one of the Drupal 8 key initiatives I figured I would share some thoughts I had on the outer fringes of the topic with more to come in the future.

Installation

One area of Drupal that has always seemed a bit odd to me has been the installation process/system. The process can play a key part in configuration management across machines and made it impossibly to implement a basic environment system during installation without hacking core. To remedy this issue I believe the installation system can be much improved, simplified, and made much more consistent with the rest of Drupal which will in tern make it easy to implement my environment system in contrib or core.

The installation system in Drupal 7 was re-factored/rewritten quite a bit and has thus been quite improved, but I think the direction of the installation system could be changed to make it much better. Currently the installer attempts to fake systems in core, like the cache, and actually duplicates a lot of code found elsewhere for module management and what not. Why not simply package a minimal database dump, similar to how the update tests used a Drupal 6 dump written in DBTNG, that can be installed to create an extremely minimal Drupal installation. At that point the installer can act like any other module and provide forms to complete the process. All modules in addition to the required modules can be installed through the standard process invoked from the modules page, but done in an automated fashion through the installer.

The reasons why this approach is beneficial are: 1) install.inc and install.core.inc could be virtually removed, 2) profiles would no longer be "hackish" during the install phase (solve the current issues related to dependency resolution and what not being the same for modules and profiles), 3) hooks like hook_system_info_alter() would work properly for profiles and modules during early installation phases, and 4) remove race conditions in general caused by maintain two sets of the same code.

Environments

In addition, my environment.module would work without hacking core. This concept is nothing new in the development world, but is something I feel would be a great candidate for Drupal 8.

<?php
/**
 * Implements hook_system_info_alter().
 */
function environment_system_info_alter(&$info$file$type) {
  if (!empty(
$info['dependencies'])) {
    
$environment environment_get();
    if (!empty(
$info['dependencies'][$environment])) {
      
$info['dependencies'] = array_merge($info['dependencies'], $info['dependencies'][$environment]);
    }
    foreach (
$info['dependencies'] as $key => $dependency) {
      if (!
is_numeric($key)) {
        unset(
$info['dependencies'][$key]);
      }
    }
  }
}

/**
 * Get the current environment.
 *
 * @return
 *   The current environment: production, staging, or development.
 */
function environment_get() {
  return 
variable_get('environment''production');
}
?>

The above code allows for the environment to be configured in the settings.php file for a site.

<?php
$conf
['environment'] = 'development';
?>

During installation a profile (or module) can perform different tasks or conditional code based on the environment. For example, generated users can have a simple password on a development machine and complex ones on production or staging machines.

<?php
function my_profile_install() {
  if (
environment_get() == 'development') {
    
// Do cool stuff that only devs get to see.
  
}
}
?>

Another cool feature that I have a use-case for on a site I am working on and seems to generally be useful is to enable modules based on the environment. Instead of having to do that in hook_install() or related it makes sense to have a way to specify that in a .info file. The above code allows for the following.

name = My profile
description = ....
version = 0.1
core = 7.x
 
dependencies[] = block
dependencies[] = dblog
 
dependencies[development][] = views_ui
dependencies[development][] = fields_ui
dependencies[development][] = devel
 
dependencies[production][] = integration_with_third_party
dependencies[staging][] = integration_with_third_party

Having this type of functionality in core would hopefully encourage better development practices and seems like a great feature to have. I have a number of scripts in combination with drush make, and the above environment utility that allow me to build out a fully functional site on a new box with a single command. I plan to cleanup the scripts, document them, and provide them in a followup post. As always I would love to hear your thoughts on this subject.

Comments

Drupal and its modules used to be installed with database dumps (imported manually). Now its an interesting idea to get that back but now with automated imports. Many things can maybe dealt that way, but I'm afraid you'll not be able to get away with just dumps for referential data, such as which user belongs under which roles. At least in Drupal 7 these are still numbers (in Drupal 8 they might not be anymore), and they might not get the same number on all installations depending on the database setup. So having logic to some degree looks inevitable.

You'll still need some of the theming, forms, localizations, etc. systems work in some kind of special way before the database is set up when the requirements checking is done for example. So not sure you'll be able to get rid of a lot of stuff.

The database dump or sorts would only be used for the required Drupal core modules (like system), all other modules would be installed like normal. Obviously another approach (probably further down the road) is to allow Drupal to run without a database in some fashion.

Well, user module is currently required, and has the problem as I've explained. You need to add data to multiple tables, one referring to both user IDs and role IDs. Now we might as well make role IDs names instead of numbers, but user IDs I doubt. There is no guarantee you have uid 0 and uid 1 for the initial users, the only reason it ends up like that is because Drupal forces that after INSERT. I'm not saying these make using dumps impossible at all, just pointing out you cannot escape logic for creating the data.

I am aware of that and in Drupal 7: modules/simpletest/tests/upgrade/drupal-6.bare.database.php you will find the anonymous user and user/1 imported as 1 and 2 and then runs:

db_query('UPDATE {users} SET uid = uid - 1');

We can do the same in the rough database dump. I'm not saying it will be perfect and it may involve some tweaking of other systems, but most of the complicated stuff should be able to be done in standard hook_installs().

I think this idea is worth exploring. chx proposed us to ship drupal with a small sqlite database for install but i think your idea is slightly different and also interesting. I also think it would speed up the installer, and reduce memory consumption. Both good things.

FYI, I also want different modules per environment and opened up a ticket for one approach - http://drupal.org/node/625444

At a core conversation in Chicago, David Strauss proposed moving all configuration to disk-first storage, with the UI writing directly to JSON on disck and using the DB only for caching. This was picked up by Larry, who is excited about the potential to be able to have configuration available very early in bootstrap without having to get the database running. We could also use prepackaged sets of these settings in the same way you propose using a database dump, and the installer could just write them back out. Ideally we will be getting away from serialized identifiers so they would be portable too, which will help a lot for many issues such as you describe.

Just wanted to chime in to say that I think having a minimum drupal setup for the installer is an interesting idea. Might it open up the floor for developers to more easily create custom installers? The current system is a bit "out there" (compared to traditional module and install profile development), and since not everyone understands how it works, there aren't lots of people poking through the code and getting creative with it. Having it work as a minimum profile might increase the amount of interest in further improving and adapting it (if I'm understanding correctly). Someone could create a new minimum install profile and contribute a new installer for others to put through its paces.

Oh hey, and I'd come across your Environment module awhile ago, and am excited by what it might lend itself to. Have you put any thought into how it might work with drush provision hooks? Might it be possible to easily change settings when provision migrates a site from staging to production servers? I'm very new to all aspects involved (provision hooks, drush plugins, environment module, etc), so apologies if I'm way off-base here!

For the minimal database dump, we'd probably only need the system and sessions tables - these are the kind of things that system module owns in lieu of the base system, so for me it'd be fine if we special cased these - we already special case system module in the installer anyway, which is far worse.