Architecture

N.B. This architecture reflect the original design proposal. While much of what has been built has stayed fairly close to this design, we will update it with actual implementations and examples regularly.

Summary

Fundamentally, Aegir5 maps a web GUI to CLI commands. It is made up of several components. First off, there is a front-end built on Drupal 8. This user interface passes configuration variables into a distributed task queue, built on Celery. Finally, queue workers receive these tasks, and run various operations, based on the variables passed into the task from the front-end. A command-line client, built on Drupal Console, can also post tasks to the queue, allowing for relatively simple scripting.

Architecture diagram

Front-end

The front-end is written in Drupal 8. It consists of some base entities and traits, along with basic admin interfaces for creating and managing fields and bundles.

Entity types

We should implement base entities that represent useful abstractions from an operational stand-point. These should come with default admin interfaces, but we can then separately build out a user-friendly UI that further abstracts low-level components.

Our current abstractions include:

  • Servers and services
  • Distributions and platforms
  • Sites

We may want to restructure our abstractions, such as:

  • Resources
  • Applications
  • Deployments (Application instances)

We may want to stick with our current abstractions for the first round of re-architecture, since we know that the model works. We could then focus on getting the queue/engine implementation sorted out.

That is, we can keep a known workflow and terminology, while still reaping the benefits of decoupling components, removing our dependence on Drush as a framework, etc. This latter one is our most pressing priority.

Queue system

The queue is implemented using Celery, a full-featured task queue written in Python. The queue client should probably, by default, run locally to the web UI, so that they can communicate using a port on localhost.

Back-end

The back-end will consume tasks from the queue, and run various commands. The initial engine is ansible-playbook, but Kubernetes is also likely to follow.

Queue worker

The back-end will be implemented as a Celery queue worker. This can then easily dispatch commands, as needed.

Ansible roles

We can write small Ansible roles to represent each task we want to implement, rather than the usual model of handling all tasks related to a given application. A small set of variables will need to be mapped to the equivalent front-end fields.

For example, a task on the front-end to deploy a Drupal codebase, could trigger a aegir.DeployDrupal8GitPlatform role, where we provide variables for a git repo from whence to clone, as well as a filesystem path where it should be deployed.

A separate aegir.VerifyDrupal8Platform task could then ensure that proper file ownership and permissions are maintained. This should only require a path variable to be passed to the task. (note: we may not need verify anymore; this is just an example)

Security model

Ansible provides a handy, secure mechanism to allow multiple servers, since it uses SSH to communicate between VMs. We don’t want to be able to run arbitrary commands on the back-end, since this could easily lead to compromised security. Rather, the Ansible roles will represent a whitelist of commands, and safe variables.

CLI provides a more or less universal API for applications. By supporting CLI tools, as our principal backend engines, we can standardize mechanisms for both calling and gathering feedback from them. SSH provides a secure, proven, widely supported communication mechanism between hosts.

In contrast, interacting with various backends via web-based APIs would generally involve multiple authentication mechanisms, differing serialization formats, etc.

Secrets

To the extent possible, secrets should neither be entered nor exposed via the UI. Ideally, these would be either generated on the backend or, where needed, deployed by an administrator via SSH.

These should likely mostly be situated on the queue worker, so as to be accessible when needed by ansible-playbook or other engines.

User experience

By default, entities should provide only admin UI components. But, being built on fieldable entities, these should then all be accessible to Views, Panels, etc. to allow for better end-user experience. We should provide a “default” UI, but ensure that this can be easily customized, or replaced entirely.

Front-end class hierarchies (examples)

Scenario 1

N.B. Scenario 2 (below) is probably better.

  • Distribution (Application?)
  • Platform (Codebase?)
  • Site (ApplicationInstance?)
    • CmsSite
      • DrupalSite
        • Drupal7Site
          • CiviCrmDrupal7Site-
        • Drupal8Site
      • WorpPressSite
        • WordPress4Site
          • CiviCrmWordPress4Site
  • Task
    • AnsibleTask
      • AnsiblePlatformTask
      • AnsibleSiteTask
        • AnsibleDrupalSiteTask
          • AnsibleInstallDrupalSiteTask
          • AnsibleDisableDrupalSiteTask
          • AnsibleBackupDrupalSiteTask
          • …

Scenario 2

Compose Sites, Platforms, (etc.) from Tasks (e.g., WriteNginxVhostTask, ProvisionMySqlDatabaseTask,…). Operations combine Tasks into user-facing actions (e.g., InstallCmsSiteOperation). Tasks are re-usable within other Operations.

  • Distribution (entity type?)
  • Platform (Distribution bundle?)
  • Site
    • CmsSite (fieldable entity, that embedding of task entities)
      • Drupal7Site
      • CiviCrmDrupal7Site
      • Drupal8Site
      • WorpPress4Site
      • CiviCrmWordPress4Site
  • Task (defines log and result fields, provides a URL to which to post log output)
    • AnsibleTask (parses log output for return code, to update result field).
      • AnsiblePlatformTask
        • AnsibleDeployDrupalPlatformFromGitTask
        • AnsibleDeployDrupalPlatformFromDrushMakefileTask
      • AnsibleSiteTask
        • AnsibleWriteNginxVhostTask
        • AnsibleGenerateDrupalSiteBackupTask (implements AcceptsFeedbackTrait, contains a backup_path field)
        • AnsibleRestoreDrupalSiteBackupTask (provides select widget populated by a View of this site’s backup tasks' backup_path fields. This, in turn, is passed to the task queue, and onto the Ansible role to execute the restore).
        • AnsibleDisableDrupalSiteTask
        • AnsibleInstallDrupalSiteTask
        • …

Traits

  • AcceptsFeedbackTrait
    • hook_menu(): (or equivalent listener) to register the feedback URL.
    • handleFeedback(): write feedback to appropriate field.

Backend implementation (examples)

Ansible-playbook queue worker

  1. Celery queue worker receives task.
  2. Writes temporary playbook (see example below) populated with vars and roles from the originating task.
  3. Runs playbook.
  4. Posts log output to front-end URL, where it populates the task log.
  5. Roles may receive a “feedback_url” variable, whence it can post data required by the front-end (e.g., backup path, git commit hash, etc.)
    1. Role is responsible for a cURL (or whatever) call to post data to the provided URL.
  6. Cleans up playbook.

Temporary playbook example

  - hosts: web0
    vars:
       ( populated from task )
    roles:
       ( populated from task )

Ansible role examples

“Task” roles are small re-usable pieces of config.

aegir.GenerateDrupalSiteBackup/
├── meta/main.yml (equiv. to .info)
├── defaults/main.yml (variable defaults)
└── tasks/main.yml (steps to execute)
    ├── drush archive-dump
    ├── move to storage location (path)
    └── post path back to front-end

aegir.WriteNginxVhost/
├── meta/main.yml
├── files/default-nginx-vhost.conf.j2
├── defaults/main.yml
└── tasks/main.yml
    ├── write vhost from template (possibly looking for overrides in various places)
    └── restart nginx (but only if the vhost changed; i.e. idempotence)

Operations vs. Tasks

Operations expose “runnable” functionality to end-users. Operations group tasks into coherent sets (e.g., install CiviCRM site). Task, on the other hand, are atomic (e.g., write a vhost).

  • Operation
    • SiteOperation
      • CmsSiteOperation

        • InstallCmsSiteOperation

          ```
::task_list = [
    AnsibleWriteVhostTask,   <-- plug-in managers
    AnsibleProvisionDatabaseTask,
    AnsibleSiteInstallTask
]
```

        • BackupCmsSiteOperation

        • UpgradeCmsSiteOperation

      • CmsPlatformOperation
        • DeployCmsPlatformOperation
        • UpgradeCmsPlatformOperation
      • CmsDistributionOperation
        • UpdateCmsDistributionOperation
          • UpdateGitCmsDistributionOperation
          • UpdateComposerCmsDistributionOperation

Plug-in Types:

  • Database:
    • MySql
    • Postgres
  • Web:
    • Nginx
    • Apache
  • Installer:
    • Drush
    • wp-cli
    • Drupal Console
    • cv