N.B. This architecture reflect the original design proposal. While much of what has been built has stayed fairly close to this design, we will update it with actual implementations and examples regularly.
Fundamentally, Aegir5 maps a web GUI to CLI commands. It is made up of several components. First off, there is a front-end built on Drupal 8. This user interface passes configuration variables into a distributed task queue, built on Celery. Finally, queue workers receive these tasks, and run various operations, based on the variables passed into the task from the front-end. A command-line client, built on Drupal Console, can also post tasks to the queue, allowing for relatively simple scripting.
The front-end is written in Drupal 8. It consists of some base entities and traits, along with basic admin interfaces for creating and managing fields and bundles.
We should implement base entities that represent useful abstractions from an operational stand-point. These should come with default admin interfaces, but we can then separately build out a user-friendly UI that further abstracts low-level components.
Our current abstractions include:
We may want to restructure our abstractions, such as:
We may want to stick with our current abstractions for the first round of re-architecture, since we know that the model works. We could then focus on getting the queue/engine implementation sorted out.
That is, we can keep a known workflow and terminology, while still reaping the benefits of decoupling components, removing our dependence on Drush as a framework, etc. This latter one is our most pressing priority.
The queue is implemented using Celery, a full-featured task queue written in Python. The queue client should probably, by default, run locally to the web UI, so that they can communicate using a port on localhost.
The back-end will consume tasks from the queue, and run various commands. The initial engine is
ansible-playbook, but Kubernetes is also likely to follow.
The back-end will be implemented as a Celery queue worker. This can then easily dispatch commands, as needed.
We can write small Ansible roles to represent each task we want to implement, rather than the usual model of handling all tasks related to a given application. A small set of variables will need to be mapped to the equivalent front-end fields.
For example, a task on the front-end to deploy a Drupal codebase, could trigger a
aegir.DeployDrupal8GitPlatform role, where we provide variables for a git repo from whence to clone, as well as a filesystem path where it should be deployed.
aegir.VerifyDrupal8Platform task could then ensure that proper file ownership and permissions are maintained. This should only require a path variable to be passed to the task. (note: we may not need verify anymore; this is just an example)
Ansible provides a handy, secure mechanism to allow multiple servers, since it uses SSH to communicate between VMs. We don’t want to be able to run arbitrary commands on the back-end, since this could easily lead to compromised security. Rather, the Ansible roles will represent a whitelist of commands, and safe variables.
CLI provides a more or less universal API for applications. By supporting CLI tools, as our principal backend engines, we can standardize mechanisms for both calling and gathering feedback from them. SSH provides a secure, proven, widely supported communication mechanism between hosts.
In contrast, interacting with various backends via web-based APIs would generally involve multiple authentication mechanisms, differing serialization formats, etc.
To the extent possible, secrets should neither be entered nor exposed via the UI. Ideally, these would be either generated on the backend or, where needed, deployed by an administrator via SSH.
These should likely mostly be situated on the queue worker, so as to be accessible when needed by
ansible-playbook or other engines.
By default, entities should provide only admin UI components. But, being built on fieldable entities, these should then all be accessible to Views, Panels, etc. to allow for better end-user experience. We should provide a “default” UI, but ensure that this can be easily customized, or replaced entirely.
N.B. Scenario 2 (below) is probably better.
Compose Sites, Platforms, (etc.) from Tasks (e.g., WriteNginxVhostTask, ProvisionMySqlDatabaseTask,…). Operations combine Tasks into user-facing actions (e.g., InstallCmsSiteOperation). Tasks are re-usable within other Operations.
resultfields, provides a URL to which to post log output)
backup_pathfields. This, in turn, is passed to the task queue, and onto the Ansible role to execute the restore).
- hosts: web0 vars: ( populated from task ) roles: ( populated from task )
“Task” roles are small re-usable pieces of config.
aegir.GenerateDrupalSiteBackup/ ├── meta/main.yml (equiv. to .info) ├── defaults/main.yml (variable defaults) └── tasks/main.yml (steps to execute) ├── drush archive-dump ├── move to storage location (path) └── post path back to front-end
aegir.WriteNginxVhost/ ├── meta/main.yml ├── files/default-nginx-vhost.conf.j2 ├── defaults/main.yml └── tasks/main.yml ├── write vhost from template (possibly looking for overrides in various places) └── restart nginx (but only if the vhost changed; i.e. idempotence)
Operations expose “runnable” functionality to end-users. Operations group tasks into coherent sets (e.g., install CiviCRM site). Task, on the other hand, are atomic (e.g., write a vhost).
::task_list = [ AnsibleWriteVhostTask, <-- plug-in managers AnsibleProvisionDatabaseTask, AnsibleSiteInstallTask ]