Aegir5 Roadmap

Aegir5 is pre-alpha softare

Under open development at https://gitlab.com/aegir/aegir.

This roadmap will be regularly updated as we iterate, to reflect current progress and status.

Upcoming milestones:

We’ve recently revised our aspirational development schedule for the pre-Alpha and Alpha release cycle, attempting to project out as far as an RC0 release by Drupal 7 (official) EOL, although that target seems to be shifting as well.

For now, only the immediate next steps to get to Beta-quality software are clear to us, and we hope to inspire broader community engagement by that point, to help further refine and implement Aegir5!

Milestones overview

Trajectory

We are currently in pre-alpha development stage. Having prototyped the basic architectural pieces to prove its utility, we are currently focused on solidifying an initial foundation, so that we can better enable the existing Aegir community to participate in the project.

Following a two-year hiatus of active development (and a working prototype), the Consensus team has been undertaking a re-invigoration of the project and working hard to solidify an architectural foundation based on our initial prototype.

In the last year or so, we have:

  • Roadmapped a few milestones ahead, and written myriad new tickets to describe our aspirations for how to proceed, at least as far as an RC release.
  • Re-architected the queue system and the Celery workers that underpin the core Aegir functionality of modeling and manipulating Platforms & Sites from the frontend, and translating those into Operations & Tasks, run on the backend. (see eg. #42, #64, #69, #70)
  • Written solid test coverage for these foundational pieces, and a simple “manual check” workflow to ensure the queues are up and running (see the HOWTO)
  • Begun fleshing out some Examples-style projects as a way to demonstrate how to approach building up Aegir5 functionality from first principles.
  • Revived this docs site, adopting a diataxis framework for structuring our documentation.
  • Overhauled our localdev workflows and tooling, leveraging DDEV to more accurately model the networked set of Aegir5 services. (eg. see #59 #79

In broad strokes, the goals for the next few milestones are roughly:

pre-Alpha

Generally our focus on the pre-Alpha releases is enabling other developers to meaningfully engage with Aegir development, and putting the pieces in place to enable us to build out Drupal functionality to get us back to feature parity with Aegir3 (core).

pre-Alpha0 (DONE!)

  • Nail down the queue system, finish re-architecture and 2 workers (relayd & dispatcherd basics)
  • Get DDEV and container images in place, and solidify related localdev tooling.

With the pre-Alpha0 release, we have completed the foundational implementation of relayd and dispatcherd, and are in the process of finalizing some tests and documentation around the queue system’s architecture.

Originally we had the Drupal front-end components requesting tasks be run on the back-end via the dispatcherd Celery/RabbitMQ message queue component, triggering Ansible tasks to be run on the server side.

With the move to DDEV for local development, we’ve more closely modeled the decoupled architecture of the front- and back-end components, where all communication should be managed through the message queue.

This allowed us to more easily build out a test-driven “front-end” daemon (dubbed relayd, and distinguished from the backend dispatcherd), and rip out the HTTP-based mechanism our prototype had used to get log data back to the frontend.

See Issue #71 and related issues for details.

While the focus here was on passing log information back to the front-end, this entailed completing the end-to-end communication over the message queue, which opens the door to building higher-level functionality.

pre-Alpha1 (target: end-Dec 2021)

  • Re-enable automated CI tests, including @javascript scenarios. (see #58).
  • Add “minion” container to localdev setup, and rough in SSH connection so dispatcherd can run tasks in minions. (see #60).
    • Deploy static site to a static minion with Nginx/Apache and a single HTML file.
    • (optional) Re-enable deploying D7 platform/site to the minion.

pre-Alpha2

  • Re-model variable structure for dispatching tasks (see #44).

  • Consolidate per-Task roles into Ansible task-files in Platform/Application-level roles. (see #91

  • Re-name Aegir entity classes, etc. (see #76).

  • One goal: define DX APIs to allow Aegir community devs to plug-in.

    • Need to figure out the ideal level of docs to allow other Aegir community folks interested in contributing to do so.
  • Another goal is to think through some outstanding architectural and workflow questions, considering our current understanding and goals for Aegir overall, in some ways broader than ever. For example: the data we pass between the front-end and back-end, and vice versa. where do we store its metadata? How do we maintain state of the system overall?

    • Sometimes we have transient data (e.g. state of a task), and the front-end polls for it, generally.
    • Sometimes we have a blob of data (e.g. the set of packages associated with a codebase), relatively stable, but periodically needing refresh.
    • Sometimes we have a list of data (e.g. a task log in progress), where we append to a list of field values regularly, and want to update the page in real-time, if somebody’s watching.
    • So the quetsion is: how do we handle these different kinds of data? In what ways are they different? What’s the conceptual model for front-end and back-end to use these well?
  • Also architecture-wise: codebases and sites are currently entity types, that are bundleable. Currently, to create one we inherit from a template/base class; there’s no other behaviour.

    • We might want to add more separate entity types for other kinds of things, such as different behaviours for servers and services than for codebases and sites. (Maybe?)

pre-Alpha3

  • Introduce “host” and “services” entities, to model Aegir3 behaviour for minions. (see #92).

pre-Alpha4

  • Implement alternative task types, so dispatcherd can run more than just Ansible playbooks. (see #52).

Here we will explore the idea of making the back-end tasks “pluggable” in the sense that Ansible playbooks is only one possibility. Currently the obvious second back-end type would be Terraform, but we’ll use this case as a basis to make Aegir flexible as to what kind of tasks to run in the backend (could be shell scripts, other config management or provisioning tools, etc.)

Alpha

The focus of the Alpha releases will be to get us back to feature parity with Aegir3 core as it stands today. This is specifically just the bare bones of CRUD operations on D8+ sites (probably not even D7 support initially), in order that we can begin to use it in real environments, and thus inform where to direct further development efforts.

Alpha0 (target: end-Feb 2022)

  • Basic Drupal 8+ support comparable to Aegir3.
  • Better security by default.
  • Feature-complete ready for regular use

In more detail:

  • Focus on D8+ as the primary application to deploy, and aim for feature parity with the Ops-focused workflows that Aegir3 does very well for Drupal 7, 8, and 9 today.
    • Create Codebases (traditionally called “platforms” in Aegir), provisioning them on one or more Servers (later containers) which have the Services to run them.
    • Create Site instances on top of those Codebases, which represent a particular install at a specific URL.
    • Automated and manual backups of Sites and Codebases, to designated Server(s).
    • Reliable restoration of backup data, to spin up a copy of a Site at a point in time.
    • Migrate Site instances between Servers and Codebases (even Services, e.g. MySQL to MariaDB), with blue-green deploy, automatic backups and rollback.
    • HTTPS, different Webservers, etc.
  • Currently, if we verify a codebase, we scan for profiles, pick up .info files, post those back to the front-end. This creates profile “entity” on the frontend as an optional available to site-installs, etc.
    • This is application specific: D7 is different than D8+
    • In D8+ we will probably use the composer.lock in a similar way
    • In general there should be an abstract form of this “set of packages” a Codebase has installed, that arbitrary applications (Wordpress, Grav) can define in their own terms.

Beta

  • Stabilizing, improving performance, etc.

  • For the adventurous :)

  • Figure out what packaging looks like. However it gets deployed should be automated as much as possible with any .deb, .rpm, Snap/Flatpak etc. for distribution. Must be automated on release. Eliminate the current day-long release cycle.

    • Currently this is painful: multiple projects, specific order, build scripts, and tests that can break things.
    • Our pipeline: Tag it, forget it, and then the release is out. Then we can do minor point releases regularly, fearlessly.
    • Ideally we’d be doing this by the time we tag RCs.