Historical Milestones

2023

Alpha MVP

As of May 2023, we had re-imagined our Milestones in terms of an MVP we can launch back into the Aegir dev community and beyond. We’ve broken MVP up into an Alpha and Beta stage.

In Alpha, the focus is on anything that’s useful to external developers who want to help with the project. We want a minimal prototype that integrates our Kubernetes backend with the familiar Drupal Frontend administration UI. We also need to document and clean things up enough that the project is accessible to interested participants.

Snapshot of Aegir 5 MVP Alpha sprint boards

Beta MVP

In Beta, we hope to have more feedback on our Aegir5 Roadmapping blog series, which lays out the trajectory we foresee from here.

post-MVP

The focus after the MVP launch will be to get us back to feature parity with Aegir3 core as it stands today. This is specifically just the bare bones of CRUD operations on D8+ sites (probably not even D7 support initially), in order that we can begin to use it in real environments, and thus inform where to direct further development efforts.

  • Basic Drupal 8+ support comparable to Aegir3.
  • Better security by default.
  • Feature-complete ready for regular use

In more detail:

  • Focus on D8+ as the primary application to deploy, and aim for feature parity with the Ops-focused workflows that Aegir3 does very well for Drupal 7, 8, and 9 today.
    • Create Codebases (traditionally called “platforms” in Aegir), provisioning them on one or more Servers (later containers) which have the Services to run them.
    • Create Site instances on top of those Codebases, which represent a particular install at a specific URL.
    • Automated and manual backups of Sites and Codebases, to designated Server(s).
    • Reliable restoration of backup data, to spin up a copy of a Site at a point in time.
    • Migrate Site instances between Servers and Codebases (even Services, e.g. MySQL to MariaDB), with blue-green deploy, automatic backups and rollback.
    • HTTPS, different Webservers, etc.
  • Currently, if we verify a codebase, we scan for profiles, pick up .info files, post those back to the front-end. This creates profile “entity” on the frontend as an optional available to site-installs, etc.
    • This is application specific: D7 is different than D8+
    • In D8+ we will probably use the composer.lock in a similar way
    • In general there should be an abstract form of this “set of packages” a Codebase has installed, that arbitrary applications (Wordpress, Grav) can define in their own terms.
  • Stabilizing, improving performance, etc.
  • Figure out what packaging looks like. However it gets deployed should be automated as much as possible with any .deb, .rpm, Snap/Flatpak etc. for distribution. Must be automated on release. Eliminate the current day-long release cycle.
    • Currently this is painful: multiple projects, specific order, build scripts, and tests that can break things.
    • Our pipeline: Tag it, forget it, and then the release is out. Then we can do minor point releases regularly, fearlessly.
    • Ideally we’d be doing this by the time we tag RCs.

2021

Following a two-year hiatus of active development (and a working prototype), the Consensus team undertook a re-invigoration of the project and working hard to solidify an architectural foundation based on our initial prototype.

Over the course of 2021 onward, we have:

  • Roadmapped a few milestones ahead, and written myriad new tickets to describe our aspirations for how to proceed, at least as far as an RC release.
  • Re-architected the queue system and the Celery workers that underpin the core Aegir functionality of modeling and manipulating Platforms & Sites from the frontend, and translating those into Operations & Tasks, run on the backend. (see eg. #42, #64, #69, #70)
  • Written solid test coverage for these foundational pieces, and a simple “manual check” workflow to ensure the queues are up and running (see the HOWTO)
  • Begun fleshing out some Examples-style projects as a way to demonstrate how to approach building up Aegir5 functionality from first principles.
  • Revived this docs site, adopting a diataxis framework for structuring our documentation.
  • Overhauled our localdev workflows and tooling, leveraging DDEV to more accurately model the networked set of Aegir5 services. (eg. see #59 #79

As noted on the main Roadmap page, we’ve revised the structure and focus of our upcoming milestones based on an MVP launch. Previously, the goals for the next few milestones were roughly:

pre-Alpha

Generally our focus on the pre-Alpha releases was enabling other developers to meaningfully engage with Aegir development, and putting the pieces in place to enable us to build out Drupal functionality to get us back to feature parity with Aegir3 (core).

pre-Alpha0 (DONE!)

  • Nail down the queue system, finish re-architecture and 2 workers (relayd & dispatcherd basics)
  • Get DDEV and container images in place, and solidify related localdev tooling.

With the pre-Alpha0 release, we completed the foundational implementation of relayd and dispatcherd, and began the process of finalizing some tests and documentation around the queue system’s architecture.

Originally we had the Drupal front-end components requesting tasks be run on the back-end via the dispatcherd Celery/RabbitMQ message queue component, triggering Ansible tasks to be run on the server side.

With the move to DDEV for local development, we more closely modeled the decoupled architecture of the front- and back-end components, where all communication should be managed through the message queue.

This allowed us to more easily build out a test-driven “front-end” daemon (dubbed relayd, and distinguished from the backend dispatcherd), and rip out the HTTP-based mechanism our prototype had used to get log data back to the frontend.

See Issue #71 and related issues for details.

While the focus here was on passing log information back to the front-end, this entailed completing the end-to-end communication over the message queue, which opens the door to building higher-level functionality.

pre-Alpha1 (target: end-Dec 2021)

  • Re-enable automated CI tests, including @javascript scenarios. (see #58).
  • Add “minion” container to localdev setup, and rough in SSH connection so dispatcherd can run tasks in minions. (see #60).
    • Deploy static site to a static minion with Nginx/Apache and a single HTML file.
    • (optional) Re-enable deploying D7 platform/site to the minion.

pre-Alpha2

  • Re-model variable structure for dispatching tasks (see #44).

  • Consolidate per-Task roles into Ansible task-files in Platform/Application-level roles. (see #91

  • Re-name Aegir entity classes, etc. (see #76).

  • One goal: define DX APIs to allow Aegir community devs to plug-in.

    • Need to figure out the ideal level of docs to allow other Aegir community folks interested in contributing to do so.

  • Another goal is to think through some outstanding architectural and workflow questions, considering our current understanding and goals for Aegir overall, in some ways broader than ever. For example: the data we pass between the front-end and back-end, and vice versa. where do we store its metadata? How do we maintain state of the system overall?

    • Sometimes we have transient data (e.g. state of a task), and the front-end polls for it, generally.
    • Sometimes we have a blob of data (e.g. the set of packages associated with a codebase), relatively stable, but periodically needing refresh.
    • Sometimes we have a list of data (e.g. a task log in progress), where we append to a list of field values regularly, and want to update the page in real-time, if somebody’s watching.
    • So the quetsion is: how do we handle these different kinds of data? In what ways are they different? What’s the conceptual model for front-end and back-end to use these well?

  • Also architecture-wise: codebases and sites are currently entity types, that are bundleable. Currently, to create one we inherit from a template/base class; there’s no other behaviour.

    • We might want to add more separate entity types for other kinds of things, such as different behaviours for servers and services than for codebases and sites. (Maybe?)

pre-Alpha3

  • Introduce “host” and “services” entities, to model Aegir3 behaviour for minions. (see #92).

pre-Alpha4

  • Implement alternative task types, so dispatcherd can run more than just Ansible playbooks. (see #52).

Here we intended to explore the idea of making the back-end tasks “pluggable” in the sense that Ansible playbooks is only one possibility. Currently the obvious second back-end type would be Terraform, but we’ll use this case as a basis to make Aegir flexible as to what kind of tasks to run in the backend (could be shell scripts, other config management or provisioning tools, etc.)

While our direct work was on the backburner later in 2022 and 2023, a new pluggable Backend presented itself, in the form of Kubernetes. We’ve therefore taken up this aspect more directly.