Historical Milestones

2021

Following a two-year hiatus of active development (and a working prototype), the Consensus team undertook a re-invigoration of the project and working hard to solidify an architectural foundation based on our initial prototype.

Over the course of 2021 onward, we have:

  • Roadmapped a few milestones ahead, and written myriad new tickets to describe our aspirations for how to proceed, at least as far as an RC release.
  • Re-architected the queue system and the Celery workers that underpin the core Aegir functionality of modeling and manipulating Platforms & Sites from the frontend, and translating those into Operations & Tasks, run on the backend. (see eg. #42, #64, #69, #70)
  • Written solid test coverage for these foundational pieces, and a simple “manual check” workflow to ensure the queues are up and running (see the HOWTO)
  • Begun fleshing out some Examples-style projects as a way to demonstrate how to approach building up Aegir5 functionality from first principles.
  • Revived this docs site, adopting a diataxis framework for structuring our documentation.
  • Overhauled our localdev workflows and tooling, leveraging DDEV to more accurately model the networked set of Aegir5 services. (eg. see #59 #79

As noted on the main Roadmap page, we’ve revised the structure and focus of our upcoming milestones based on an MVP launch. Previously, the goals for the next few milestones were roughly:

pre-Alpha

Generally our focus on the pre-Alpha releases was enabling other developers to meaningfully engage with Aegir development, and putting the pieces in place to enable us to build out Drupal functionality to get us back to feature parity with Aegir3 (core).

pre-Alpha0 (DONE!)

  • Nail down the queue system, finish re-architecture and 2 workers (relayd & dispatcherd basics)
  • Get DDEV and container images in place, and solidify related localdev tooling.

With the pre-Alpha0 release, we completed the foundational implementation of relayd and dispatcherd, and began the process of finalizing some tests and documentation around the queue system’s architecture.

Originally we had the Drupal front-end components requesting tasks be run on the back-end via the dispatcherd Celery/RabbitMQ message queue component, triggering Ansible tasks to be run on the server side.

With the move to DDEV for local development, we more closely modeled the decoupled architecture of the front- and back-end components, where all communication should be managed through the message queue.

This allowed us to more easily build out a test-driven “front-end” daemon (dubbed relayd, and distinguished from the backend dispatcherd), and rip out the HTTP-based mechanism our prototype had used to get log data back to the frontend.

See Issue #71 and related issues for details.

While the focus here was on passing log information back to the front-end, this entailed completing the end-to-end communication over the message queue, which opens the door to building higher-level functionality.

pre-Alpha1 (target: end-Dec 2021)

  • Re-enable automated CI tests, including @javascript scenarios. (see #58).
  • Add “minion” container to localdev setup, and rough in SSH connection so dispatcherd can run tasks in minions. (see #60).
    • Deploy static site to a static minion with Nginx/Apache and a single HTML file.
    • (optional) Re-enable deploying D7 platform/site to the minion.

pre-Alpha2

  • Re-model variable structure for dispatching tasks (see #44).

  • Consolidate per-Task roles into Ansible task-files in Platform/Application-level roles. (see #91

  • Re-name Aegir entity classes, etc. (see #76).

  • One goal: define DX APIs to allow Aegir community devs to plug-in.

    • Need to figure out the ideal level of docs to allow other Aegir community folks interested in contributing to do so.

  • Another goal is to think through some outstanding architectural and workflow questions, considering our current understanding and goals for Aegir overall, in some ways broader than ever. For example: the data we pass between the front-end and back-end, and vice versa. where do we store its metadata? How do we maintain state of the system overall?

    • Sometimes we have transient data (e.g. state of a task), and the front-end polls for it, generally.
    • Sometimes we have a blob of data (e.g. the set of packages associated with a codebase), relatively stable, but periodically needing refresh.
    • Sometimes we have a list of data (e.g. a task log in progress), where we append to a list of field values regularly, and want to update the page in real-time, if somebody’s watching.
    • So the quetsion is: how do we handle these different kinds of data? In what ways are they different? What’s the conceptual model for front-end and back-end to use these well?

  • Also architecture-wise: codebases and sites are currently entity types, that are bundleable. Currently, to create one we inherit from a template/base class; there’s no other behaviour.

    • We might want to add more separate entity types for other kinds of things, such as different behaviours for servers and services than for codebases and sites. (Maybe?)

pre-Alpha3

  • Introduce “host” and “services” entities, to model Aegir3 behaviour for minions. (see #92).

pre-Alpha4

  • Implement alternative task types, so dispatcherd can run more than just Ansible playbooks. (see #52).

Here we intended to explore the idea of making the back-end tasks “pluggable” in the sense that Ansible playbooks is only one possibility. Currently the obvious second back-end type would be Terraform, but we’ll use this case as a basis to make Aegir flexible as to what kind of tasks to run in the backend (could be shell scripts, other config management or provisioning tools, etc.)

While our direct work was on the backburner later in 2022 and 2023, a new pluggable Backend presented itself, in the form of Kubernetes. We’ve therefore taken up this aspect more directly.