Ansible provides a handy, secure mechanism to allow multiple servers, since it uses SSH to communicate between VMs. We don’t want to be able to run arbitrary commands on the back-end, since this could easily lead to compromised security. Rather, the Ansible roles will represent a whitelist of commands, and safe variables.
CLI provides a more or less universal API for applications. By supporting CLI tools, as our principal backend engines, we can standardize mechanisms for both calling and gathering feedback from them. SSH provides a secure, proven, widely supported communication mechanism between hosts.
In contrast, interacting with various backends via web-based APIs would generally involve multiple authentication mechanisms, differing serialization formats, etc.
To the extent possible, secrets should neither be entered nor exposed via the UI. Ideally, these would be either generated on the backend or, where needed, deployed by an administrator via SSH.
These should likely mostly be situated on the queue worker, so as to be accessible when needed by ansible-playbook or other engines.