Using an image-based OS brings advantages and challenges. One challenge is the customization of a read-only image with additional host-level software and configuration, and how to manage this customization through the lifetime of a machine. It turns out that configuration management has similar problems to solve because reliable reconfiguration often struggles with atomic updating and cleaning of old state.
Instead of placing and managing a bunch of config files in /etc and binaries in /opt, we can now extend the filesystem hierarchy at runtime through overlay mounts of extension images with systemd-sysext and systemd-confext. Extension images can be supplied by the user, by the OS vendor, or third parties. They are either bound to the OS version or may be independently updated.
The use of single filesystem image files mounted at runtime allows to reliably manage the changes compared to unpacking files to the root filesystem and trying to keep track of them. For an immutable OS the runtime overlay also allows to bring in deeper changes that previously required a custom image build when changing contents on the read-only /usr partition.
This presentation will share use cases for extension images in Flatcar Container Linux, as well as possible uses with general purpose distros and for configuration management. It will cover rough edges, conceptual limitations, and future improvements.
