Fedora CoreOS (FCOS)

Starting with a provisioned infrastructure, such as on AWS, or provisioning the infrastructure yourself.

Supplying a few pieces of information, such as credentials and cluster name, in an install-config.yaml file when running openshift-install.

Kubernetes workload objects, such as DaemonSet and Deployment: If you need to add services or other user-level features to your cluster, consider adding them as Kubernetes workload objects. Keep features outside node configurations. This approach reduces the risk of breaking the cluster during upgrades.

Day-2 customizations: If possible, open a cluster without making any customizations to cluster nodes and make necessary node changes after the cluster is up. Those changes are easier to track later and less likely to break updates. Creating machine configs or modifying Operator custom resources are ways of making these customizations.

Day-1 customizations: Some customizations must be implemented when the cluster first boots. Ways exist for modifying your cluster so changes are implemented on first boot. Implement day-1 customizations through Ignition configs during openshift-install. Alternatively, add boot options during user-provisioned ISO installs.

Kernel arguments: If particular kernel features or tuning is needed on nodes when the cluster first boots.

Disk encryption: If your security needs require that the root file system on the nodes is encrypted, such as with FIPS support.

Kernel modules: If a particular hardware device, such as a network card or video card, does not have a usable module available by default in the Linux kernel.

Chronyd: If you want to provide specific clock settings to your nodes, such as the location of time servers.

Installer-provisioned infrastructure: Some cloud environments offer preconfigured infrastructures so that you can start an OKD cluster with minimal configuration. For these types of installations, you can supply Ignition configs that place content on each node so it is there when the cluster first boots.

User-provisioned infrastructure: If you are provisioning your own infrastructure, you have more flexibility in how you add content to a FCOS node. For example, you could add kernel arguments when you boot the FCOS ISO installer to install each system. However, in most cases where configuration is required on the operating system itself, it is best to provide that configuration through an Ignition config.

Ignition runs from an initial RAM disk that is separate from the system you are installing to. Because of that, Ignition can repartition disks, set up file systems, and perform other changes to the machine’s permanent file system. In contrast, cloud-init runs as part of the machine init system when the system boots. This makes foundational changes like disk partitioning more difficult. With cloud-init, difficulty exists to reconfigure the boot process while you are in the middle of the node boot process.

Ignition is meant to initialize systems, not change existing systems. After a machine initializes and the kernel is running from the installed system, the Machine Config Operator from the OKD cluster completes all future machine configuration.

Instead of completing a defined set of actions, Ignition implements a declarative configuration. Ignition checks that all partitions, files, services, and other items are in place before the new machine starts. Ignition then makes the changes, like copying files to disk that are necessary for the new machine to meet the specified configuration.

After Ignition finishes configuring a machine, the kernel keeps running but discards the initial RAM disk and pivots to the installed system on disk. All of the new system services and other features start without requiring a system reboot.

Because Ignition confirms that all new machines meet the declared configuration, you cannot have a partially configured machine. If a machine setup fails, the initialization process does not finish, and Ignition does not start the new machine. Your cluster never contain partially configured machines. If Ignition cannot complete, the machine is not added to the cluster. You must add a new machine instead. This behavior prevents the difficult case of debugging a machine when the results of a failed configuration task are not known until something that depended on it fails at a later date.

If there is a problem with an Ignition config that causes the setup of a machine to fail, Ignition does not try to use the same config to set up another machine. For example, a failure could result from an Ignition config made up of a parent and child config that both want to create the same file. A failure in such a case would prevent that Ignition config from being used again to set up on other machines until the problem is resolved.

If you have multiple Ignition config files, you get a union of that set of configs. Because Ignition is declarative, conflicts between the configs could cause Ignition to fail to set up the machine. The order of information in those files does not matter. Ignition sorts and implements each setting in ways that make the most sense. For example, if a file needs a directory several levels deep, if another file needs a directory along that path, the later file is created first. Ignition sorts and creates all files, directories, and links by depth.

Ignition can start with an empty disk and set up bare-metal systems from scratch by using PXE boot. This is a capability that cloud-init cannot do. In the bare metal case, the Ignition config is injected into the boot partition so that Ignition can find it and configure the system correctly.

The machine gets its Ignition config file. Control plane machines get their Ignition config files from the bootstrap machine, and worker machines get Ignition config files from a control plane machine.

Ignition creates disk partitions, file systems, directories, and links on the machine. It supports RAID arrays but does not support LVM volumes.

Ignition mounts the root of the permanent file system to the /sysroot directory in the initramfs and starts working in that /sysroot directory.

Ignition configures all defined file systems and sets them up to mount appropriately at runtime.

Ignition runs systemd temporary files to populate required files in the /var directory.

Ignition runs the Ignition config files to set up users, systemd unit files, and other configuration files.

Ignition unmounts all components in the permanent system that were mounted in the initramfs.

Ignition starts up the init process of the new machine, which in turn starts up all other services on the machine that run during system boot.