Guide: libvirt Provider

Register a libvirt/KVM host with banlieue, import a guest image onto it, and run a VM.

Unlike the vSphere provider, this one speaks libvirt's RPC protocol directly — a first-party client in banlieue-libvirt, no libvirt-dev, no virsh subprocess, no third-party crate (ADR-0011). Two consequences shape everything below:

• Mutual TLS is the only transport. qemu+ssh:// and qemu+tcp:// are rejected at reconcile time rather than silently retried, because there is no ssh client to tunnel over.
• The x509 client certificate is the credential. libvirtd runs with auth_tls="none", so there is no password anywhere in this guide.

Scope

The libvirt provider verifies host capabilities, imports images, and — since ADR-0050 — provisions VMs through the LibvirtMachine CRD.

cloud-init user-data is delivered too, as of ADR-0054.

Prerequisites

• A libvirt host reachable from the cluster, running libvirtd with TLS enabled on port 16514.
• The core controller and the operator installed — see Core Controller and Provider Lifecycle & Install.
• For image import: banlieue-imagebuilder and the Kairos operator, per Using banlieue-imagebuilder.

1. Enable TLS on the libvirt host

scripts/bootstrap-libvirt-tls.sh provisions the PKI (CA, server certificate, client certificate), writes /etc/libvirt/libvirtd.conf, and restarts the socket units in the right order.

scp scripts/bootstrap-libvirt-tls.sh kvm-1.example:/tmp/
ssh kvm-1.example 'sudo /tmp/bootstrap-libvirt-tls.sh'

It leaves the credentials on the host, in libvirt's documented locations:

/etc/pki/CA/cacert.pem            the CA that signed the server certificate
/etc/pki/libvirt/clientcert.pem   the client identity
/etc/pki/libvirt/private/clientkey.pem

Copy them down to the machine you will run virsh, the live tests, or kubectl from. Keep one directory per host — several hypervisors may share a CA but each has its own server certificate, and a single flat directory quietly becomes "whichever host I set up last":

mkdir -p ~/.config/banlieue/<host>/libvirt
cd ~/.config/banlieue/<host>/libvirt
scp <host>:/etc/pki/CA/cacert.pem              ca.pem
scp <host>:/etc/pki/libvirt/clientcert.pem     client-cert.pem
sudo scp <host>:/etc/pki/libvirt/private/clientkey.pem client-key.pem
chmod 600 client-key.pem

The filenames change on the way down, and that is deliberate

The host uses libvirt's names (cacert.pem, clientcert.pem, clientkey.pem); the workstation copy uses ca.pem, client-cert.pem and client-key.pem, which is what LIBVIRT_TLS_DIR consumers expect — every live test in this repository reads exactly those three.

Copy them across without renaming and the tests fail with No such file or directory, which reads like a missing directory rather than a naming mismatch.

Keep the directory outside any repository; client-key.pem is a private key and is mode 600 above for that reason.

Confirm the host answers before involving Kubernetes:

virsh -c "qemu+tls://kvm-1.example/system" pool-list --all

If that hangs rather than errors, the usual cause is a firewall dropping 16514.

2. Create the credentials Secret and CA ConfigMap

The Secret carries the client identity; the ConfigMap carries the CA that signed libvirtd's server certificate. A CA is required — a private CA is used in every realistic deployment, so falling back to public trust roots would only fail later and less clearly.

CREDS=~/.config/banlieue/<host>/libvirt

kubectl -n banlieue-system create secret generic libvirt-edge-creds \
  --from-file=tls.crt="$CREDS/client-cert.pem" \
  --from-file=tls.key="$CREDS/client-key.pem"

kubectl -n banlieue-system create configmap libvirt-edge-ca \
  --from-file=ca.crt="$CREDS/ca.pem"

The Secret keys are tls.crt / tls.key regardless of the filenames on disk — those are what the provider reads (crates/banlieue-provider-libvirt/src/credentials.rs), and the CA ConfigMap's key defaults to ca.crt.

3. Register the Provider

examples/04-provider-libvirt-edge.yaml (excerpt)

apiVersion: banlieue.io/v1alpha1
kind: Provider
metadata:
  name: libvirt-edge-host-7
  namespace: banlieue-system
spec:
  providerClassRef:
    name: libvirt
  connection:
    endpoint: qemu+tls://kvm-7.edge.example/system
    credentialsRef:
      name: libvirt-edge-creds
    caBundle:
      configMapRef:
        name: libvirt-edge-ca
  capabilities:
    storageClasses:
      - name: gold          # admin asserts: on this host, gold = nvme pool
        target:
          pool: nvme-pool
      - name: standard
        target:
          pool: default
    networkClasses:
      - name: prod
        target:
          network: br-prod

capabilities is declared, not discovered. The provider connects and narrows your declaration to what is really present, publishing the result as status.failureDomains[]. A pool you named that does not exist on the host is dropped and reported — which is the point of probing at all, since a Provider reporting Ready without ever having reached the host is actively misleading.

kubectl apply -f examples/04-provider-libvirt-edge.yaml
kubectl -n banlieue-system get provider libvirt-edge-host-7

Applying the Provider is all that is needed: the operator creates the Deployment, ServiceAccount, Role, RoleBinding and ClusterRoleBinding for it (ADR-0012).

A libvirt host is a single failure boundary, so exactly one failure domain is published per Provider — unlike vSphere, where a datacenter/cluster hierarchy yields several.

4. Import an image

With a VMImage whose source has providerClass: libvirt and kind: Url, the pipeline runs in two halves (ADR-0010):

1. banlieue-imagebuilder turns the OCI reference into a raw cloud image on a PVC and publishes status.buildArtifact (kind: cloudImage).
2. This provider waits for phase: Ready, then creates one import Job per storage pool the Provider advertises.

kubectl get vmimage kairos-ubuntu-2404 \
  -o jsonpath='{.status.perProvider[*].zones[*]}' | jq

# Watch the transfer. Import Jobs run in the build namespace (ADR-0016),
# where the shared artifacts PVC lives — not in the Provider's namespace.
kubectl -n banlieue-imagebuild get jobs -l banlieue.io/vmimage=kairos-ubuntu-2404
kubectl -n banlieue-imagebuild logs -l banlieue.io/vmimage=kairos-ubuntu-2404 -f

Why a Job

Importing a guest image moves gigabytes. A reconcile loop blocked for minutes on I/O stops reconciling everything else, holds memory proportional to the image, and leaves a half-written volume behind if the pod restarts.

The Job runs the banlieue binary itself — banlieue provider libvirt import — not a third-party virsh/qemu-img image, so the data path stays inside banlieue's own supply chain and the same banlieue-libvirt code is exercised in both roles.

Three properties are worth knowing when reading a failure:

• Job names are deterministic (import-<image>-<provider>-<pool>). A re-reconcile adopts a running import rather than starting a second copy of a multi-gigabyte transfer.
• backoffLimit is 1. A partial upload is resumable only by starting over, so retrying forever would hammer the host for no benefit.
• The import is idempotent. A volume already present in the pool is left alone, so the one retry can finish the work rather than trip over its predecessor's.

Running an import by hand

The Job's flags are stable, so a failed import can be reproduced directly:

banlieue provider libvirt import \
  --vmimage kairos-ubuntu-2404 \
  --provider libvirt-edge-host-7 \
  --provider-namespace banlieue-system \
  --pool default \
  --source /artifacts/kairos-ubuntu-2404.raw

It reads the Provider for the endpoint and TLS material rather than taking them on the command line — passing a private key as a process argument would expose it through /proc to anything sharing the namespace.

5. Run a VM

Everything above is setup. This is the part that produces a machine.

examples/18-virtualmachine-libvirt.yaml is a complete, applyable set: a VMImage, a VMClass and a VirtualMachine. Apply it and watch both objects:

kubectl apply -f examples/18-virtualmachine-libvirt.yaml
kubectl get virtualmachine,libvirtmachine -n banlieue-system -w

What runs where

VirtualMachine  ──(banlieue-controller schedules)──▶  LibvirtMachine
                                                              │
                                        (banlieue-provider-libvirt realises)
                                                              ▼
                                                        libvirt domain

The controller never talks to libvirt and the provider never reads a VirtualMachine. They communicate only through the LibvirtMachine object and its status — the CRD-only contract that lets the same provider serve a CAPI Machine unchanged.

The domain's name

A domain is named <namespace>-<name>, so web-01 in banlieue-system becomes banlieue-system-web-01.

The qualification is not decoration. One libvirt host has a single flat domain namespace, while two Kubernetes namespaces can each hold a web-01 — and DOMAIN_DEFINE_XML is an upsert, so the second one would silently redefine the first's domain rather than failing. vCenter has no equivalent problem because its inventory is a folder tree.

Volume names follow from the domain name for the same reason: banlieue-system-web-01-os.qcow2.

The two disk shapes

Which one a machine gets follows from its image's installMode (ADR-0040), not from anything set per-VM:

installMode	Shape	What the provider does
Immediate	backingVolume	Creates a qcow2 overlay over the imported image. Fast.
Deferred / Manual	installMedia	Creates an empty qcow2 disk and attaches the image as a boot CD-ROM. The guest installs itself.

Deferred is slower (a full unattended install per VM), but it is one of two shapes that produce a disk sealed to that VM's own vTPM. On libvirt it is also the simpler path: no template, no backing chain, nothing to copy.

The other is an Immediate image that declares spec.firstBootSealing (ADR-0088, Kairos v4.4.0 or later). The imagebuilder's raw cloudImage carries the sealing policy on COS_OEM and no key material. Each overlay's first boot runs Kairos's recovery reset, which lays out COS_PERSISTENT, and the next boot seals it to that domain's own swtpm before anything mounts it; the base volume is never written. Without a vTPM the guest halts on a "KAIROS BOOT FAILED" screen instead of booting plaintext, and GuestReady waits for the guest to report the partition sealed (/run/banlieue/sealed, read through the guest agent). Per-VM user-data still lands on COS_OEM in plaintext, so a sealed disk does not protect secrets passed in spec.userData.

The overlay's backing format is taken from the backing volume's name (.qcow2/.qcow → qcow2, otherwise raw), never from its contents. libvirt refuses to probe a backing file's format because a raw image whose first bytes resemble a qcow2 header would be reinterpreted as one, and the backing file it then names can be any path the daemon can read. banlieue does not reintroduce that.

The installer is ejected once the guest reports installed

An installMedia machine does not keep its CD-ROM forever. When the guest's GuestReady marker appears (ADR-0043), the provider ejects the medium with virDomainUpdateDeviceFlags and records status.installMediaDetached: true (ADR-0044). Two reasons:

• the ISO carries the image's baked cloud-config overlay, and a workload that can still mount it can read that overlay;
• it is bootable, so a domain that keeps it is one reboot away from re-running the installer over its own disk.

GuestReady is published only after the eject succeeds. That ordering is what lets a VirtualMachinePool bind on GuestReady alone and still be sure no bound member has install media attached. A member whose eject fails therefore never becomes available, and is reaped at provisioningTimeoutSeconds — look for reason=InstallMediaAttached on the GuestReady condition:

kubectl get libvirtmachine my-vm -o jsonpath='{.status.installMediaDetached}'
kubectl describe libvirtmachine my-vm | grep -A3 GuestReady

The tray is never force-ejected. A guest holding it locked is still reading the installer, and that is worth surfacing rather than overriding.

The cloud-init seed is not ejected

Only the installer is removed. The NoCloud cidata ISO stays attached, because cloud-init re-reads its datasource on every boot. A guest can therefore still read its own rendered user-data from CIDATA — a deliberate, recorded trade-off (threat model §8), not an oversight.

EFI needs no configuration

firmware: efi on a VMClass renders <os firmware='efi'>, which makes libvirt select the loader and varstore template from its own firmware descriptors. There is deliberately no OVMF path to set: Debian, Fedora and Arch each put it somewhere different, so a hardcoded path is not portable.

Each domain gets its own varstore. Without that, every EFI domain on the host would share one set of UEFI variables — and for a Secure Boot domain, that set is the enrolled key database.

vTPM

tpmEnabled: true on a VMClass attaches an emulated TPM 2.0 (tpm-crb, backed by swtpm). Two things to know:

• The Provider must advertise it. The scheduler rejects a candidate whose failure domain does not list vtpm in capabilities.features. This is never auto-discovered (ADR-0039) — set it by hand once you have confirmed swtpm works on the host.
• Never clone a domain that has TPM state. swtpm keys its state by domain UUID, so a fresh domain always gets a fresh TPM — but a copy carries the original's sealed keys, which is the shared-vTPM problem ADR-0040 exists to avoid. banlieue never clones a libvirt domain, and nothing in this guide should lead you to.

cloud-init user-data

spec.userData on a VirtualMachine reaches a libvirt guest as a NoCloud seed: a small ISO labelled CIDATA, attached as a second CD-ROM, holding meta-data and user-data at its root. vSphere needs no such disk — it has guestinfo — but libvirt has no hypervisor channel, so the payload has to arrive as a filesystem.

banlieue builds that image in-process. There is no genisoimage in the provider image and no subprocess: the image stays distroless, and a subprocess would need the payload on disk to hand it a path, writing the guest's bootstrap credentials into the container's filesystem on every reconcile.

The seed carries a generated meta-data, and an empty #cloud-config as user-data when you set none: cloud-init requires both and ignores a seed missing either (before v0.4.13 a VM without userData got no user-data file, and its whole seed was ignored):

instance-id: <the domain's UUID>
local-hostname: <the domain name>

instance-id comes from the domain's UUID deliberately. cloud-init keys "have I already run for this instance?" on that value, so a rebuilt VM — a replaced pool member, say — is a new instance by construction. Reusing an id would make a fresh VM skip every per-instance module as though it had merely rebooted.

The volume is <domain>-cidata.iso in the machine's pool, and it is deleted with the domain.

For the full picture — what the seed contains, why it is Joliet, how to verify a guest consumed it, and the two ways to get a VM that boots but never networks — see cloud-init on libvirt.

Why the image is Joliet

cloud-init looks for user-data and meta-data. ISO9660 Level 1 names are 8.3, uppercase, and exclude the hyphen — meta-data cannot be written in the primary tree at all. A Joliet supplementary tree carries the real names, which is why the documented genisoimage command passes -joliet. You can confirm any seed banlieue produced by mounting it: hdiutil attach seed.iso (macOS) or mount -o loop seed.iso /mnt.

Deletion

Deleting the VirtualMachine blocks at the Kubernetes API until the domain and its disks are really gone. The order is destroy → undefine → delete volumes, and the undefine always passes NVRAM | TPM | MANAGED_SAVE.

Those flags are not optional. libvirt fails an undefine outright on a UEFI domain with no --nvram, and without --tpm a deleted VM leaves its swtpm state behind — which for a sealed sandbox means leaking the very per-VM secret material the vTPM existed to isolate. The provider re-looks-up the domain afterwards rather than trusting the undefine, because a teardown that reports success while leaving a domain defined is how the next VM of the same name quietly inherits a stale one.

The shared boot image is never deleted: it belongs to the VMImage, and removing it with one machine would break every other VM using it.

Rotating credentials

1. Issue a new client certificate from the CA the libvirt host trusts.
2. Update the Secret's tls.crt and tls.key in place. If the CA itself changes, add the new CA to the host's trust store first, then update the CA ConfigMap.
3. Wait about 30 seconds, and check that the Provider stays Ready (kubectl get provider <name>).
4. Revoke the old certificate, or remove the old CA from the host.

banlieue never watches the Secret. Each reconcile reads it by name and builds a fresh connection from it, so the new credential is in use within one requeue interval (about 30 seconds), with no restart. Watching would need read access to every Secret in the namespace, which the provider deliberately does not have (ADR-0095). To cut a credential off immediately, revoke it on the backend: that stops it everywhere at once.

Troubleshooting

Symptom	Cause
scheme "qemu+ssh" is not supported	This provider is TLS-only (ADR-0011). Use qemu+tls://host/system.
Ready=False, reason CapabilitiesIncomplete	A declared pool or network is not on the host. The message names which.
Ready=False, reason ConnectFailed	TLS or reachability. Check 16514, then reproduce with virsh -c qemu+tls://….
Ready=False, reason CredentialsUnavailable	The Secret or CA ConfigMap is missing a key. tls.crt / tls.key / ca.crt.
Zone stuck at BuildPending	status.buildArtifact is not Ready yet — the problem is upstream, in banlieue-imagebuilder.
Import Job 403 on jobs	The ProviderClass is missing additionalRules. See examples/11-providerclass-libvirt.yaml.
Import Job Pending, ServiceAccount not found	The Job's namespace and the provider's differ. The provider falls back to that namespace's default SA, which needs the grant.
VirtualMachine stuck Scheduled=False, reason ImageNotReady	The VMImage has no ready libvirt source. For a BackingFile, the volume must already be in every pool the Provider targets: its row then says VolumeNotFound and names the pools that lack it, or HostUnreachable if the host could not be asked.
LibvirtMachine Ready=False: volume … is not in pool	The image has not finished importing, or a BackingFile names a volume that is not there. virsh vol-list <pool>.
LibvirtMachine Ready=False: storage pool … does not exist	spec.pool came from the Provider's storageClasses mapping. The mapping names a pool the host does not have.
VirtualMachine never gets a LibvirtMachine	The chosen Provider's class has no builder in banlieue-controller. The Scheduled condition names the class.
tpmEnabled class will not schedule	The Provider's failure domain does not advertise vtpm in capabilities.features. Never auto-discovered (ADR-0039).
Guest boots but ignores its user-data	Check the seed mounted: virsh domblklist <domain> should list a second CD-ROM. Mount the volume and confirm user-data is at the root.
cloud-init skips per-instance modules on a rebuilt VM	instance-id was reused. banlieue derives it from the domain UUID, so this means the domain was redefined rather than recreated.
Deletion hangs	Expected while the domain is still being torn down. If it persists, the message on LibvirtMachine says which step failed — teardown errors are never swallowed.

Next: keep VMs standing by

Once a single VirtualMachine works here, a VirtualMachinePool keeps a set of them already-provisioned so a consumer does not wait out provisioning. Members are ordinary VirtualMachines on this same Provider — nothing in this guide changes.

Full schema reference

• Provider API reference
• VMImage API reference
• LibvirtMachine API reference
• ADR-0050 — LibvirtMachine: the InfraMachine contract on libvirt