Cloud Hypervisor vTPM pool timings¶
Measured 2026-10-10 and 2026-10-11, banlieue v0.4.1, one host: a run of fourteen members on stock Cloud Hypervisor v53.0, then ten on v53.0 with the vTPM fix below. A point-in-time measurement, not a living table: rerun it with
make ch-pool-timingswhen the provider's pacing, the image or the VMM version changes. The broader per-provider numbers are in Provider comparison.
This page answers one question for roadmap 17's sandbox pools on Cloud
Hypervisor: how long does each stage of a GuestReady pool take, from the
pool's creation to a claimed sandbox being destroyed? Every member is a
tpmEnabled machine that installs itself from an installer ISO
(installMode: Deferred), seals its persistent partition to its own vTPM, and
reports over vsock (ADR-0065). Nothing is cloned; every member is a full
encrypted install.
Summary¶
| What | Result |
|---|---|
A member becomes claimable (GuestReady), from VirtualMachine creation |
108 s median (97–111 s), 10 of 11 healthy members; one outlier at 190 s |
Three members installing at once, all healthy, to Warm |
107.5 s (the refill after three simultaneous claims) |
Refill after one claim, back to Warm |
103 s and 122 s |
Refill after three claims at once, back to Warm |
107.5 s when all three installs succeeded |
Claim created to Bound |
0–1 s server-side (creationTimestamp to boundAt), all 8 claims |
| Claim deleted to claim, VM and host unit all gone | 6.4 s and 6.7 s for 4 claims at once |
| Pool deleted to pool and members gone | 5.1 s |
Six members installing at once, to Warm |
141 s (installs slow under contention: 104 s each, against 77 s at three) |
| Members lost to the vTPM fault below | 3 of 14 on stock v53.0, each costing the pool's full provisioningTimeoutSeconds (900 s); 0 of 10 with the fix |
vSphere's 2-member GuestReady pool warmed in 130.3 s (roadmap 17); this one
brings three concurrent members to Warm in 107.5 s with no split-image work (roadmap 18).
Sealed pre-laid members on XFS (2026-11-08)¶
The table above is a Deferred install pool. A pre-laid, first-boot-sealed
image (ADR-0113,
ADR-0112) on a host whose guest storage reflinks (XFS) is four to five times
faster:
| Image | creation → GuestReady, median (range) |
Three at once |
|---|---|---|
| Debian 13, kairos-init v4.4.0 | 25 s (22–26) | 22–26 s |
| Kairos Hadron v0.6.3 | 21.5 s (19–25) | 19–25 s |
Of that, the host's part (pick-up, a reflink clone, the vTPM, starting
swtpm and the VMM) is about 3.5 s, the firmware 0.4 s, GRUB 0.2 to 1 s, and
the guest (kernel, initramfs, sealing COS_PERSISTENT, the reports) 16 to
21 s. On ext4 every clone is a full copy and the firmware's first disk flush
waits for it: 6.8 s more per member, queued behind every other clone in a
warming pool. See the host guide, Put guest storage on a filesystem with
reflinks, and roadmap 18 phase C for the per-stage table.
Environment¶
- Host: two 8-core Intel Xeon E5-2630 v3 (32 threads), 62 GiB, one SATA SSD, ext4; Debian 13. The same host as the Cloud Hypervisor column in Provider comparison. All fourteen members landed on it: with two Cloud Hypervisor Providers serving the class, the scheduler broke the tie alphabetically rather than spreading. Since ADR-0098 it spreads pool members across failure domains.
- Hypervisor: Cloud Hypervisor v53.0, swtpm 0.7.1, one
banlieue-ch@unit and onebanlieue-swtpm@unit per guest (ADR-0063, ADR-0065). - banlieue: v0.4.1 provider on the host; v0.4.1 controller and operator on a separate three-node management cluster.
- Class: 2 vCPU, 4 GiB, 20 GiB thin OS disk,
tpmEnabled: true, one NIC on the host's NAT bridge. - Image: Kairos Hadron v0.4.0 core installer ISO,
BackingFilesource,installMode: deferred. User-data: oneusers:entry,install.autoto the empty disk withencrypted_partitions: [COS_PERSISTENT],reboot: true, and abootstage that runssystemd-notify phase=installedovervsock-stream:2:1024once the installed system is active. - Pool:
warmReplicas: 3,readiness: GuestReady,provisioningTimeoutSeconds: 900.
Per-stage timings¶
Times are from the VirtualMachine's creationTimestamp, over the 11
members that reached GuestReady. Median, then min–max.
| Stage | From | Median (min–max), s |
|---|---|---|
| Scheduled; infrastructure CR created | VM created | 0.0 (0.0–1.0) |
| Guest uid allocated on the host | VM created | 0.7 (0.3–1.7) |
vTPM manufactured (swtpm_setup run time) |
its unit start | 0.2 (0.1–0.2) |
| swtpm started | VM created | 4.1 (1.2–5.1) |
| VMM unit started | VM created | 7.1 (1.2–8.2) |
| VM created in the VMM; the installer boots | VM created | 10.2 (1.4–11.3) |
Ready (the VM is running) |
VM created | 10.0 (1.0–11.0) |
| Install: installer boot to the installed system's kernel | VM boot | 77.0 (72.8–82.0) |
| Installed system's boot to its report | its kernel start | 18.7 (18.4–105.9) |
| Report to installer ejected | the report | 0.2 (0.0–0.2) |
GuestReady, the member is claimable |
VM created | 108.0 (97.0–190.0) |
Where the 108 s goes, for a typical member:
0 s VM created, scheduled, infra CR created
1 s guest uid allocated; vTPM manufactured (0.2 s of work)
4 s swtpm started
7 s VMM unit started
10 s VM created and booted from the installer ISO -> Ready
...installer: partition, install, encrypt COS_PERSISTENT
to the vTPM, reboot (about 77 s)...
87 s installed system's kernel starts
106 s boot stage reports phase=installed over vsock
106 s provider ejects the installer (0.2 s later) -> GuestReady
Reading it.
- The install is 70 % of the time and is all the guest's own work:
writing the image, creating and sealing the LUKS volume, and a reboot. The
provider cannot shorten it. First-boot sealing (ADR-0112, 2026-10-28) has
since removed the install for an
Immediateimage: a sealed pool of two warms in 125 s, a recovery boot, a reset and a second boot. Roadmap 18 phase C is what is left. - About 9 s of the first 10 s is pacing, not work. The provider advances one step per reconcile pass (manufacture, then swtpm, then the VMM), and passes are about 3 s apart; the steps themselves take well under a second. One member went through all of them in 1.4 s, so the pacing is not inherent. This is a cheap optimization target.
- The installed boot takes about 19 s to report. The report is the last
thing its
bootstage does. - One healthy member reported 106 s after its installed boot, not 19 s, and so became claimable at 190 s. Its install time was normal; the delay was inside the installed guest before the report ran. Not investigated.
Claims¶
Eight claims in two admission modes against the same warm pool: four created by the claim's own subject, an OIDC user, through ADR-0047's subject-authorization policy; four created by a declared broker on behalf of another subject. Each mode made one claim, waited for the pool to refill, then three claims at once.
| Subject's own claim (OIDC) | Broker on behalf of a subject | |
|---|---|---|
creationTimestamp to boundAt (server-side) |
0, 1, 0, 0 s | 0, 0, 0, 0 s |
Three claims at once, all Bound |
2.5 s observed | 4.3 s observed |
| Refill after one claim | 103.1 s | 121.6 s |
| Refill after three claims | 107.5 s | 1101 s (one install lost to the vTPM fault, replaced after 900 s) |
| Delete to claim gone | 3.6–5.4 s | 3.6–5.6 s |
| Delete of all four to claims, VMs and host units gone | 6.4 s | 6.7 s |
Reading it.
- Binding is effectively instant: the controller binds a warm member in the same second the claim is created, in both modes. Observed times of 2–8 s are the harness's poll interval (about 3 s), not the system.
- Admission mode makes no measurable difference at one-second resolution.
- A claim never waits for an install while the pool has a warm member; the refill runs behind it. A claim that arrives when the pool is empty waits for a full install (about 108 s), or longer if that install hits the vTPM fault.
- Release is a destroy, and it is complete in about 6.5 s: the claim and
its
VirtualMachinego, and the guest's VMM unit is stopped on the host (ADR-0047 Decision 6). Four releases at once take no longer than one.
The vTPM fault, and its fix¶
On stock Cloud Hypervisor v53.0, three of fourteen members never reported.
In the guests that were inspected, the kernel logged tpm_recv: error -5
about 0.2–0.4 s after the Kairos installer first opened /dev/tpmrm0, then
/dev/tpmrm0 was "State not recoverable" and the installer exited at the
encryption step. swtpm logged nothing.
Cause: a 100 ms read in Cloud Hypervisor, not banlieue. The VMM passes
each TPM command to swtpm over a stream socket and reads the response with a
single recvfrom, on a socket whose receive timeout is 100 ms
(tpm/src/emulator.rs, prepare_data_fd and deliver_request). A command
swtpm takes longer than that to answer (key generation, which LUKS sealing
starts with, on a busy host) is reported to the guest as failed: the CRB
device sets CTRL_STS.tpmSts, the TPM's fatal-error bit, which the kernel's
crb_recv turns into -EIO and which nothing ever clears. A single read can
also return part of a response. The code is unchanged on upstream main as
of 2026-10-11.
Proof. Throttling one guest's swtpm to 3 % CPU with systemd CPUQuota
makes the fault deterministic on stock v53.0: the guest kernel's own TPM
probe at boot fails with error -5, and /dev/tpmrm0 never appears. With
the fix, the same throttle produces slow answers instead of errors: the
member installs, seals COS_PERSISTENT to its vTPM and is claimable in about
102 s, as fast as an unthrottled one.
Fix. Raise the data socket's receive timeout from 100 ms to 120 s
(Linux allows TPM2 commands up to 300 s), still set once when the socket is
prepared, and read the 10-byte response header and then exactly the size it
declares with read_exact, so a partial read is never taken as a whole
response. A declared size smaller than a header or larger than the buffer is
an error. Unit tests cover a response split across delays, the timeout, an
impossible size, and swtpm closing mid-response. This is the shape the
maintainer asked for in review of
cloud-hypervisor#9054.
An earlier revision that polled a 100 ms timeout up to a deadline had ten
installs, six of them concurrent, with no fault (2026-10-11).
The read path must not call setsockopt
TPM commands are served on the guest's vCPU threads, and Cloud
Hypervisor's vCPU seccomp filter (vcpu_thread_rules in
vmm/src/seccomp_filters.rs) allows no setsockopt. One revision of
the fix set the receive timeout inside the read loop, and every guest's
VMM died with SIGSYS on its first TPM command (2026-10-12: "Probable
seccomp violation: si_syscall=54 ... thread=vcpu"). Its unit tests
passed, because they run without seccomp. Set socket options when the
socket is prepared, on the VMM thread, never when reading.
Two hosts, 2026-10-12. With the reviewed fix (cloud-hypervisor#9054 at 3d6f462cf) installed on two hosts (one 96
cores, one 32), each running its own Provider against one management
cluster (banlieue v0.4.1), make ch-pool-claim-e2e with
BANLIEUE_E2E_READINESS=GuestReady, a tpmEnabled class and the Kairos
Hadron v0.4.0 installer passed on both: warm-up of two members 115 s and
110 s, one-member refill 110 s and 100 s, claim bound in 5 s, the released
guest gone from the host in 5 s. Ten installs in all (five per host), every
one sealed to its vTPM and reported (an install that hits the TPM fault
never reports), and no VMM crash in either host's journal.
Status. The fix is a Cloud Hypervisor change, reported upstream as
cloud-hypervisor#9053 (2026-10-11). A
host takes a patched VMM through
banlieue host cloud-hypervisor install --vmm-url … --vmm-sha256 … --vmm-version …
(ADR-0084). Until a host has it, a shorter provisioningTimeoutSeconds
(about 3× the 108 s median) bounds the cost of each fault. Tracked in
roadmap 17 phase G.
How it was measured¶
make ch-pool-timings (scripts/ch-pool-timings.py; its module docs list
the settings) creates the pool and claims through the Kubernetes API and polled the pool, its VirtualMachines, their
CloudHypervisorMachines and the claims about every 3 s, recording the
first time each transition was seen. Every stage above uses the more precise
source available rather than the poll:
- Cluster stages use the objects' own timestamps (
creationTimestamp, conditionlastTransitionTime, the claim'sboundAt), to the second. - Host stages use the provider's and systemd's journal on the host
(
guest uid allocated,manufacturing the vTPM, thebanlieue-swtpm-setup@unit's start and finish,starting swtpm,starting VMM unit,creating VM,ejecting the installer, teardown), to the microsecond. - Guest stages come from each installed guest's own journal, read over
SSH while it was alive: the installed boot's first entry and the report.
The installer's boot is not kept in the guest's journal, so the install is
measured from the host's
creating VMto the installed kernel's start, and includes the installer's reboot.
Host and guest clocks were NTP-synchronised; the guest's report and the
host's GuestReady agree to within 0.6 s throughout, which bounds the skew.