The vTPM that wouldn't boot: swtpm on Kolla 2025.1

Why I needed a TPM in a VM#

I’m standing up OpenBao, the open-source fork of HashiCorp Vault, on my OpenStack cloud. Vault-style servers start sealed after every reboot, and something has to unseal them. If that something is me typing in unseal keys, then a two-minute power cut at 3 a.m. leaves every secret-dependent service down until I wake up.

The usual fix is auto-unseal from another key store. But a second OpenBao just moves the problem: what unseals that one when the whole rack loses power? The chain has to end at something that comes back on its own. In my case that’s the cloud itself. A virtual TPM in the OpenBao VM can hold the unseal material, and the vTPM’s own secret lives in Barbican, which has to be up before any VM can start anyway.

The catch: the cloud runs on Dell M620 blades, which have no hardware TPM. So it had to be a software vTPM, using swtpm.

The stack, for context: Kolla-Ansible 20.5.0 deploying OpenStack 2025.1 on Ubuntu Noble, three converged blades, Cinder volumes on TrueNAS via The Cinder Driver i wrote for my company’s internal cloud.

The part that looked easy#

On paper, a software vTPM in Nova takes three things.

1. swtpm in the libvirt container. Kolla’s nova_libvirt image ships it. docker exec nova_libvirt which swtpm returned /usr/bin/swtpm, so that box was ticked.

2. Nova told it may use it. In the nova.conf override:

   1 2	[libvirt] swtpm_enabled = True

   Then kolla-ansible reconfigure -t nova.

3. A flavor that asks for a TPM: hw:tpm_version=2.0 and hw:tpm_model=tpm-crb.

Two small gotchas on the way, both worth knowing (Mainly for myself next troubleshooting session).

• My first flavor was public; the rest are private. I created it with the CLI and Horizon only offered Modify Access on the others. openstack flavor show on an existing flavor showed is_public: False and four projects, so the TPM flavor got recreated with --private and the same project list (This is mainly a consistancy issue not a real problem outside of correcting my internal documentation).
• That comparison also caught a missing property. Every other flavor carries hw:numa_nodes=1. These are dual-socket blades, and keeping each guest on one NUMA node is the whole reason I left Proxmox, where memory striped across both sockets made VMs unstable. A new flavor without it would quietly reintroduce that. Diff new flavors against an existing one. This one is much more critical for the flavours under 256GiB (on these nodes that have 512GiB ram) due to the way the memory in multisocket servers works but this is a moving goal post depending on the server config and probably worthy of its own blog post one day.

Exhausted all hosts#

The first test boot failed with the most unhelpful error Nova has:

1

Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance.

My first guess was that Nova hadn’t picked up the config. That was wrong, and the wording says why: exhausted all hosts means the scheduler found hosts that could take a TPM guest, and the build failed on every one of them.

openstack server event show gave the order it tried (slot 2, then 3, then 1) but blank tracebacks, so the real error was in the compute log on the first blade:

1
2
3

Calculated Secrets uuid ref: secrets/714b2ef2-...
libvirt.libvirtError: internal error: Could not run '/usr/bin/swtpm_setup'. exitstatus: 1;
Check error log '/var/log/swtpm/libvirt/qemu/instance-00000040-swtpm.log'

That first line mattered: Barbican worked, and the vTPM secret was stored. The failure was later, in swtpm itself.

Another trap: that log path is inside the container. cat on the host says the file doesn’t exist. docker exec nova_libvirt cat ... finally showed the actual problem:

1
2
3
4

Starting vTPM manufacturing as tss:tss
Successfully created RSA 2048 EK with handle 0x81010001.
Need read/write rights on statedir /var/lib/swtpm-localca for user tss.
swtpm_localca exit with status 1: An error occurred.

A permissions problem. That should have been the end of it.

Three fixes that didn’t work#

Fix 1: tell Nova to use the swtpm user. The directory on disk was owned by swtpm, not tss:

1

drwxr-x--- 2 swtpm root 4096 /var/lib/swtpm-localca

Nova has swtpm_user and swtpm_group options in [libvirt], defaulting to tss. Set them to swtpm, kolla reconfigure, boot again. Same error, and the log still said Starting vTPM manufacturing as tss:tss.

That was a useful failure. Nova’s options don’t choose the user. libvirt does, from its own qemu.conf; Nova’s settings only have to agree with it. And grep swtpm /etc/libvirt/qemu.conf inside the container returned nothing, so libvirt was on its built-in default: tss.

Fix 2: override qemu.conf. Kolla merges INI overrides like nova.conf, so I dropped a qemu.conf with the two swtpm lines next to it. Reconfigure. The container’s qemu.conf was unchanged.

Fix 3: the template path from upstream. Kolla-Ansible master has a task that looks for config/nova/qemu.conf.j2 (and a per-host variant) and uses it as the whole template. So I copied Kolla’s template, added the two lines, and saved it there. Another reconfigure, no failures in the recap, and the rendered file on the host, /etc/kolla/nova-libvirt/qemu.conf, still had no swtpm lines.

At that point I stopped guessing paths and asked the installed copy:

1

grep -rn -A12 "qemu.conf" .venv --include=config.yml

1
2
3

  with_items:
    - { src: "qemu.conf.j2", dest: "qemu.conf" }
    - { src: "libvirtd.conf.j2", dest: "libvirtd.conf" }

In kolla-ansible 20.5.0, qemu.conf comes from the role’s built-in template, full stop. The override I’d been reading about exists on master, not in my release. There is no supported way to set swtpm_user by config in 20.x.

The lesson I should have applied two rebuilds earlier: read the version you’re running, not the docs for the version you aren’t. Each wrong guess cost a multi-minute reconfigure.

The root cause: two defaults that disagree#

Nothing here is broken on its own. Two reasonable defaults simply don’t match.

So swtpm runs as tss and tries to write to a directory owned by swtpm. The upstream swtpm and Ubuntu bugs with this exact error message are all closed, because the packaging is right. The mismatch only appears once Kolla swaps in its own qemu.conf.

The fix: one layer on the upstream image#

If libvirt is going to run swtpm as tss, make the directory belong to tss. That means changing the image, and I rejected two other options first:

• Fix it by hand in the running container. It works until the next deploy recreates the container.
• Bind-mount a host directory. Kolla supports extra volumes, but something outside Kolla would have to create the directory with the right owner, and I don’t want a second config tool on Kolla-managed nodes.

No kolla-build either. I already ship a custom cinder-volume image for the TrueNAS driver, and the pattern there is a plain Dockerfile on top of the published Kolla image. The nova-libvirt version is five lines:

1
2
3
4
5
6
7
8
9

ARG KOLLA_BASE=quay.io/openstack.kolla/nova-libvirt:2025.1-ubuntu-noble
FROM ${KOLLA_BASE}

USER root

RUN set -eux; \
    mkdir -p /var/lib/swtpm-localca; \
    chown -R tss:tss /var/lib/swtpm-localca; \
    su -s /bin/sh tss -c 'test -w /var/lib/swtpm-localca'

The su line proves tss can actually write to the directory, so a future base image that changes the user or the package layout fails the build, not a VM boot. I checked the base image’s user before writing this rather than assuming: nova-libvirt’s config has no User at all, so it runs as root. That’s different from cinder-volume, which sets USER cinder and needs switching back.

It builds in a GitHub Action: pull requests build without pushing, and a vX.Y.Z tag publishes to GHCR and writes a release containing the line to paste. Kolla picks it up from one pin:

1
2

# globals.d/02-compute.yml
nova_libvirt_image_full: "ghcr.io/atlasdigitalengineering/nova-libvirt-swtpm:1.0.0"

It goes in globals.d, not globals.yml, because kolla-ansible passes the globals.d files after globals.yml and the last one wins. Then kolla-ansible deploy -t nova. It has to be deploy: reconfigure doesn’t recreate containers from a new image.

All three blades came back clean, and the test guest finally had a TPM:

1
2
3
4
5

ubuntu@test:~$ ls -la /dev/tpm*
crw-rw---- 1 tss root  10,   224 Nov  3 14:46 /dev/tpm0
crw-rw---- 1 tss tss  252, 65536 Nov  3 14:46 /dev/tpmrm0
ubuntu@test:~$ sudo tpm2_getrandom 8 --hex
da8db90f7782a9b9

The image and its write-up are at AtlasDigitalEngineering/nova-libvirt-swtpm. It should be retired if the kolla-ansible release build for Ubuntu 26.04 fixes the ability to read a custom qemu.conf.j2, at that point the fix becomes two lines of config. At the time of writing Kolla-Ansible does not support Ubuntu server 26.04 in its official support matrix yet.

What I’d tell myself before starting#

1. “Exhausted all hosts” means the build failed, not the scheduling. Go straight to the compute log of the first host in server event show.
2. Logs from containerised services live inside the container. If a path in an error doesn’t exist on the host, docker exec before concluding anything.
3. Find out who actually makes the decision. Nova’s swtpm_user looked like the switch, but libvirt chooses the user and Nova only has to agree. Changing the setting that agrees doesn’t change the outcome.
4. Read the version you run. One grep against the installed kolla-ansible would have saved two rebuilds. Docs and master branches could describe a different system from yours.
5. When config can’t fix it, the smallest durable change wins. A one-layer image on the upstream base, tested at build time, released by tag and pinned by version, is easier to reason about and to delete later than a hand-patched container or a second config tool on the nodes.