Proxmox to OpenStack without copying a byte: writing a TrueNAS Cinder driver

Between 26 August and 10 September 2026 I moved AtlasDigitalEngineering’s production Proxmox estate, running on a Dell PowerEdge M1000e blade chassis, onto OpenStack 2025.1 deployed with Kolla-Ansible. Three converged nodes, full control-plane quorum, production workloads, and not one byte of VM disk copied to get there. The thing that made that possible was a Cinder driver I had to write first. This post is the driver, the migration, and the parts that broke.

Why move at all#

Four reasons, in the order they mattered.

Reliability. Proxmox was unstable on the big dual-socket blades: 2x Xeon E5-2670 v2, 512 GB per node. Chasing that was expensive and living with it was worse.

Scale. The chassis holds sixteen blades and three were in use. OpenStack’s scheduler model maps onto “fill the chassis over time” far better than a Proxmox cluster does.

Skill alignment. Atlas Digital Engineering contracts DevSecOps services. OpenStack, Ansible, Terraform/Tofu, cloud-init, Heat and Magnum along with other DevOps tooling are what clients actually run. This chassis holds our internal production environment but also serves as the testbed for client work, and a platform that rehearses real client work is worth more than one that doesn’t.

Storage continuity. Every VM disk lived on TrueNAS Scale as a ZFS zvol over iSCSI, and the obvious OpenStack answer, Ceph, is off the table. Ceph wants raw block devices for its OSDs. ZFS is a complete storage stack that owns its own allocation. There is no supported way to layer one on the other.

That last point became the binding constraint on the whole design.

The constraint that forced a driver#

Terabytes of live VM disks, one appliance holding the only copy of them, and no working backup path going in. Proxmox Backup Server existed but hadn’t functioned properly due to the instability for some time, which was part of why the migration was happening.

A copy-based migration meant either doubling the storage footprint or a very long per-VM outage. Neither was acceptable. What I needed was for an existing zvol to become a Cinder volume where it sat. Cinder has an interface for exactly that, the manage_existing family, but no existing TrueNAS driver implemented it the way I needed. So the driver came first.

iXsystems Cinder Driver - This was the first path i tried however the driver has not been updated in 3 years and has never come out of experimental status it does however impliment the basic api required. About everything else i found investigating this migration was very old and designed for Core and much older versions of openstack.

The driver#

TrueNAS Cinder Driver is an OpenStack Cinder volume driver for TrueNAS Scale over iSCSI. Each Cinder volume is a zvol; the iSCSI extent and target binding are built and torn down around each attach.

Shape#

Two modules, deliberately separable:

• api_client.py, a REST wrapper over the TrueNAS API. Knows nothing about Cinder, depends only on requests, raises a TrueNASAPIError subclass for every failure. Testable without OpenStack installed.
• driver.py, the Cinder driver, subclassing SanISCSIDriver. Translates Cinder’s lifecycle into client calls and client errors into Cinder exceptions.

That split is why the test suites are split too. The unit suite runs on requests alone. The driver suite needs Cinder, which pulls in about 58 packages. A third functional suite talks to a real appliance and skips entirely unless one is configured, so it can never run by accident against the wrong box. About 4,600 lines of Python across the four modules, 166 commits, 1.0.0 on 3 September and 1.1.0 on 6 September.

What it does#

The full volume lifecycle: create, delete, attach, detach, extend, snapshot, clone, clone from snapshot. Multipath, with one target bound to every configured portal address. And the part it exists for:

• get_manageable_volumes enumerates zvols the appliance holds that Cinder doesn’t manage, each with a safety verdict and the reason if it’s unsafe.
• manage_existing_get_size reads the real zvol size.
• manage_existing adopts a zvol by renaming it into Cinder’s namespace.

A rename is a ZFS metadata operation. Adoption of a 500 GB disk costs the same as a 1 GB disk, and the zvol’s creation timestamp survives it unchanged. The safety gate refuses a zvol that holds a live iSCSI session and tells you exactly what to remove.

Every design-doc assumption was wrong at least once#

The driver started from a written implementation spec. Trusting it produced real defects: /zfs/zvol was not an endpoint, volmode: GEOM is FreeBSD-only, name__startswith is not a valid filter operator, and the EULA endpoint returns a bare boolean. The project’s planning doc now says it plainly: no document outranks observed behaviour. Every endpoint the driver depends on has been exercised against a real appliance, and the functional suite exists so findings can be re-checked rather than taken on trust.

That rule had a cost. The production appliance holds the only copy of every VM disk, so driver development could not touch it. That made a separate development OpenStack a hard dependency of the driver, and the dependency is a loop: the driver’s first milestone is a Nova instance booting from a TrueNAS-backed volume, which needs an OpenStack to demonstrate it in. The development all-in-one on blade slot 3 was that OpenStack and a small vm with Truenas served as the development appliance.

One limitation worth knowing about#

The driver inherits SUPPORTS_ACTIVE_ACTIVE = False, and Cinder’s volume manager raises rather than warns for a clustered non-active/active driver. So the cluster runs a single cinder-volume instance pinned to one node. That is a correctness constraint, not a misconfiguration, and it is the principal remaining single point of failure in the storage control path.

The target platform#

Kolla-Ansible, chosen over DevStack (not production), MicroStack and Sunbeam (don’t scale to sixteen blades), OpenStack-Ansible (LXC-based, which is what I was leaving), TripleO (being sunset) and OpenStack-Helm (needs Kubernetes for no gain).

Three converged nodes, each running control plane, network, compute and storage roles. At this scale a split topology idles hardware for no resilience gain; three converged nodes tolerate one node loss for everything, which a 2+1 split would not.

Each node: RAID1 across two 1.2 TB SAS disks, ext4 throughout because XFS can never be shrunk and the LV sizing was a guess, 300 GB for Docker, 250 GB for local instance disks, and roughly 446 GB left deliberately unallocated for lvextend and LVM snapshots before risky upgrades. The internal SD module was excluded on purpose: both mirrored cards wear identically under a constantly writing Linux root and likely fail together.

Hostnames are immutable after deployment. Kolla keys the Nova service record, the OVN chassis identity and Cinder’s backend host string on the OS hostname. Rename a node later and you orphan all three.

Five network planes under a running estate#

The network was rebuilt beneath production VMs that kept running. Five planes: OpenStack management on VLAN 40, storage on VLAN 1, a Geneve underlay on VLAN 209, a flat provider network for floating IPs on VLAN 110, and the API VIP on its own VLAN 50. Host addressing is slot number times ten on every plane, so slot 3 is .30 everywhere, which removes an entire class of mistake when reading logs.

Two decisions there were learned the hard way. The storage and tunnel planes never leave the chassis: no SVI, no gateway, not trunked upstream, jumbo frames stay inside the stack. And the API VIP needs its own VLAN, because the floating-IP VLAN’s interface is handed to OVS as br-ex and a keepalived address on it is deaf.

Tenant networking is ML2/OVN. Each project gets three tiered networks, an OVN router with SNAT egress by default, floating IPs as opt-in for inbound only, and per-tier security groups using remote groups so the rules follow membership rather than addresses. That is materially better isolation than the flat VLAN bridging it replaced.

The migration#

Phase 1 consolidated everything worth keeping onto one blade. Phase 2 rebuilt the other two to the node spec and deployed OpenStack as a two-node control plane, below quorum, for a deliberately short window. Phase 3 moved the workloads. Phase 4 drained, wiped and rebuilt the last blade and joined it to reach a three-member Galera primary component and three RabbitMQ nodes.

Each VM followed the same sequence:

1. Capture qm config <vmid> on Proxmox first: disk bus, NIC count, static addressing, firmware. Some of this is awkward to recover afterwards.
2. Shut the VM down.
3. Remove the zvol’s iSCSI extent on TrueNAS. This is what drops the session and makes the zvol adoptable. Shutting the VM down is not sufficient.
4. Confirm the zvol reports safe to manage.
5. cinder manage, using the reference from manageable-list verbatim, because zvol names don’t reliably match VM names.
6. Set hw_disk_bus and hw_scsi_model on the volume before first boot.
7. Create the instance, with a pinned port if the guest holds a static address.

Removing the extent is the rollback boundary. Before it, the VM boots on Proxmox unchanged. After cinder manage, rollback means unmanage, rename back, recreate the extent.

Two practical notes that cost time. openstack volume manage does not exist; the manage family only lives in python-cinderclient. And the cinder shell cannot authenticate with application credentials, because it always requests a scoped token and application credentials forbid it. Use a username and password for the migration, or drive the API with curl.

Cutting scope was the right call#

The first adoption showed how much hand-work an imported VM needs: no cloud-init, stale netplan, services bound to old addresses. The queue went from 27 candidate disks to 10. Everything else gets rebuilt fresh from a Glance image, which is lower risk, not higher. A rebuilt instance has correct image metadata, working cloud-init, a DHCP address and no rollback window.

Ten disks, about 1.96 TB, adopted in place.

What went wrong#

Two outages and a run of latent defects. These are the valuable part.

RSTP cut two management planes for 11 minutes. Single-instance RSTP is VLAN-blind. It saw three port-channels to what it considered one downstream place and blocked two of them. A loop-protection mechanism that doesn’t understand your segmentation will protect you from a loop you don’t have by breaking connectivity you do.

Tagging a VLAN onto plain switchports cut live iSCSI. On Dell FTOS, tagging a plain switchport into a second VLAN silently removes it from its default VLAN. Four storage ports dropped out of the storage VLAN; the blade carrying all of production lost its iSCSI path for about five minutes, 320 I/O errors, 24 VMs rebooted. No data lost, zvols intact. Only portmode hybrid ports may carry a second tagged VLAN now, and making a port hybrid bounces it, so that change only happens on a drained blade.

The adopted Windows VM bugchecked INACCESSIBLE_BOOT_DEVICE. A volume adopted via cinder manage has no image metadata, so Nova falls back to virtio-blk. Proxmox’s default is VirtIO SCSI. Every adopted disk changes bus underneath its guest. Windows only enables the storage driver its boot device used. Linux with UUIDs in fstab doesn’t care; Linux with /dev/sdaX in fstab drops to an initramfs prompt. Set the bus to match on every adopted volume, regardless of guest, before first boot. Reaching a bugcheck proved the adoption itself was correct: the bootloader and kernel loaded fine.

A config file the orchestrator had never read. Tenant MTU was capped at 1442 despite a 9000 underlay. A custom neutron.conf setting global_physnet_mtu sat in a subdirectory Kolla’s neutron role doesn’t merge from. The ML2 file in the same subdirectory is merged, which is why one applied and the other didn’t. A misplaced config file is silently ignored, not an error.

A health guard checking the wrong thing. Horizon failed on half of page loads for two days. One node’s Django Compressor manifest was stale, and Kolla only regenerates it if the settings hash changed. The manifest’s own validity is never checked, so restarting the container, the obvious remedy, did nothing.

A collision with a network the vendor creates on its own. 10.255.0.0/16 was about to be reserved for tenants. 10.255.253.0/24 is the UniFi controller’s hidden inter-VLAN transit, generated by the controller and unrenumberable. Tenant space became 10.255.0.0/17. Caught by reading before building.

The DNS circular dependency. The migrated DNS Servers now runs on the cluster whose blades depend on it for DNS. Control plane returned 504s and 500s while workstations were fine. Same species as running a CA inside the cloud it secures. At least one resolver now has to live off-cluster.

And the rest: HAProxy’s map_dom matches domain suffixes, so a dashboard at the parent name swallowed every service beneath it. Tenant VMs can’t reach the chassis-internal storage VLAN, so NFS media libraries broke until the appliance got a leg on the provider VLAN. Port-forwards to tenant addresses are silently dead; forward to the floating IP instead. SNMP polling and traps travel in opposite directions and need opposite firewall rules. Certificate renewal on an always-on host quietly made that host the only machine Kolla can deploy from.

Final state#

If you’re doing this yourself#

The driver’s docs are written for exactly this: a configuration.md with every option and the prerequisites the driver checks at startup, a migration.md that walks the adoption one disk at a time, and a troubleshooting.md that starts from symptoms the driver has actually produced. Start there, and start with a disk you can afford to lose.

Update Since the Migration#

iXsystems on July 15 created a new repo which appears to be developing a new modern Cinder driver of their own TrueNas truenas_cinder. Its still relativly new and i have not tested it but it may be a suitable replacement for maintianing an in house driver as it matures. I would also love to see them impliment our solution (or somthing like it) around volume management.