Commit 27722a5
storage: KVM - enable RBD/Ceph volume encryption support (#13556)
* storage: enable RBD/Ceph volume encryption support (shared base)
Flip StoragePoolType.RBD from EncryptionSupport.Unsupported to Hypervisor so the
existing encryption control plane (allocator, endpoint selector, offerings) treats
RBD pools as encryption-capable.
The agent-side encrypted RBD create path is not implemented yet; it is delivered by
two follow-up tracks (qemu-native engine='qemu' and ceph-native engine='librbd').
Until then, fail closed at the two RBD create chokepoints in LibvirtStorageAdaptor
(createPhysicalDisk and createDiskFromTemplate) when a passphrase is present, so we
never silently produce a plaintext volume that the control plane believes is encrypted.
No change for existing unencrypted RBD volumes (guards only fire when a passphrase
is set; supportsEncryption() only affects volumes that require encryption).
* kvm: Ceph-native LUKS2 encryption for RBD volumes (engine='librbd')
Implements encrypted RBD data and root disks using librbd's native LUKS2
encryption, decrypted at runtime by libvirt/qemu via <encryption
engine='librbd'>. CloudStack manages the passphrase (existing model).
- RbdEncryption: isolated helper wrapping `rbd encryption format luks2`,
cephx via --id + keyfile (secret not on the command line), LUKS passphrase
via KeyFile. Kept separate so the CLI can later be swapped for a JNA binding
(rados-java has no rbd_encryption_format API).
- LibvirtStorageAdaptor: create/clone the raw RBD image, then apply
`rbd encryption format luks2`; mark the disk LUKS2 so encrypt_format
propagates to the volume. Replaces the fail-closed guards.
- QemuObject.EncryptFormat: add LUKS2.
- LibvirtVMDef: render <encryption format='luks2' engine='librbd'>; the
encrypt details now carry an optional engine.
- attach (KVMStorageProcessor) and boot (LibvirtComputingResource): set
engine='librbd' for RBD-backed encrypted volumes.
NOTE: the CoW-clone-then-format path (encrypted root from an unencrypted
template) needs live-cluster validation for the parent-grow / usable-size
behaviour described in the Ceph image-encryption docs.
Builds: api + plugins/hypervisors/kvm (JDK11).
* kvm: gate host encryption probe on librbd support for RBD
hostSupportsVolumeEncryption() now advertises encryption capability if the
host supports EITHER qemu-native LUKS (qemu-img LUKS + cryptsetup) OR librbd
native encryption (rbd CLI with the encryption subcommand). Previously a
Ceph-only host that lacked cryptsetup would not advertise encryption even
though librbd can encrypt RBD volumes.
Split into hostSupportsQemuNativeVolumeEncryption() and
hostSupportsRbdVolumeEncryption(); kept HOST_VOLUME_ENCRYPTION as the single
host-wide flag (documented limitation: not per-pool).
* kvm: resize support for librbd-encrypted RBD volumes (#5)
Encrypted RBD volumes are encrypted natively by librbd and must be resized
with `rbd resize --encryption-passphrase-file` so librbd grows the encrypted
payload and keeps the LUKS header consistent. The existing encrypted-resize
path (resizeEncryptedQcowFile) uses qemu-img --object secret, which is for
qemu-native LUKS and does not fit the librbd LUKS2 layout.
- RbdEncryption.resize(): new `rbd resize` wrapper (cephx via --id + keyfile,
passphrase via KeyFile, optional --allow-shrink).
- LibvirtResizeVolumeCommandWrapper: detect encrypted RBD and route to the rbd
resize path, bypassing the libvirt v.resize and qemu-img paths.
Snapshot/revert, RBD<->RBD copy, and migration of encrypted RBD volumes need
no code changes: they operate on the raw (LUKS-containing) image at the block
level, and the destination passphrase secret is already created engine-agnostic
in LibvirtPrepareForMigrationCommandWrapper. These still require live validation.
Builds: plugins/hypervisors/kvm (JDK11).
* kvm: route online resize of encrypted RBD through virsh blockresize
For a running VM, an librbd-encrypted RBD volume must be resized in-band by
qemu/librbd, not out-of-band by the rbd CLI. Gate the CLI rbd-resize path on
!vmIsRunning so:
- offline -> `rbd resize --encryption-passphrase-file` (librbd-aware), and
- online -> existing NOTIFYONLY path -> virsh blockresize, where qemu's
block_resize delegates to librbd to grow the encrypted payload and notify
the guest in one step (no passphrase needed; qemu holds the secret).
This avoids notify-less out-of-band growth and qemu/librbd size divergence
while the image is open. Online behaviour still needs live validation that
blockresize resizes the encrypted payload for engine='librbd' disks.
* kvm: encrypted RBD root disks (thin CoW clone + full-copy fallback)
Root disks could not be encrypted: cloning a plaintext template and then
`rbd encryption format`ing the clone leaves the inherited OS data unreadable
(the LUKS header offsets it), so the guest could not mount root.
Fix, in createDiskFromTemplateOnRBD, with two paths:
- Option A (same-cluster cached RBD template): grow the template base to
reserve LUKS2 header space, snapshot+protect it (cloudstack-base-snap-luks),
clone from it, apply the LUKS2 header, resize the clone to the requested
size. Inherited template data stays readable through the clone's encryption
and the clone is a thin CoW image (only the header is written).
- Option B (first-use / non-RBD template): create an empty image, apply a
LUKS2 header, then import the template THROUGH the encryption layer via
RbdEncryption.importTemplate (qemu-img convert -n into encrypt.key-secret).
Correct but a full copy.
Validated end-to-end on Ubuntu 26.04 / libvirt 12.0.0: both boot; A is thin
(3.5 GiB provisioned, ~120 MiB used); LUKS2 verified at rest on Ceph.
* kvm: harden and align librbd-encrypted RBD volume code
Review pass over the librbd LUKS2 encryption feature to fix latent issues
and bring it in line with CloudStack conventions:
- RbdEncryption: reject empty/null passphrase with a clear error; round
rbd --size up to MiB so a non-aligned request never shrinks the volume
below what was asked for; create the temporary cephx conf/keyring 0600
explicitly instead of relying on the umask.
- LibvirtStorageAdaptor: close Rados/IoCTX/RbdImage in a finally block on
the encrypted-root paths (mirrors deleteVolume) so handles are not
leaked on exceptions; use parameterized log messages instead of string
concatenation; extract the encrypted-root Option A/B logic into
createEncryptedRootCoWClone / createEncryptedRootFullCopy.
- RbdEncryption: use an instance logger (matching the plugin convention)
and split argv construction into build{Format,Resize,Convert}Script so
the generated commands can be unit-tested.
* kvm: add RbdEncryption unit tests
Assert the rbd/qemu-img argv built for format, resize and
convert-through-encryption (RBD and file sources), and that empty/null
passphrases are rejected. Command construction is verified without a
live Ceph cluster.
* kvm: refuse encrypted RBD hot-plug on libvirt < 10.1.0
libvirt 10.0.0 has an object apply-order bug (fixed in 10.1.0) that breaks
hot-plug of an encrypted rbd blockdev: on attach the disk is opened before its
LUKS secret object is defined, so the attach fails with "No secret with id
'...-format-encryption-secret0'". Booting a VM from an encrypted RBD disk is
unaffected (the QEMU command line resolves all -object before -blockdev).
Refuse the attach up front with a clear error (mirroring the existing
openvswitch/io_uring libvirt-version gates) instead of letting libvirt fail
opaquely. Only the RBD hot-plug path is gated; boot/root/detach are untouched.
* docs: add PendingReleaseNotes entry for librbd-encrypted RBD volumes
* kvm: route the encrypted RBD template import through QemuImg
RbdEncryption built its own 'qemu-img convert' command line, which
duplicated qemu-img knowledge outside of QemuImg. QemuImg could only
write to a plain filename destination, so importing a template through
the librbd encryption layer was not expressible with it.
QemuImg now supports a destination described by image options
(--target-image-opts, with -n implied since such a target always exists
already), exposed as convertIntoExistingTarget(). QemuImageOptions can
render its parameters under either image-opts flag.
RbdEncryption.importTemplate now composes QemuImageOptions and a
QemuObject secret and delegates to QemuImg; its hand-built convert
script is removed. The rbd CLI calls (encryption format, resize,
support probe) stay, as qemu-img cannot perform them.
No functional change to the generated command.
* kvm: use readable variable names in the encrypted RBD root helpers
Review feedback: single-letter and abbreviated names are hard to read.
Renamed in the two methods added by this PR only (renaming the rest of
the class is out of scope here): r -> radosConnection, io -> ioContext,
rbd -> rbdClient, base -> templateImage, s -> snapshotInfo, encSnap ->
luksReservedSnapshotName, haveEncSnap -> luksSnapshotExists, createSize
-> imageSizeWithLuksHeader, srcIsRbd -> sourceIsRbdPool.
No functional change.
* server, kvm: report and require RBD volume encryption separately
Review feedback: distinguish the two volume encryption mechanisms
instead of advertising them under one host flag.
host.volume.encryption goes back to meaning qemu-native LUKS only
(qemu-img LUKS + cryptsetup), as it did before this PR, and hosts now
additionally report host.volume.encryption.rbd for librbd encryption
(rbd encryption format).
The deployment planner requires the flag matching the pool type of each
encrypted volume - librbd for volumes on RBD pools, qemu-native for any
other pool type - at all three places it validated encryption support
before. The pool is taken from the pools proposed alongside the host
when present, so first deployments are matched accurately too; an
encrypted volume with no pool yet accepts either mechanism and the
storage pool allocator picks a pool the host can serve.
This also stops a host whose librbd is too old for 'rbd encryption
format' from being selected for encrypted RBD volumes; it previously
advertised encryption through the qemu stack and the VM failed to
start.
---------
Co-authored-by: Václav Rozsypálek <vaclav.rozsypalek@master.cz>
Co-authored-by: calvix <7136358+calvix@users.noreply.github.com>1 parent 79189c8 commit 27722a5
16 files changed
Lines changed: 907 additions & 28 deletions
File tree
- api/src/main/java/com/cloud
- host
- storage
- plugins/hypervisors/kvm/src
- main/java
- com/cloud/hypervisor/kvm
- resource
- wrapper
- storage
- org/apache/cloudstack/utils
- qemu
- rbd
- test/java
- com/cloud/hypervisor/kvm/storage
- org/apache/cloudstack/utils/rbd
- server/src
- main/java/com/cloud/deploy
- test/java/com/cloud/deploy
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| 60 | + | |
60 | 61 | | |
61 | 62 | | |
62 | 63 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
172 | 172 | | |
173 | 173 | | |
174 | 174 | | |
175 | | - | |
| 175 | + | |
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
| |||
Lines changed: 17 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
92 | 93 | | |
93 | 94 | | |
94 | 95 | | |
| 96 | + | |
95 | 97 | | |
96 | 98 | | |
97 | 99 | | |
| |||
3898 | 3900 | | |
3899 | 3901 | | |
3900 | 3902 | | |
3901 | | - | |
| 3903 | + | |
| 3904 | + | |
| 3905 | + | |
3902 | 3906 | | |
3903 | 3907 | | |
3904 | 3908 | | |
| |||
4427 | 4431 | | |
4428 | 4432 | | |
4429 | 4433 | | |
| 4434 | + | |
4430 | 4435 | | |
4431 | 4436 | | |
4432 | 4437 | | |
| |||
6212 | 6217 | | |
6213 | 6218 | | |
6214 | 6219 | | |
6215 | | - | |
| 6220 | + | |
| 6221 | + | |
| 6222 | + | |
| 6223 | + | |
6216 | 6224 | | |
6217 | 6225 | | |
6218 | 6226 | | |
| |||
6237 | 6245 | | |
6238 | 6246 | | |
6239 | 6247 | | |
| 6248 | + | |
| 6249 | + | |
| 6250 | + | |
| 6251 | + | |
| 6252 | + | |
| 6253 | + | |
| 6254 | + | |
6240 | 6255 | | |
6241 | 6256 | | |
6242 | 6257 | | |
| |||
Lines changed: 12 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
788 | 788 | | |
789 | 789 | | |
790 | 790 | | |
| 791 | + | |
791 | 792 | | |
792 | 793 | | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
793 | 798 | | |
794 | 799 | | |
| 800 | + | |
795 | 801 | | |
796 | 802 | | |
797 | 803 | | |
798 | 804 | | |
| 805 | + | |
799 | 806 | | |
800 | 807 | | |
801 | 808 | | |
| |||
1446 | 1453 | | |
1447 | 1454 | | |
1448 | 1455 | | |
1449 | | - | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
1450 | 1461 | | |
1451 | 1462 | | |
1452 | 1463 | | |
| |||
Lines changed: 27 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| |||
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
96 | 104 | | |
97 | 105 | | |
98 | 106 | | |
| |||
115 | 123 | | |
116 | 124 | | |
117 | 125 | | |
118 | | - | |
| 126 | + | |
119 | 127 | | |
120 | 128 | | |
121 | 129 | | |
| |||
139 | 147 | | |
140 | 148 | | |
141 | 149 | | |
142 | | - | |
143 | | - | |
144 | | - | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
145 | 154 | | |
146 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
147 | 159 | | |
148 | 160 | | |
149 | 161 | | |
| |||
213 | 225 | | |
214 | 226 | | |
215 | 227 | | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
216 | 238 | | |
217 | 239 | | |
218 | 240 | | |
| |||
Lines changed: 26 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
182 | 182 | | |
183 | 183 | | |
184 | 184 | | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
185 | 191 | | |
186 | 192 | | |
187 | 193 | | |
| |||
1795 | 1801 | | |
1796 | 1802 | | |
1797 | 1803 | | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
| 1811 | + | |
| 1812 | + | |
| 1813 | + | |
| 1814 | + | |
| 1815 | + | |
| 1816 | + | |
| 1817 | + | |
1798 | 1818 | | |
1799 | 1819 | | |
1800 | 1820 | | |
| |||
1807 | 1827 | | |
1808 | 1828 | | |
1809 | 1829 | | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
1810 | 1833 | | |
1811 | | - | |
| 1834 | + | |
| 1835 | + | |
| 1836 | + | |
1812 | 1837 | | |
1813 | 1838 | | |
1814 | 1839 | | |
| |||
0 commit comments