Skip to content

Commit 27722a5

Browse files
calvixVáclav Rozsypálek
andauthored
storage: KVM - enable RBD/Ceph volume encryption support (#13556)
* storage: enable RBD/Ceph volume encryption support (shared base) Flip StoragePoolType.RBD from EncryptionSupport.Unsupported to Hypervisor so the existing encryption control plane (allocator, endpoint selector, offerings) treats RBD pools as encryption-capable. The agent-side encrypted RBD create path is not implemented yet; it is delivered by two follow-up tracks (qemu-native engine='qemu' and ceph-native engine='librbd'). Until then, fail closed at the two RBD create chokepoints in LibvirtStorageAdaptor (createPhysicalDisk and createDiskFromTemplate) when a passphrase is present, so we never silently produce a plaintext volume that the control plane believes is encrypted. No change for existing unencrypted RBD volumes (guards only fire when a passphrase is set; supportsEncryption() only affects volumes that require encryption). * kvm: Ceph-native LUKS2 encryption for RBD volumes (engine='librbd') Implements encrypted RBD data and root disks using librbd's native LUKS2 encryption, decrypted at runtime by libvirt/qemu via <encryption engine='librbd'>. CloudStack manages the passphrase (existing model). - RbdEncryption: isolated helper wrapping `rbd encryption format luks2`, cephx via --id + keyfile (secret not on the command line), LUKS passphrase via KeyFile. Kept separate so the CLI can later be swapped for a JNA binding (rados-java has no rbd_encryption_format API). - LibvirtStorageAdaptor: create/clone the raw RBD image, then apply `rbd encryption format luks2`; mark the disk LUKS2 so encrypt_format propagates to the volume. Replaces the fail-closed guards. - QemuObject.EncryptFormat: add LUKS2. - LibvirtVMDef: render <encryption format='luks2' engine='librbd'>; the encrypt details now carry an optional engine. - attach (KVMStorageProcessor) and boot (LibvirtComputingResource): set engine='librbd' for RBD-backed encrypted volumes. NOTE: the CoW-clone-then-format path (encrypted root from an unencrypted template) needs live-cluster validation for the parent-grow / usable-size behaviour described in the Ceph image-encryption docs. Builds: api + plugins/hypervisors/kvm (JDK11). * kvm: gate host encryption probe on librbd support for RBD hostSupportsVolumeEncryption() now advertises encryption capability if the host supports EITHER qemu-native LUKS (qemu-img LUKS + cryptsetup) OR librbd native encryption (rbd CLI with the encryption subcommand). Previously a Ceph-only host that lacked cryptsetup would not advertise encryption even though librbd can encrypt RBD volumes. Split into hostSupportsQemuNativeVolumeEncryption() and hostSupportsRbdVolumeEncryption(); kept HOST_VOLUME_ENCRYPTION as the single host-wide flag (documented limitation: not per-pool). * kvm: resize support for librbd-encrypted RBD volumes (#5) Encrypted RBD volumes are encrypted natively by librbd and must be resized with `rbd resize --encryption-passphrase-file` so librbd grows the encrypted payload and keeps the LUKS header consistent. The existing encrypted-resize path (resizeEncryptedQcowFile) uses qemu-img --object secret, which is for qemu-native LUKS and does not fit the librbd LUKS2 layout. - RbdEncryption.resize(): new `rbd resize` wrapper (cephx via --id + keyfile, passphrase via KeyFile, optional --allow-shrink). - LibvirtResizeVolumeCommandWrapper: detect encrypted RBD and route to the rbd resize path, bypassing the libvirt v.resize and qemu-img paths. Snapshot/revert, RBD<->RBD copy, and migration of encrypted RBD volumes need no code changes: they operate on the raw (LUKS-containing) image at the block level, and the destination passphrase secret is already created engine-agnostic in LibvirtPrepareForMigrationCommandWrapper. These still require live validation. Builds: plugins/hypervisors/kvm (JDK11). * kvm: route online resize of encrypted RBD through virsh blockresize For a running VM, an librbd-encrypted RBD volume must be resized in-band by qemu/librbd, not out-of-band by the rbd CLI. Gate the CLI rbd-resize path on !vmIsRunning so: - offline -> `rbd resize --encryption-passphrase-file` (librbd-aware), and - online -> existing NOTIFYONLY path -> virsh blockresize, where qemu's block_resize delegates to librbd to grow the encrypted payload and notify the guest in one step (no passphrase needed; qemu holds the secret). This avoids notify-less out-of-band growth and qemu/librbd size divergence while the image is open. Online behaviour still needs live validation that blockresize resizes the encrypted payload for engine='librbd' disks. * kvm: encrypted RBD root disks (thin CoW clone + full-copy fallback) Root disks could not be encrypted: cloning a plaintext template and then `rbd encryption format`ing the clone leaves the inherited OS data unreadable (the LUKS header offsets it), so the guest could not mount root. Fix, in createDiskFromTemplateOnRBD, with two paths: - Option A (same-cluster cached RBD template): grow the template base to reserve LUKS2 header space, snapshot+protect it (cloudstack-base-snap-luks), clone from it, apply the LUKS2 header, resize the clone to the requested size. Inherited template data stays readable through the clone's encryption and the clone is a thin CoW image (only the header is written). - Option B (first-use / non-RBD template): create an empty image, apply a LUKS2 header, then import the template THROUGH the encryption layer via RbdEncryption.importTemplate (qemu-img convert -n into encrypt.key-secret). Correct but a full copy. Validated end-to-end on Ubuntu 26.04 / libvirt 12.0.0: both boot; A is thin (3.5 GiB provisioned, ~120 MiB used); LUKS2 verified at rest on Ceph. * kvm: harden and align librbd-encrypted RBD volume code Review pass over the librbd LUKS2 encryption feature to fix latent issues and bring it in line with CloudStack conventions: - RbdEncryption: reject empty/null passphrase with a clear error; round rbd --size up to MiB so a non-aligned request never shrinks the volume below what was asked for; create the temporary cephx conf/keyring 0600 explicitly instead of relying on the umask. - LibvirtStorageAdaptor: close Rados/IoCTX/RbdImage in a finally block on the encrypted-root paths (mirrors deleteVolume) so handles are not leaked on exceptions; use parameterized log messages instead of string concatenation; extract the encrypted-root Option A/B logic into createEncryptedRootCoWClone / createEncryptedRootFullCopy. - RbdEncryption: use an instance logger (matching the plugin convention) and split argv construction into build{Format,Resize,Convert}Script so the generated commands can be unit-tested. * kvm: add RbdEncryption unit tests Assert the rbd/qemu-img argv built for format, resize and convert-through-encryption (RBD and file sources), and that empty/null passphrases are rejected. Command construction is verified without a live Ceph cluster. * kvm: refuse encrypted RBD hot-plug on libvirt < 10.1.0 libvirt 10.0.0 has an object apply-order bug (fixed in 10.1.0) that breaks hot-plug of an encrypted rbd blockdev: on attach the disk is opened before its LUKS secret object is defined, so the attach fails with "No secret with id '...-format-encryption-secret0'". Booting a VM from an encrypted RBD disk is unaffected (the QEMU command line resolves all -object before -blockdev). Refuse the attach up front with a clear error (mirroring the existing openvswitch/io_uring libvirt-version gates) instead of letting libvirt fail opaquely. Only the RBD hot-plug path is gated; boot/root/detach are untouched. * docs: add PendingReleaseNotes entry for librbd-encrypted RBD volumes * kvm: route the encrypted RBD template import through QemuImg RbdEncryption built its own 'qemu-img convert' command line, which duplicated qemu-img knowledge outside of QemuImg. QemuImg could only write to a plain filename destination, so importing a template through the librbd encryption layer was not expressible with it. QemuImg now supports a destination described by image options (--target-image-opts, with -n implied since such a target always exists already), exposed as convertIntoExistingTarget(). QemuImageOptions can render its parameters under either image-opts flag. RbdEncryption.importTemplate now composes QemuImageOptions and a QemuObject secret and delegates to QemuImg; its hand-built convert script is removed. The rbd CLI calls (encryption format, resize, support probe) stay, as qemu-img cannot perform them. No functional change to the generated command. * kvm: use readable variable names in the encrypted RBD root helpers Review feedback: single-letter and abbreviated names are hard to read. Renamed in the two methods added by this PR only (renaming the rest of the class is out of scope here): r -> radosConnection, io -> ioContext, rbd -> rbdClient, base -> templateImage, s -> snapshotInfo, encSnap -> luksReservedSnapshotName, haveEncSnap -> luksSnapshotExists, createSize -> imageSizeWithLuksHeader, srcIsRbd -> sourceIsRbdPool. No functional change. * server, kvm: report and require RBD volume encryption separately Review feedback: distinguish the two volume encryption mechanisms instead of advertising them under one host flag. host.volume.encryption goes back to meaning qemu-native LUKS only (qemu-img LUKS + cryptsetup), as it did before this PR, and hosts now additionally report host.volume.encryption.rbd for librbd encryption (rbd encryption format). The deployment planner requires the flag matching the pool type of each encrypted volume - librbd for volumes on RBD pools, qemu-native for any other pool type - at all three places it validated encryption support before. The pool is taken from the pools proposed alongside the host when present, so first deployments are matched accurately too; an encrypted volume with no pool yet accepts either mechanism and the storage pool allocator picks a pool the host can serve. This also stops a host whose librbd is too old for 'rbd encryption format' from being selected for encrypted RBD volumes; it previously advertised encryption through the qemu stack and the VM failed to start. --------- Co-authored-by: Václav Rozsypálek <vaclav.rozsypalek@master.cz> Co-authored-by: calvix <7136358+calvix@users.noreply.github.com>
1 parent 79189c8 commit 27722a5

16 files changed

Lines changed: 907 additions & 28 deletions

File tree

PendingReleaseNotes

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,3 +53,12 @@ example.ver.1 > example.ver.2:
5353
the guest while the NVRAM sidecar is copied, so that the captured firmware
5454
state is consistent with the disk snapshot. Non-UEFI VMs are unaffected and
5555
continue to snapshot live.
56+
57+
4.23.0.0 > 24.0.0:
58+
* KVM/Ceph: RBD volumes can now be encrypted at rest using native librbd
59+
LUKS2 (<encryption format='luks2' engine='librbd'>), for both data disks
60+
and root disks. Encryption is transparent to the guest and reuses the
61+
existing CloudStack volume-encryption passphrase handling, so no
62+
additional key store is required. Note: attaching an encrypted RBD volume
63+
to a running Instance requires libvirt >= 10.1.0; booting an Instance from
64+
an encrypted RBD root disk works on older libvirt.

api/src/main/java/com/cloud/host/Host.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ public static String[] toStrings(Host.Type... types) {
5757
String HOST_UEFI_ENABLE = "host.uefi.enable";
5858
String HOST_KVM_DISK_ONLY_VM_SNAPSHOT_NVRAM = "host.kvm.diskonlyvmsnapshot.nvram";
5959
String HOST_VOLUME_ENCRYPTION = "host.volume.encryption";
60+
String HOST_RBD_VOLUME_ENCRYPTION = "host.volume.encryption.rbd";
6061
String HOST_INSTANCE_CONVERSION = "host.instance.conversion";
6162
String HOST_VDDK_SUPPORT = "host.vddk.support";
6263
String HOST_VDDK_LIB_DIR = "vddk.lib.dir";

api/src/main/java/com/cloud/storage/Storage.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,7 @@ public static enum StoragePoolType {
172172
LVM(false, false, EncryptionSupport.Unsupported), // XenServer local LVM SR
173173
CLVM(true, false, EncryptionSupport.Unsupported),
174174
CLVM_NG(true, false, EncryptionSupport.Hypervisor),
175-
RBD(true, true, EncryptionSupport.Unsupported), // http://libvirt.org/storage.html#StorageBackendRBD
175+
RBD(true, true, EncryptionSupport.Hypervisor), // http://libvirt.org/storage.html#StorageBackendRBD ; encrypted natively by librbd (LUKS2, engine='librbd')
176176
SharedMountPoint(true, true, EncryptionSupport.Hypervisor),
177177
VMFS(true, true, EncryptionSupport.Unsupported), // VMware VMFS storage
178178
PreSetup(true, true, EncryptionSupport.Unsupported), // for XenServer, Storage Pool is set up by customers.

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import static com.cloud.host.Host.HOST_INSTANCE_CONVERSION;
2121
import static com.cloud.host.Host.HOST_KVM_DISK_ONLY_VM_SNAPSHOT_NVRAM;
2222
import static com.cloud.host.Host.HOST_OVFTOOL_VERSION;
23+
import static com.cloud.host.Host.HOST_RBD_VOLUME_ENCRYPTION;
2324
import static com.cloud.host.Host.HOST_VDDK_LIB_DIR;
2425
import static com.cloud.host.Host.HOST_VDDK_SUPPORT;
2526
import static com.cloud.host.Host.HOST_VDDK_VERSION;
@@ -92,6 +93,7 @@
9293
import org.apache.cloudstack.storage.volume.VolumeOnStorageTO;
9394
import org.apache.cloudstack.utils.bytescale.ByteScaleUtils;
9495
import org.apache.cloudstack.utils.cryptsetup.CryptSetup;
96+
import org.apache.cloudstack.utils.rbd.RbdEncryption;
9597
import org.apache.cloudstack.utils.hypervisor.HypervisorUtils;
9698
import org.apache.cloudstack.utils.linux.CPUStat;
9799
import org.apache.cloudstack.utils.linux.KVMHostInfo;
@@ -3898,7 +3900,9 @@ public int compare(final DiskTO arg0, final DiskTO arg1) {
38983900
if (volumeObjectTO.requiresEncryption() &&
38993901
pool.getType().encryptionSupportMode() == Storage.EncryptionSupport.Hypervisor ) {
39003902
String secretUuid = createLibvirtVolumeSecret(conn, volumeObjectTO.getPath(), volumeObjectTO.getPassphrase());
3901-
DiskDef.LibvirtDiskEncryptDetails encryptDetails = new DiskDef.LibvirtDiskEncryptDetails(secretUuid, QemuObject.EncryptFormat.enumValue(volumeObjectTO.getEncryptFormat()));
3903+
// RBD volumes are encrypted natively by librbd, so request the librbd encryption engine.
3904+
String encryptEngine = (pool.getType() == StoragePoolType.RBD) ? "librbd" : null;
3905+
DiskDef.LibvirtDiskEncryptDetails encryptDetails = new DiskDef.LibvirtDiskEncryptDetails(secretUuid, QemuObject.EncryptFormat.enumValue(volumeObjectTO.getEncryptFormat()), encryptEngine);
39023906
disk.setLibvirtDiskEncryptDetails(encryptDetails);
39033907
}
39043908
}
@@ -4427,6 +4431,7 @@ public StartupCommand[] initialize() {
44274431
cmd.setGatewayIpAddress(localGateway);
44284432
cmd.setIqn(getIqn());
44294433
cmd.getHostDetails().put(HOST_VOLUME_ENCRYPTION, String.valueOf(hostSupportsVolumeEncryption()));
4434+
cmd.getHostDetails().put(HOST_RBD_VOLUME_ENCRYPTION, String.valueOf(hostSupportsRbdVolumeEncryption()));
44304435
cmd.setHostTags(getHostTags());
44314436
boolean instanceConversionSupported = hostSupportsInstanceConversion();
44324437
cmd.getHostDetails().put(HOST_INSTANCE_CONVERSION, String.valueOf(instanceConversionSupported));
@@ -6212,7 +6217,10 @@ public boolean isHostSecured() {
62126217
}
62136218

62146219
/**
6215-
* Test host for volume encryption support
6220+
* Test host for qemu-native LUKS volume encryption (qemu-img LUKS support + cryptsetup),
6221+
* reported as {@code host.volume.encryption}. RBD/librbd encryption support is a separate
6222+
* capability, reported as {@code host.volume.encryption.rbd}
6223+
* (see {@link #hostSupportsRbdVolumeEncryption()}).
62166224
* @return boolean
62176225
*/
62186226
public boolean hostSupportsVolumeEncryption() {
@@ -6237,6 +6245,13 @@ public boolean hostSupportsVolumeEncryption() {
62376245
return true;
62386246
}
62396247

6248+
/**
6249+
* Test host for librbd native LUKS encryption support (rbd CLI with the encryption subcommand).
6250+
*/
6251+
public boolean hostSupportsRbdVolumeEncryption() {
6252+
return new RbdEncryption().isSupported();
6253+
}
6254+
62406255
public boolean isSecureMode(String bootMode) {
62416256
if (StringUtils.isNotBlank(bootMode) && "secure".equalsIgnoreCase(bootMode)) {
62426257
return true;

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -788,14 +788,21 @@ public static class DiskDef {
788788
public static class LibvirtDiskEncryptDetails {
789789
String passphraseUuid;
790790
QemuObject.EncryptFormat encryptFormat;
791+
String engine; // optional libvirt encryption engine (e.g. "librbd"); null => libvirt/qemu default
791792

792793
public LibvirtDiskEncryptDetails(String passphraseUuid, QemuObject.EncryptFormat encryptFormat) {
794+
this(passphraseUuid, encryptFormat, null);
795+
}
796+
797+
public LibvirtDiskEncryptDetails(String passphraseUuid, QemuObject.EncryptFormat encryptFormat, String engine) {
793798
this.passphraseUuid = passphraseUuid;
794799
this.encryptFormat = encryptFormat;
800+
this.engine = engine;
795801
}
796802

797803
public String getPassphraseUuid() { return this.passphraseUuid; }
798804
public QemuObject.EncryptFormat getEncryptFormat() { return this.encryptFormat; }
805+
public String getEngine() { return this.engine; }
799806
}
800807

801808
public static class DiskGeometry {
@@ -1446,7 +1453,11 @@ public String toString() {
14461453
}
14471454

14481455
if (encryptDetails != null) {
1449-
diskBuilder.append("<encryption format='" + encryptDetails.encryptFormat + "'>\n");
1456+
diskBuilder.append("<encryption format='" + encryptDetails.encryptFormat + "'");
1457+
if (encryptDetails.engine != null) {
1458+
diskBuilder.append(" engine='" + encryptDetails.engine + "'");
1459+
}
1460+
diskBuilder.append(">\n");
14501461
diskBuilder.append("<secret type='passphrase' uuid='" + encryptDetails.passphraseUuid + "' />\n");
14511462
diskBuilder.append("</encryption>\n");
14521463
}

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
import org.apache.cloudstack.utils.qemu.QemuImg.PhysicalDiskFormat;
3434
import org.apache.cloudstack.utils.qemu.QemuImgException;
3535
import org.apache.cloudstack.utils.qemu.QemuObject;
36+
import org.apache.cloudstack.utils.rbd.RbdEncryption;
3637
import org.libvirt.Connect;
3738
import org.libvirt.Domain;
3839
import org.libvirt.DomainInfo;
@@ -93,6 +94,13 @@ public Answer execute(final ResizeVolumeCommand command, final LibvirtComputingR
9394
final String path = vol.getPath();
9495
String type = notifyOnlyType;
9596

97+
// Encrypted RBD volumes are encrypted natively by librbd; they must be resized via
98+
// `rbd resize --encryption-passphrase-file` so librbd grows the encrypted payload and keeps
99+
// the LUKS header consistent. The libvirt/qemu-img resize paths below are for qemu-native
100+
// encryption and would not handle the librbd LUKS2 layout.
101+
final boolean rbdEncrypted = pool.getType() == StoragePoolType.RBD
102+
&& command.getPassphrase() != null && command.getPassphrase().length > 0;
103+
96104
if (spool.getType().equals(StoragePoolType.PowerFlex) && vol.getFormat().equals(PhysicalDiskFormat.QCOW2)) {
97105
// PowerFlex QCOW2 sizing needs to consider overhead.
98106
newSize = ScaleIOStorageAdaptor.getUsableBytesFromRawBytes(newSize);
@@ -115,7 +123,7 @@ public Answer execute(final ResizeVolumeCommand command, final LibvirtComputingR
115123
/* libvirt doesn't support resizing (C)LVM devices, and corrupts QCOW2 in some scenarios, so we have to do these via qemu-img */
116124
if (pool.getType() != StoragePoolType.CLVM && pool.getType() != StoragePoolType.CLVM_NG
117125
&& pool.getType() != StoragePoolType.Linstor && pool.getType() != StoragePoolType.PowerFlex
118-
&& vol.getFormat() != PhysicalDiskFormat.QCOW2) {
126+
&& vol.getFormat() != PhysicalDiskFormat.QCOW2 && !rbdEncrypted) {
119127
logger.debug("Volume " + path + " can be resized by libvirt. Asking libvirt to resize the volume.");
120128
try {
121129
final LibvirtUtilitiesHelper libvirtUtilitiesHelper = libvirtComputingResource.getLibvirtUtilitiesHelper();
@@ -139,11 +147,15 @@ public Answer execute(final ResizeVolumeCommand command, final LibvirtComputingR
139147

140148
boolean vmIsRunning = isVmRunning(vmInstanceName, libvirtComputingResource);
141149

142-
/* when VM is offline, we use qemu-img directly to resize encrypted volumes.
143-
If VM is online, the existing resize script will call virsh blockresize which works
144-
with both encrypted and non-encrypted volumes.
150+
/* when VM is offline, we use qemu-img (or rbd, for librbd-encrypted RBD) directly to resize
151+
encrypted volumes. If VM is online, the existing resize script calls virsh blockresize,
152+
which for an librbd-encrypted RBD disk lets qemu/librbd grow the encrypted payload and
153+
notify the guest in one step (no passphrase needed, qemu already has the secret loaded).
145154
*/
146-
if (!vmIsRunning && command.getPassphrase() != null && command.getPassphrase().length > 0 ) {
155+
if (rbdEncrypted && !vmIsRunning) {
156+
logger.debug("Invoking rbd to resize an offline, encrypted (librbd) RBD volume");
157+
resizeRbdEncryptedVolume(pool, vol, newSize, shrinkOk, command.getPassphrase());
158+
} else if (!vmIsRunning && command.getPassphrase() != null && command.getPassphrase().length > 0 ) {
147159
logger.debug("Invoking qemu-img to resize an offline, encrypted volume");
148160
QemuObject.EncryptFormat encryptFormat = QemuObject.EncryptFormat.enumValue(command.getEncryptFormat());
149161
resizeEncryptedQcowFile(vol, encryptFormat,newSize, command.getPassphrase(), libvirtComputingResource);
@@ -213,6 +225,16 @@ private void resizeEncryptedQcowFile(final KVMPhysicalDisk vol, final QemuObject
213225
}
214226
}
215227

228+
private void resizeRbdEncryptedVolume(final KVMStoragePool pool, final KVMPhysicalDisk vol, long newSize,
229+
boolean shrinkOk, byte[] passphrase) throws CloudRuntimeException {
230+
try {
231+
new RbdEncryption().resize(pool.getSourceHost(), pool.getSourcePort(), pool.getAuthUserName(),
232+
pool.getAuthSecret(), pool.getSourceDir(), vol.getName(), newSize, shrinkOk, passphrase);
233+
} finally {
234+
Arrays.fill(passphrase, (byte) 0);
235+
}
236+
}
237+
216238
private Answer handleMultipathSCSIResize(ResizeVolumeCommand command, KVMStoragePool pool) {
217239
((MultipathSCSIPool)pool).resize(command.getPath(), command.getInstanceName(), command.getNewSize());
218240
return new ResizeVolumeAnswer(command, true, "");

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,12 @@ public class KVMStorageProcessor implements StorageProcessor {
182182
private static final String CEPH_AUTH_KEY = "key";
183183
private static final String CEPH_CLIENT_MOUNT_TIMEOUT = "client_mount_timeout";
184184
private static final String CEPH_DEFAULT_MOUNT_TIMEOUT = "30";
185+
186+
// libvirt < 10.1.0 has an object apply-order bug (fixed in 10.1.0) that breaks hot-plug of an encrypted
187+
// blockdev: on attach the disk is opened before its LUKS secret object is defined, so the attach fails with
188+
// "No secret with id '...-format-encryption-secret0'". Booting a VM from an encrypted disk is unaffected (the
189+
// QEMU command line resolves all -object before -blockdev). See qemuBlockStorageSourceAttachApply() in libvirt.
190+
private static final long MIN_LIBVIRT_VERSION_FOR_RBD_ENCRYPTED_HOTPLUG = 10001000L; // libvirt 10.1.0
185191
/**
186192
* Time interval before rechecking virsh commands
187193
*/
@@ -1795,6 +1801,20 @@ protected DiskDef.DiskBus getAttachDiskBusType(int deviceId, List<DiskDef> disks
17951801
return DiskDef.DiskBus.VIRTIO;
17961802
}
17971803

1804+
/**
1805+
* libvirt &lt; 10.1.0 cannot hot-plug an encrypted rbd blockdev (the LUKS secret is applied after the disk is
1806+
* opened), so refuse the attach with a clear message rather than letting libvirt fail with an opaque
1807+
* "No secret with id ..." error. Only the RBD hot-plug path is affected; booting a VM from an encrypted RBD
1808+
* disk works on older libvirt, so this does not gate the boot/root path.
1809+
*/
1810+
protected void ensureLibvirtSupportsEncryptedRbdHotplug(StoragePoolType poolType) {
1811+
if (poolType == StoragePoolType.RBD
1812+
&& resource.getHypervisorLibvirtVersion() < MIN_LIBVIRT_VERSION_FOR_RBD_ENCRYPTED_HOTPLUG) {
1813+
throw new CloudRuntimeException("Libvirt version 10.1.0 required to attach an encrypted RBD volume to a running VM, but version "
1814+
+ resource.getHypervisorLibvirtVersion() + " detected. Booting a VM from an encrypted RBD disk is not affected.");
1815+
}
1816+
}
1817+
17981818
@Override
17991819
public Answer attachVolume(final AttachCommand cmd) {
18001820
final DiskTO disk = cmd.getDisk();
@@ -1807,8 +1827,13 @@ public Answer attachVolume(final AttachCommand cmd) {
18071827
final Connect conn = LibvirtConnection.getConnectionByVmName(vmName);
18081828
DiskDef.LibvirtDiskEncryptDetails encryptDetails = null;
18091829
if (vol.requiresEncryption()) {
1830+
// Encrypted RBD is decrypted by librbd inside qemu; hot-plugging it needs a libvirt new enough to
1831+
// emit the LUKS secret before the rbd blockdev. Booting from an encrypted RBD disk is unaffected.
1832+
ensureLibvirtSupportsEncryptedRbdHotplug(primaryStore.getPoolType());
18101833
String secretUuid = resource.createLibvirtVolumeSecret(conn, vol.getPath(), vol.getPassphrase());
1811-
encryptDetails = new DiskDef.LibvirtDiskEncryptDetails(secretUuid, QemuObject.EncryptFormat.enumValue(vol.getEncryptFormat()));
1834+
// RBD volumes are encrypted natively by librbd, so request the librbd encryption engine.
1835+
String encryptEngine = (primaryStore.getPoolType() == StoragePoolType.RBD) ? "librbd" : null;
1836+
encryptDetails = new DiskDef.LibvirtDiskEncryptDetails(secretUuid, QemuObject.EncryptFormat.enumValue(vol.getEncryptFormat()), encryptEngine);
18121837
vol.clearPassphrase();
18131838
}
18141839

0 commit comments

Comments
 (0)