Protect your data with MariaDB Server and Intel® TDX

Written by Written by Răzvan Vărzaru

In 2023, Intel launched Intel® Trust Domain Extensions (Intel® TDX), their latest Confidential Computing technology that ensures data in use protection for hardware-isolated virtual machines and verified hardware and software integrity through attestation.

In this blog post, we will show that MariaDB Server can run correctly on an Intel® TDX enabled VM and that the in-memory data is protected from outside actors, even from the host OS running the VM. By coupling Intel’s Confidential Computing technology with MariaDB Server build in data-at-rest and data-in-transit protection, customers can gain an end-to-end protected solution.

This advancement is crucial as the growing use of cloud services is inscreasing the need to protect sensitive data. Cloud providers now offer Confidential Computing services based on Intel® TDX. For more technology specific details, see Intel’s website.

Prologue

This year, we engaged in discussions with Intel representatives at SUSECON, leading us to collaborate on testing MariaDB Server with Intel® TDX. As part of this partnership we will publish several blog posts highlighting the combination of MariaDB Server and Intel® TDX.

This specific post will demonstrate how data stored in the server’s memory is protected from unauthorized access, even from attackers exploiting kernel vulnerabilities or rogue administrative users.

We will make use of Intel® TDX from two perspectives:

  • A DBA configuring MariaDB on their own VM to store sensitive banking information.
  • A third-party hosting provider.

Now, let’s keep the introduction brief and delve into the technical details!

Host Configuration

We’re working on a Ubuntu 24.04 OS installed on top of a system equipped with a CPU from the Xeon Scalable 5th Gen Emerald Rapids family.

~/tdx$ cat /etc/os-release PRETTY_NAME="Ubuntu 24.04 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04 LTS (Noble Numbat)" ~/tdx$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 45 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 224 On-line CPU(s) list: 0-223 Vendor ID: GenuineIntel Model name: INTEL(R) XEON(R) PLATINUM 8570 CPU family: 6 Model: 207 Thread(s) per core: 2 Core(s) per socket: 56 Socket(s): 2 Stepping: 2

We’re following Canonical’s instructions for setting up the host OS and guest OS. If you’re looking for a more detailed alternative, Intel offers excellent documentation on setting up Intel® TDX.

On the host side, the BIOS settings were already in place for us as described in Canonical’s instructions. If you’re setting up your own bare-metal instance, remember to enable TDX in the BIOS settings.

We just need to run the configuration script.

# Clone the Canonical's Intel® TDX repository git clone -b noble-24.04 https://github.com/canonical/tdx.git # Setup Intel® TDX in host OS cd tdx sudo ./setup-tdx-host.sh # Reboot sudo reboot # Verify that Intel® TDX is enabled $ sudo dmesg | grep -i tdx [ 5.340815] virt/tdx: BIOS enabled: private KeyID range [64, 128) ... ... [ 21.984224] virt/tdx: module initialized

Configure VMs

First, we need to generate the guest images that are later used by QEMU. Canonical provides a script to generate guest images with Intel® TDX tools inside, with the baseline image being an ubuntu-24.04 cloud image, which does not require any kernel changes.

# Creating an Intel® TDX qcow2 disk image cd tdx/guest-tools/image/ sudo ./create-td-image.sh

To later compare the VM protection provided by Intel® TDX with a regular VM, we also create a regular disk image, by simply removing Intel® TDX tooling from the image creation script.

We will end up with two disk images, under the repository tree.

:~/tdx/guest-tools/image$ ls -l *qcow* -rwxrwxrwx 1 root root 1903362048 Oct 17 23:30 regular-guest-ubuntu-24.04-generic.qcow2 -rwxrwxrwx 1 root root 4116250624 Oct 17 06:23 tdx-guest-ubuntu-24.04-generic.qcow2

In the Canonical TDX repository we can also find two sample domain XML’s for booting both an Intel® TDX protected VM (also called Trust Domain) and a regular VM. I’ve added hostfwd=tcp::0-:3306 to the XML’s to let QEMU assign a random port for the client-server connection.

The recommended way for booting both VM’s is using the libvirt wrapper tdvirsh provided in the repository.

./tdvirsh new \ --td-image ./image/regular-guest-ubuntu-24.04-generic.qcow2 \ --xml-template ./regular_vm.xml.template ./tdvirsh new \ --td-image ./image/tdx-guest-ubuntu-24.04-generic.qcow2 \ --xml-template ./trust_domain.xml.template

As a result both VM’s will start and both will be assigned random ports for SSH and MariaDB Server in the example below.

./tdvirsh list Id Name State --------------------------------------------------------------------------- 27 tdvirsh-trust_domain-ae41a94c-43f3-4bf0-9c30-b66a13ca6b42 running (ssh:38231 45017, cid:3) 30 tdvirsh-regular_vm-00c643ab-205d-41b5-873d-a60800b284f3 running (ssh:45545 38053, cid:4)

Using SSH to connect into the Intel® TDX protected VM, we can easily verify that Intel® TDX is enabled inside this VM.

root@tdx-guest:~# sudo dmesg | grep -i tdx [ 0.000000] tdx: Guest detected [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2024.02-3+tdx1.0 07/03/2024 [ 0.278473] process: using TDX aware idle routine [ 0.330149] Memory Encryption Features active: Intel TDX [ 6.454503] systemd[1]: Detected confidential virtualization tdx. [ 6.470694] systemd[1]: Hostname set to <tdx-guest>.

Configure MariaDB server

We will configure MariaDB Server 11.4.3 on both VM’s. Since 11.4.1 the connection between the client and the server has SSL enabled by default, which automatically brings us data-in transit protection. The server will generate certificates in memory and use them. For further reading we’ve blog posted about it.

Note that Intel® TDX does not inherently protect data that is written to disk, i.e. data-at-rest. There are two options to protect data in this state: (1) application-level file encryption and (2) guest OS-based file encryption. We opt for the first option as this is already provided by MariaDB Server.

In more details, we will set up the server with data-at-rest protection for the InnoDB storage engine using the file-key-management plugin.

Overall, we perform the following steps:

  • Install the server. You can download the latest MariaDB Server 11.4 from the downloads page.
  • Generate a keyfile for data-at-rest protection
  • Enable data-at-rest protection in the server configuration file
  • Bind to a public interface to allow remote connections
  • Restart the server to apply changes

Steps executed inside both VM’s after installing MariaDB Server:

# Keyfile generation root@tdx-guest:~# mkdir /etc/mysql/encryption root@tdx-guest:~# (echo -n "1;" ; openssl rand -hex 32 ) | sudo tee -a /etc/mysql/encryption/keyfile 1;56f9e484de0016b19c4de543430e729367838033db1d3ea724dc9624f1f3ae4b # Server configuration root@tdx-guest:~# cat <<EOL >> /etc/mysql/my.cnf [mariadb] plugin_load_add = file_key_management loose_file_key_management_filename = /etc/mysql/encryption/keyfile innodb_encrypt_log = ON innodb_encrypt_temporary_tables = ON innodb_encryption_threads = 4 bind-address = 0.0.0.0 EOL root@tdx-guest:~# systemctl restart mariadb

The last step is to create a user for the remote connection.

CREATE USER 'user12'@'%' IDENTIFIED BY 'password123'; GRANT ALL PRIVILEGES ON *.* TO 'user12'@'%' WITH GRANT OPTION;

Dataset preparation.

For our deployment, we will use the mariadb command-line client installed in the host OS to interact with the database server. In an actual production environment, the client would be installed on another machine.

The following steps will be repeated for both VM’s. First, let’s verify that MariaDB Server is configured to protect data in transit and at rest.

# Connect to the server mariadb -uuser12 -ppassword123 -h localhost -P #FW_PORT# # Verify that the connection is secured (data-in-transit protection) MariaDB [(none)]> SHOW STATUS LIKE 'Ssl_cipher'; +---------------+------------------------+ | Variable_name | Value | +---------------+------------------------+ | Ssl_cipher | TLS_AES_256_GCM_SHA384 | +---------------+------------------------+ # Check that the encryption plugin is ON (data-at-rest protection) MariaDB [(none)]> SHOW VARIABLES LIKE '%encryption%'; +------------------------------------------+---------+ | Variable_name | Value | +------------------------------------------+---------+ | file_key_management_encryption_algorithm | aes_cbc | | innodb_default_encryption_key_id | 1 | | innodb_encryption_rotate_key_age | 1 | | innodb_encryption_rotation_iops | 100 | | innodb_encryption_threads | 4 | +------------------------------------------+---------+ MariaDB [(none)]> SELECT PLUGIN_NAME, PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME LIKE "%key%"; +---------------------+---------------+ | PLUGIN_NAME | PLUGIN_STATUS | +---------------------+---------------+ | file_key_management | ACTIVE | +---------------------+---------------+

While setting up our database, we create encrypted tables to ensure data-at-rest protection and add some dummy data to the tables. We execute the following from the client:

# Prepare dataset CREATE DATABASE Customers; USE Customers; CREATE TABLE Individuals ( ID INT AUTO_INCREMENT PRIMARY KEY, Name VARCHAR(100) NOT NULL, Email VARCHAR(100), PhoneNumber VARCHAR(15) NOT NULL, Address VARCHAR(255) ) ENGINE=InnoDB, ENCRYPTED='YES'; INSERT INTO Individuals (Name, Email, PhoneNumber, Address) VALUES ('Razvan Varzaru', 'razvan@provider.com', '+40700060000', '456 Real Rd, Bucharest, Romania'); INSERT INTO Individuals (Name, Email, PhoneNumber, Address) VALUES ('John Doe', 'john.doe@provider.com', '+40700060001', '123 Fake St, Springfield, USA'); CREATE TABLE Accounts ( AccountID INT AUTO_INCREMENT PRIMARY KEY, CustomerID INT, AccountNumber VARCHAR(50) NOT NULL, Currency VARCHAR(3) NOT NULL, Balance DECIMAL(15, 2) NOT NULL, FOREIGN KEY (CustomerID) REFERENCES Individuals(ID) ) ENGINE=InnoDB, ENCRYPTED='YES'; INSERT INTO Accounts (CustomerID, AccountNumber, Currency, Balance) VALUES (1, 'ACC123456789', 'EUR', 1000.00); INSERT INTO Accounts (CustomerID, AccountNumber, Currency, Balance) VALUES (2, 'ACC444555666', 'GBP', 1500.00);

If we examine the data, we see that sensitive account information is present:

MariaDB [Customers]> SELECT * FROM Individuals; +----+----------------+-----------------------+--------------+---------------------------------+ | ID | Name | Email | PhoneNumber | Address | +----+----------------+-----------------------+--------------+---------------------------------+ | 1 | Razvan Varzaru | razvan@provider.com | +40700060000 | 456 Real Rd, Bucharest, Romania | | 2 | John Doe | john.doe@provider.com | +40700060001 | 123 Fake St, Springfield, USA | +----+----------------+-----------------------+--------------+---------------------------------+ MariaDB [Customers]> SELECT * FROM Accounts; +-----------+------------+---------------+----------+---------+ | AccountID | CustomerID | AccountNumber | Currency | Balance | +-----------+------------+---------------+----------+---------+ | 1 | 1 | ACC123456789 | EUR | 1000.00 | | 2 | 2 | ACC444555666 | GBP | 1500.00 | +-----------+------------+---------------+----------+---------+

Memory dump

Now it’s time to check out how this setup protects data-in-transit, data-at-rest and data-in-use. To show this, we perform a strong attack assuming a malicious administrator with full host OS privileges. Such an administrator can perform a memory dump of every running VM.

First, the regular VM without Intel® TDX protection:

# Memory dump of the regular domain :~/memdumps$ virsh dump tdvirsh-regular_vm-00c643ab-205d-41b5-873d-a60800b284f3 ./regular-dom-mem-dump.file --memory-only # Suppose ACC123 it's just a standard pattern for identifying the bank and the account type. :~/memdumps$ strings regular-dom-mem-dump.file | grep -aE -B 5 -A 5 'ACC123' Balance Razvan Varzaru razvan@provider.com +40700060000 456 Real Rd, Bucharest, Romania ACC123456789 1000.00i John Doe john.doe@provider.com +40700060001 123 Fake St, Springfield, USA -- AUE1 default-character-set=utf8mb3 default-collation=utf8mb3_general_ci infimum supremum ACC123456789EUR ACC444555666GBP I:6769510 E:ID_MM_CANDIDATE=1 G:systemd Q:systemd ... ... # and there's more output to it

With enough patience and further investigation into the results, we can discover that Razvan has the account number ACC123456789 and a balance of 1,000.00 EUR.

As we can see, an administrator can basically read all the sensitive data, even with data-at-rest and data-in-transit protection.

Now let’s try the same on the Intel® TDX protected VM.

# Memory dump of the TD :~/memdumps$ virsh dump tdvirsh-trust_domain-ae41a94c-43f3-4bf0-9c30-b66a13ca6b42 ./tdx-dom-mem-dump.file --memory-only :~/memdumps$ strings tdx-dom-mem-dump.file | grep -aE -B 5 -A 5 'ACC123' # Nothing! :~/memdumps$ strings tdx-dom-mem-dump.file | grep -aE -B 5 -A 5 'Razvan' :~/memdumps$ # Nothing x2!

Indeed, it works seamlessly. With Intel® TDX and a properly configured MariaDB Server installation, data protection is effectively ensured, at rest, in transit and in use. This setup ensures that no external actor can access or read the sensitive information stored within the Intel® TDX protected MariaDB Server.

Still, there are certain limits of the state we are in now:

  • we say that data is protected in use by doing a main memory dump but if you deploy the same workload at any external infrastructure provider, you normally do not have host OS access and thus cannot perform this test. Therefore, to address this, MariaDB Server can be combined with attestation to remotely verify the protection guarantees.
  • the key used for data-at-rest protection is created inside the TD. As a result, this requires a VM administrator that is allowed to login. Additionally, this key is written to a file which is not protected by Intel® TDX.

In conclusion, by leveraging the features of Intel® TDX, we can fully protect our data from unauthorized access and maintain the integrity of our information assets. We will explore in a future blog post how we can address the limits mentioned above.