From 00a90f35f946d212ef9374a60371986085c05aa5 Mon Sep 17 00:00:00 2001 From: Kailun Qin Date: Mon, 6 Jan 2025 23:50:51 -0500 Subject: [PATCH] [PAL,LibOS,common] Add file recovery support for encrypted files Previously, a fatal error during writes to encrypted files could cause file corruption due to incorrect GMACs and/or encryption keys. To address this, we introduce a file recovery mechanism using a "shadow" recovery file that stores data about to change and a `has_pending_write` flag in the metadata node indicating the start of a write transaction. During file flush, all cached blocks that are about to change are saved to the recovery file in the format of physical node numbers (offsets) plus encrypted block data. Before saving the main file contents, the `has_pending_write` flag is set in the file's metadata node and cleared only when the transaction is complete. If an encrypted file is opened and the `has_pending_write` flag is set, a recovery process starts to revert partial changes using the recovery file, returning to the last known good state. The "shadow" recovery file is cleaned up on file close. This commit adds a new mount parameter `enable_recovery = [true|false]` for encrypted files mounts to optionally enable this feature. We extend the file flush logic of protected files (pf) to include the recovery file dump and the setting/unsetting of the update flag. We make changes to the public pf APIs: the `pf_open()` API is extended to make the pf aware of the underlying recovery file managed by the LibOS, and recovery information (e.g., whether the pf needs recovery) is exposed back to the LibOS via a new `pf_get_recovery_info()` API. To facilitate the LibOS to initiate a file recovery process on file open, a new PAL API `PalRecoverEncryptedFile()` is introduced. Signed-off-by: Kailun Qin --- Documentation/devel/encfiles.rst | 17 +- .../encfiles/02_encfiles_representation.svg | 1074 ++++++++++++- .../img/encfiles/04_encfiles_write_less3k.svg | 1244 ++++++++++++++- .../img/encfiles/05_encfiles_read_less3k.svg | 1242 ++++++++++++++- .../encfiles/06_encfiles_write_greater3k.svg | 1357 ++++++++++++++++- .../encfiles/08_encfiles_read_greater3k.svg | 1321 +++++++++++++++- Documentation/manifest-syntax.rst | 8 +- Documentation/pal/host-abi.rst | 3 + common/src/protected_files/protected_files.c | 132 +- common/src/protected_files/protected_files.h | 32 +- .../protected_files/protected_files_format.h | 6 + .../protected_files_internal.h | 3 + libos/include/libos_fs.h | 6 + libos/include/libos_fs_encrypted.h | 28 +- libos/src/fs/chroot/encrypted.c | 4 +- libos/src/fs/libos_fs.c | 28 +- libos/src/fs/libos_fs_encrypted.c | 112 +- libos/test/fs/manifest.template | 2 +- pal/include/host/linux-common/linux_utils.h | 2 + pal/include/pal/pal.h | 12 + pal/include/pal_internal.h | 3 + pal/src/host/linux-common/file_utils.c | 68 + pal/src/host/linux-sgx/enclave_ocalls.c | 30 + pal/src/host/linux-sgx/enclave_ocalls.h | 2 + pal/src/host/linux-sgx/host_ocalls.c | 6 + pal/src/host/linux-sgx/pal_misc.c | 10 + pal/src/host/linux-sgx/pal_ocall_types.h | 7 + pal/src/host/linux/pal_misc.c | 9 + pal/src/host/skeleton/pal_misc.c | 8 + pal/src/pal_misc.c | 7 + pal/src/pal_symbols | 1 + python/graminelibos/manifest_check.py | 1 + tools/common/pf_util/pf_util.c | 4 +- 33 files changed, 6740 insertions(+), 49 deletions(-) diff --git a/Documentation/devel/encfiles.rst b/Documentation/devel/encfiles.rst index 2a4e9001d5..8206d00cb0 100644 --- a/Documentation/devel/encfiles.rst +++ b/Documentation/devel/encfiles.rst @@ -508,10 +508,19 @@ Additional details least one process writes to the file), the file may become corrupted or inaccessible to one of the processes. -- There is no support for file recovery. If the file was only partially written - to storage when the app abruptly terminated, Gramine will treat this file as - corrupted and will return an ``-EACCES`` error. (This is in contrast to Intel - SGX SDK which supports file recovery.) +- File recovery: Gramine supports recovery for encrypted files, which can be + enabled via the ``enable_recovery`` mount parameter in the Gramine manifest. + This allows a file to be recovered from a corrupted state (caused by e.g., + incorrect GMACs and/or encryption keys) when it was only partially written to + storage due to a fatal error (e.g., abrupt app termination). Similar to Intel + SGX SDK’s recovery mechanism, Gramine uses a "shadow" recovery file and a + ``has_pending_write`` flag in the metadata node to manage write transactions. + During file flush, cached blocks about to change are saved to the recovery + file. If an encrypted file is opened with the flag set, a recovery process + reverts partial changes using the recovery file, restoring the last known good + state. The "shadow" recovery file is cleaned up on file close. Note that + enabling this feature can impact performance due to additional writes to the + shadow file on each flush. - There is no key rotation scheme. The application must perform key rotation of the KDK by itself (by overwriting the ``/dev/attestation/keys/`` diff --git a/Documentation/img/encfiles/02_encfiles_representation.svg b/Documentation/img/encfiles/02_encfiles_representation.svg index e00651413e..7980ef0e63 100644 --- a/Documentation/img/encfiles/02_encfiles_representation.svg +++ b/Documentation/img/encfiles/02_encfiles_representation.svg @@ -1 +1,1073 @@ -#0: Metadata node (first 58 bytesin plaintext)host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Function“GRAFS_PF”0x010x00KDF nonceMAC07894157encrypted metadataover583941Constant padding39424095file_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)data_key1data_mac1…mht_key1mht_mac1…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071encrypted file dataArrows legend:Encrypted on storageCopied in/out of enclave869 \ No newline at end of file + +host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Functionfile_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)data_key1data_mac1…mht_key1mht_mac1…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071Arrows legend:Encrypted on storageCopied in/out of enclave#0: Metadata node (first 66 bytesin plaintext)“GRAFS_PF”0x010x00KDF nonceMACpending_write0789415765encrypted metadataover663949Constant padding39504095encrypted file data877 diff --git a/Documentation/img/encfiles/04_encfiles_write_less3k.svg b/Documentation/img/encfiles/04_encfiles_write_less3k.svg index 1e772be333..85da58e48f 100644 --- a/Documentation/img/encfiles/04_encfiles_write_less3k.svg +++ b/Documentation/img/encfiles/04_encfiles_write_less3k.svg @@ -1 +1,1243 @@ -#0: Metadata node (first 58 bytes in plaintext)host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Function“GRAFS_PF”0x010x00KDF nonceMAC07894157encrypted metadataover583941Constant padding39424095file_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)data_key1data_mac1…mht_key1mht_mac1…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071encrypted file dataArrows legend:Encrypted on storageCopied out of enclaveUnused in this caseCase of file size < 3KB:Encrypting new file②KDF nonce = rand()③derive key④encrypt⑤copy out869①file_data← write(buf)②④ \ No newline at end of file + +host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Functionfile_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)data_key1data_mac1…mht_key1mht_mac1…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071Arrows legend:Encrypted on storageCopied out of enclaveUnused in this caseCase of file size < 3KB:Encrypting new file②KDF nonce = rand()③derive key④encrypt⑤copy out①file_data← write(buf)#0: Metadata node (first 66 bytesin plaintext)“GRAFS_PF”0x010x00KDF nonceMACpending_write0789415765encrypted metadataover663949Constant padding39504095encrypted file data877④② diff --git a/Documentation/img/encfiles/05_encfiles_read_less3k.svg b/Documentation/img/encfiles/05_encfiles_read_less3k.svg index 9d87f8860c..b8e333c2d5 100644 --- a/Documentation/img/encfiles/05_encfiles_read_less3k.svg +++ b/Documentation/img/encfiles/05_encfiles_read_less3k.svg @@ -1 +1,1241 @@ -#0: Metadata node (first 58 bytes in plaintext)host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Function“GRAFS_PF”0x010x00KDF nonceMAC07894157encrypted metadataover583941Constant padding39424095file_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)data_key1data_mac1…mht_key1mht_mac1…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071encrypted file dataArrows legend:Decrypted from storageCopied into enclaveUnused in this caseCase of file size < 3KB:Decrypting existing file②KDF nonce = loaded③derive key④decrypt①copy in (on file open)869⑤read(buf) ← file_data②④ \ No newline at end of file + +host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Functionfile_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)data_key1data_mac1…mht_key1mht_mac1…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071Arrows legend:Decrypted from storageCopied into enclaveUnused in this caseCase of file size < 3KB:Decrypting existing file②KDF nonce = loaded③derive key④decrypt①copy in (on file open)⑤read(buf) ← file_data#0: Metadata node (first 66 bytesin plaintext)“GRAFS_PF”0x010x00KDF nonceMACpending_write0789415765encrypted metadataover663949Constant padding39504095encrypted file data877④② diff --git a/Documentation/img/encfiles/06_encfiles_write_greater3k.svg b/Documentation/img/encfiles/06_encfiles_write_greater3k.svg index dd8802f3f0..f56619171b 100644 --- a/Documentation/img/encfiles/06_encfiles_write_greater3k.svg +++ b/Documentation/img/encfiles/06_encfiles_write_greater3k.svg @@ -1 +1,1356 @@ -#0: Metadata node (first 58 bytes in plaintext)host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Function“GRAFS_PF”0x010x00KDF nonceMAC07894157encrypted metadataover583941Constant padding39424095file_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071encrypted file dataArrows legend:Encrypted on storageCopied out of enclaveUpdates MHT dataCase of file size > 3KB:Encrypting new file⑦KDF nonce = rand()⑧derive key⑨encrypt⑩copy out869①file_data← write(buf) …⑦⑨type = DATA_NODElogical_node= 0physical_node= 2encrypteddecrypted…Bouncebuffer#2: Data node (all encrypted)②… new.decrypted← write(buf)③key = rand()④encrypt⑤key = rand()⑥encrypt⑩copy out⑩copy out③④⑤⑥ \ No newline at end of file + +host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Functionfile_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071Arrows legend:Encrypted on storageCopied out of enclaveUpdates MHT dataCase of file size > 3KB:Encrypting new file⑦KDF nonce = rand()⑧derive key⑨encrypt⑩copy out①file_data← write(buf) …type = DATA_NODElogical_node= 0physical_node= 2encrypteddecrypted…Bouncebuffer#2: Data node (all encrypted)②… new.decrypted← write(buf)③key = rand()④encrypt⑤key = rand()⑥encrypt⑩copy out⑩copy out③④⑤⑥#0: Metadata node (first 66 bytesin plaintext)“GRAFS_PF”0x010x00KDF nonceMACpending_write0789415765encrypted metadataover663949Constant padding39504095encrypted file data877⑨⑦ diff --git a/Documentation/img/encfiles/08_encfiles_read_greater3k.svg b/Documentation/img/encfiles/08_encfiles_read_greater3k.svg index 696f2aeb68..5a40d875dd 100644 --- a/Documentation/img/encfiles/08_encfiles_read_greater3k.svg +++ b/Documentation/img/encfiles/08_encfiles_read_greater3k.svg @@ -1 +1,1320 @@ -#0: Metadata node (first 58 bytes in plaintext)host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Function“GRAFS_PF”0x010x00KDF nonceMAC07894157encrypted metadataover583941Constant padding39424095file_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071encrypted file dataArrows legend:Decrypted from storageCopied into enclaveCase of file size > 3KB:Decrypting existing file②KDF nonce = loaded③derive key④decrypt869②④type = DATA_NODElogical_node= 0physical_node= 2encrypteddecrypted…Bouncebuffer#2: Data node (all encrypted)⑧decrypt⑥decrypt⑤copy in⑦copy in①copy in (on file open)⑥⑥⑧⑧⑩… read(buf) ← node.decrypted⑨read(buf) ← file_data… \ No newline at end of file + +host_file_handlekdkmode = READ|WRITEmetadata_nodemetadata_decryptedroot_mht_nodeGlossary:•KDK = Key Derivation Key•MHT = Merkle Hash Tree•KDF = Key Derivation Functionfile_path[772]file_sizeroot_mht_node_keyroot_mht_node_macfile_data[3072]type = MHT_NODElogical_node= 0physical_node= 1encrypteddecrypted#1: Root MHT node (all fields encrypted)…data_key1data_mac1data_key96data_mac96…mht_key1mht_mac1mht_key32mht_mac32…0153130724095BouncebufferBounce bufferSGX enclave (trusted)Host storage (untrusted)3071Arrows legend:Decrypted from storageCopied into enclaveCase of file size > 3KB:Decrypting existing file②KDF nonce = loaded③derive key④decrypttype = DATA_NODElogical_node= 0physical_node= 2encrypteddecrypted…Bouncebuffer#2: Data node (all encrypted)⑧decrypt⑥decrypt⑤copy in⑦copy in①copy in (on file open)⑥⑥⑧⑧⑩… read(buf) ← node.decrypted⑨read(buf) ← file_data…#0: Metadata node (first 66 bytesin plaintext)“GRAFS_PF”0x010x00KDF nonceMACpending_write0789415765encrypted metadataover663949Constant padding39504095encrypted file data877④② diff --git a/Documentation/manifest-syntax.rst b/Documentation/manifest-syntax.rst index c41bf02332..f110520f21 100644 --- a/Documentation/manifest-syntax.rst +++ b/Documentation/manifest-syntax.rst @@ -1088,7 +1088,7 @@ Encrypted files :: fs.mounts = [ - { type = "encrypted", path = "[PATH]", uri = "[URI]", key_name = "[KEY_NAME]" }, + { type = "encrypted", path = "[PATH]", uri = "[URI]", key_name = "[KEY_NAME]", enable_recovery = [true|false] }, ] fs.insecure__keys.[KEY_NAME] = "[32-character hex value]" @@ -1154,6 +1154,12 @@ Gramine: in the application is insecure. If you need to derive encryption keys from such a "doubly-used" key, you must apply a KDF. +The ``enable_recovery`` mount parameter (default: ``false``) determines whether +file recovery is enabled for the mount. This feature allows selective enabling +or disabling of recovery for different mounted files or directories. Note that +enabling this feature can negatively impact performance, as it writes to a +second shadow file for later recovery purposes on each flush. + .. _untrusted-shared-memory: Untrusted shared memory diff --git a/Documentation/pal/host-abi.rst b/Documentation/pal/host-abi.rst index 13199586df..acd9315c67 100644 --- a/Documentation/pal/host-abi.rst +++ b/Documentation/pal/host-abi.rst @@ -372,3 +372,6 @@ random bits, to obtain an attestation report and quote, etc. .. doxygenfunction:: PalFreeThenLazyReallocCommittedPages :project: pal + +.. doxygenfunction:: PalRecoverEncryptedFile + :project: pal diff --git a/common/src/protected_files/protected_files.c b/common/src/protected_files/protected_files.c index 2fbac44c1f..3b0bf3d135 100644 --- a/common/src/protected_files/protected_files.c +++ b/common/src/protected_files/protected_files.c @@ -429,6 +429,84 @@ static bool ipf_update_metadata_node(pf_context_t* pf) { return true; } +static bool ipf_write_recovery_node(pf_context_t* pf, uint64_t physical_node_number, + const void* buffer, uint64_t offset) { + assert(pf->host_recovery_file_handle); + + recovery_node_t recovery_node = { .physical_node_number = physical_node_number }; + memcpy(recovery_node.bytes, buffer, sizeof(recovery_node.bytes)); + + pf_status_t status = g_cb_write(pf->host_recovery_file_handle, (void*)&recovery_node, offset, + sizeof(recovery_node)); + if (PF_FAILURE(status)) { + pf->last_error = status; + return false; + } + + return true; +} + +static bool ipf_write_recovery_file(pf_context_t* pf) { + assert(pf->host_recovery_file_handle); + + pf_status_t status = g_cb_truncate(pf->host_recovery_file_handle, 0); + if (PF_FAILURE(status)) { + pf->last_error = status; + return false; + } + + void* node; + uint64_t offset = 0; + for (node = lruc_get_first(pf->cache); node != NULL; node = lruc_get_next(pf->cache)) { + file_node_t* file_node = (file_node_t*)node; + if (!file_node->need_writing) + continue; + + if (!ipf_write_recovery_node(pf, file_node->physical_node_number, &file_node->encrypted, + offset)) + return false; + + offset += sizeof(recovery_node_t); + } + + if (!ipf_write_recovery_node(pf, /*physical_node_number=*/1, &pf->root_mht_node.encrypted, + offset)) + return false; + + offset += sizeof(recovery_node_t); + + if (!ipf_write_recovery_node(pf, /*physical_node_number=*/0, &pf->metadata_node, offset)) + return false; + + return true; +} + +static bool ipf_set_pending_write(pf_context_t* pf) { + pf->metadata_node.plaintext_part.has_pending_write = 1; + bool ret = ipf_write_node(pf, /*physical_node_number=*/0, &pf->metadata_node); + + /* Unset the `has_pending_write` in memory, which will be cleared on disk at the end of the + * flush when we write the metadata to disk. */ + pf->metadata_node.plaintext_part.has_pending_write = 0; + + return ret; +} + +static bool ipf_clear_pending_write(pf_context_t* pf) { + assert(pf->metadata_node.plaintext_part.has_pending_write == 0); + + if (!ipf_write_node(pf, /*physical_node_number=*/0, &pf->metadata_node)) + return false; + + pf_status_t status = g_cb_fsync(pf->host_file_handle); + if (PF_FAILURE(status)) { + pf->last_error = status; + return false; + } + + return true; +} + static bool ipf_internal_flush(pf_context_t* pf) { if (!pf->need_writing) { DEBUG_PF("no need to write"); @@ -436,11 +514,25 @@ static bool ipf_internal_flush(pf_context_t* pf) { } if (pf->metadata_decrypted.file_size > MD_USER_DATA_SIZE && pf->root_mht_node.need_writing) { + if (pf->host_recovery_file_handle) { + if (!ipf_write_recovery_file(pf)) { + pf->file_status = PF_STATUS_FLUSH_ERROR; + DEBUG_PF("failed to write changes to the recovery file"); + goto recoverable_error; + } + + if (!ipf_set_pending_write(pf)) { + pf->file_status = PF_STATUS_FLUSH_ERROR; + DEBUG_PF("failed to set the pending write flag"); + goto recoverable_error; + } + } + if (!ipf_update_all_data_and_mht_nodes(pf)) { // this is something that shouldn't happen, can't fix this... pf->file_status = PF_STATUS_CRYPTO_ERROR; DEBUG_PF("failed to update data and MHT nodes"); - return false; + goto unrecoverable_error; } } @@ -448,17 +540,23 @@ static bool ipf_internal_flush(pf_context_t* pf) { // this is something that shouldn't happen, can't fix this... pf->file_status = PF_STATUS_CRYPTO_ERROR; DEBUG_PF("failed to update metadata node"); - return false; + goto unrecoverable_error; } if (!ipf_write_all_changes_to_disk(pf)) { pf->file_status = PF_STATUS_WRITE_TO_DISK_FAILED; DEBUG_PF("failed to write changes to disk"); - return false; + goto recoverable_error; } pf->need_writing = false; return true; + +unrecoverable_error: + if (pf->host_recovery_file_handle) + (void)ipf_clear_pending_write(pf); +recoverable_error: + return false; } static file_node_t* ipf_get_mht_node(pf_context_t* pf, uint64_t offset) { @@ -751,6 +849,7 @@ static bool ipf_init_fields(pf_context_t* pf) { ipf_init_root_mht(&pf->root_mht_node); pf->host_file_handle = NULL; + pf->host_recovery_file_handle = NULL; pf->need_writing = false; pf->file_status = PF_STATUS_UNINITIALIZED; pf->last_error = PF_STATUS_SUCCESS; @@ -852,7 +951,8 @@ static void ipf_try_clear_error(pf_context_t* pf) { } static pf_context_t* ipf_open(const char* path, pf_file_mode_t mode, bool create, pf_handle_t file, - uint64_t real_size, const pf_key_t* kdk_key, pf_status_t* status) { + uint64_t real_size, const pf_key_t* kdk_key, + pf_handle_t recovery_file_handle, pf_status_t* status) { *status = PF_STATUS_NO_MEMORY; pf_context_t* pf = calloc(1, sizeof(*pf)); @@ -892,6 +992,8 @@ static pf_context_t* ipf_open(const char* path, pf_file_mode_t mode, bool create pf->host_file_handle = file; pf->mode = mode; + pf->host_recovery_file_handle = recovery_file_handle; + if (!create) { if (!ipf_init_existing_file(pf, path)) goto out; @@ -1126,12 +1228,14 @@ void pf_set_callbacks(pf_read_f read_f, pf_write_f write_f, pf_fsync_f fsync_f, } pf_status_t pf_open(pf_handle_t handle, const char* path, uint64_t underlying_size, - pf_file_mode_t mode, bool create, const pf_key_t* key, pf_context_t** context) { + pf_file_mode_t mode, bool create, const pf_key_t* key, + pf_handle_t recovery_file_handle, pf_context_t** context) { if (!g_initialized) return PF_STATUS_UNINITIALIZED; pf_status_t status; - *context = ipf_open(path, mode, create, handle, underlying_size, key, &status); + *context = ipf_open(path, mode, create, handle, underlying_size, key, recovery_file_handle, + &status); return status; } @@ -1297,3 +1401,19 @@ pf_status_t pf_flush(pf_context_t* pf) { return PF_STATUS_SUCCESS; } + +pf_status_t pf_get_recovery_info(pf_context_t* pf, bool* out_recovery_needed, + size_t* out_node_size) { + if (out_recovery_needed) { + // read metadata node + if (!ipf_read_node(pf, /*physical_node_number=*/0, (uint8_t*)&pf->metadata_node)) + return pf->last_error; + + *out_recovery_needed = (pf->metadata_node.plaintext_part.has_pending_write == 1); + } + + if (out_node_size) + *out_node_size = sizeof(((recovery_node_t*)0)->bytes); + + return PF_STATUS_SUCCESS; +} diff --git a/common/src/protected_files/protected_files.h b/common/src/protected_files/protected_files.h index 4bebe71a5c..18a845af27 100644 --- a/common/src/protected_files/protected_files.h +++ b/common/src/protected_files/protected_files.h @@ -212,19 +212,21 @@ const char* pf_strerror(int err); /*! * \brief Open a protected file. * - * \param handle Open underlying file handle. - * \param path Path to the file. If NULL and \p create is false, don't check path - * for validity. - * \param underlying_size Underlying file size. - * \param mode Access mode. - * \param create Overwrite file contents if true. - * \param key Wrap key. - * \param[out] context PF context for later calls. + * \param handle Open underlying file handle. + * \param path Path to the file. If NULL and \p create is false, don't check path + * for validity. + * \param underlying_size Underlying file size. + * \param mode Access mode. + * \param create Overwrite file contents if true. + * \param key Wrap key. + * \param recovery_file_handle (optional)Underlying recovery file handle. + * \param[out] context PF context for later calls. * * \returns PF status. */ pf_status_t pf_open(pf_handle_t handle, const char* path, uint64_t underlying_size, - pf_file_mode_t mode, bool create, const pf_key_t* key, pf_context_t** context); + pf_file_mode_t mode, bool create, const pf_key_t* key, + pf_handle_t recovery_file_handle, pf_context_t** context); /*! * \brief Close a protected file and commit all changes to disk. @@ -302,3 +304,15 @@ pf_status_t pf_rename(pf_context_t* pf, const char* new_path); * \returns PF status. */ pf_status_t pf_flush(pf_context_t* pf); + +/*! + * \brief Get the recovery info of a PF. + * + * \param pf PF context. + * \param[out] out_recovery_needed (optional) Whether recovery is needed for \p pf. + * \param[out] out_node_size (optional) Size of the \p pf node. + * + * \returns PF status. + */ +pf_status_t pf_get_recovery_info(pf_context_t* pf, bool* out_recovery_needed, + size_t* out_node_size); diff --git a/common/src/protected_files/protected_files_format.h b/common/src/protected_files/protected_files_format.h index 955351f561..8f250cca20 100644 --- a/common/src/protected_files/protected_files_format.h +++ b/common/src/protected_files/protected_files_format.h @@ -56,6 +56,7 @@ typedef struct { uint8_t minor_version; pf_nonce_t metadata_key_nonce; pf_mac_t metadata_mac; /* GCM mac */ + uint8_t has_pending_write; /* flag for file recovery */ } metadata_plaintext_t; typedef struct { @@ -95,6 +96,11 @@ typedef struct { } encrypted_node_t; static_assert(sizeof(encrypted_node_t) == PF_NODE_SIZE, "sizeof(encrypted_node_t)"); +typedef struct { + uint64_t physical_node_number; + uint8_t bytes[PF_NODE_SIZE]; +} recovery_node_t; + static_assert(sizeof(mht_node_t) == sizeof(data_node_t), "sizes of MHT and data nodes differ"); // Data struct that wraps the 4KB encrypted-node buffer (bounce buffer) and the corresponding 4KB diff --git a/common/src/protected_files/protected_files_internal.h b/common/src/protected_files/protected_files_internal.h index fe45794ca6..aaf808e3dc 100644 --- a/common/src/protected_files/protected_files_internal.h +++ b/common/src/protected_files/protected_files_internal.h @@ -17,6 +17,9 @@ struct pf_context { pf_file_mode_t mode; // read-only, write-only or read-write bool need_writing; // whether file was modified and thus needs writing to storage + pf_handle_t host_recovery_file_handle; // opaque recovery file handle (e.g. PAL handle) used by + // callbacks + pf_status_t file_status; // PF_STATUS_SUCCESS, PF_STATUS_CRYPTO_ERROR, etc. pf_status_t last_error; // FIXME: unclear why this is needed diff --git a/libos/include/libos_fs.h b/libos/include/libos_fs.h index c4d0ef4d4f..d6e3567903 100644 --- a/libos/include/libos_fs.h +++ b/libos/include/libos_fs.h @@ -62,6 +62,10 @@ struct libos_mount_params { /* Key name (used by `chroot_encrypted` filesystem), or NULL if not applicable */ const char* key_name; + + /* Whether to enable file recovery (used by `chroot_encrypted` filesystem), false if not + * applicable */ + bool enable_recovery; }; struct libos_fs_ops { @@ -532,6 +536,8 @@ struct libos_mount { void* data; + bool enable_recovery; + void* cpdata; size_t cpsize; diff --git a/libos/include/libos_fs_encrypted.h b/libos/include/libos_fs_encrypted.h index 9890058c51..de825dfb72 100644 --- a/libos/include/libos_fs_encrypted.h +++ b/libos/include/libos_fs_encrypted.h @@ -17,8 +17,11 @@ #include "libos_types.h" #include "list.h" #include "pal.h" +#include "perm.h" #include "protected_files.h" +#define RECOVERY_FILE_PERM_RW PERM_rw_rw_rw_ + /* * Represents a named key for opening files. The key might not be set yet: value of a key can be * specified in the manifest, or set using `update_encrypted_files_key`. Before the key is set, @@ -49,6 +52,11 @@ struct libos_encrypted_file { /* `pf` and `pal_handle` are non-null as long as `use_count` is greater than 0 */ pf_context_t* pf; PAL_HANDLE pal_handle; + + /* `recovery_file_pal_handle` is non-null as long as `enable_recovery` is true and `use_count` + * is greater than 0 */ + bool enable_recovery; + PAL_HANDLE recovery_file_pal_handle; }; /* @@ -109,31 +117,33 @@ void update_encrypted_files_key(struct libos_encrypted_files_key* key, const pf_ /* * \brief Open an existing encrypted file. * - * \param uri PAL URI to open, has to begin with "file:". - * \param key Key, has to be already set. - * \param[out] out_enc On success, set to a newly created `libos_encrypted_file` object. + * \param uri PAL URI to open, has to begin with "file:". + * \param key Key, has to be already set. + * \param enable_recovery Whether to enable file recovery. + * \param[out] out_enc On success, set to a newly created `libos_encrypted_file` object. * * `uri` has to correspond to an existing file that can be decrypted with `key`. * * The newly created `libos_encrypted_file` object will have `use_count` set to 1. */ int encrypted_file_open(const char* uri, struct libos_encrypted_files_key* key, - struct libos_encrypted_file** out_enc); + bool enable_recovery, struct libos_encrypted_file** out_enc); /* * \brief Create a new encrypted file. * - * \param uri PAL URI to open, has to begin with "file:". - * \param perm Permissions for the new file. - * \param key Key, has to be already set. - * \param[out] out_enc On success, set to a newly created `libos_encrypted_file` object. + * \param uri PAL URI to open, has to begin with "file:". + * \param perm Permissions for the new file. + * \param key Key, has to be already set. + * \param enable_recovery Whether to enable file recovery. + * \param[out] out_enc On success, set to a newly created `libos_encrypted_file` object. * * `uri` must not correspond to an existing file. * * The newly created `libos_encrypted_file` object will have `use_count` set to 1. */ int encrypted_file_create(const char* uri, mode_t perm, struct libos_encrypted_files_key* key, - struct libos_encrypted_file** out_enc); + bool enable_recovery, struct libos_encrypted_file** out_enc); /* * \brief Deallocate an encrypted file. diff --git a/libos/src/fs/chroot/encrypted.c b/libos/src/fs/chroot/encrypted.c index 1803a0bdf5..a3b6099233 100644 --- a/libos/src/fs/chroot/encrypted.c +++ b/libos/src/fs/chroot/encrypted.c @@ -154,7 +154,7 @@ static int chroot_encrypted_lookup(struct libos_dentry* dent) { file_off_t size; struct libos_encrypted_files_key* key = dent->mount->data; - ret = encrypted_file_open(uri, key, &enc); + ret = encrypted_file_open(uri, key, dent->mount->enable_recovery, &enc); if (ret < 0) { if (ret == -EACCES) { /* allow the inode to be created even if the underlying encrypted file is corrupted; @@ -233,7 +233,7 @@ static int chroot_encrypted_creat(struct libos_handle* hdl, struct libos_dentry* struct libos_encrypted_files_key* key = dent->mount->data; struct libos_encrypted_file* enc; - ret = encrypted_file_create(uri, HOST_PERM(perm), key, &enc); + ret = encrypted_file_create(uri, HOST_PERM(perm), key, dent->mount->enable_recovery, &enc); if (ret < 0) goto out; diff --git a/libos/src/fs/libos_fs.c b/libos/src/fs/libos_fs.c index 22022391df..e6ac0f168c 100644 --- a/libos/src/fs/libos_fs.c +++ b/libos/src/fs/libos_fs.c @@ -121,6 +121,7 @@ static int mount_root(void) { char* fs_root_type = NULL; char* fs_root_uri = NULL; char* fs_root_key_name = NULL; + bool fs_root_enable_recovery; assert(g_manifest_root); @@ -145,9 +146,18 @@ static int mount_root(void) { goto out; } + ret = toml_bool_in(g_manifest_root, "fs.root.enable_recovery", /*defaultval=*/false, + &fs_root_enable_recovery); + if (ret < 0) { + log_error("Cannot parse 'fs.root.enable_recovery'"); + ret = -EINVAL; + goto out; + } + struct libos_mount_params params = { .path = "/", .key_name = fs_root_key_name, + .enable_recovery = fs_root_enable_recovery, }; if (!fs_root_type && !fs_root_uri) { @@ -212,6 +222,7 @@ static int mount_one_nonroot(toml_table_t* mount, const char* prefix) { char* mount_path = NULL; char* mount_uri = NULL; char* mount_key_name = NULL; + bool mount_enable_recovery; ret = toml_string_in(mount, "type", &mount_type); if (ret < 0) { @@ -241,6 +252,13 @@ static int mount_one_nonroot(toml_table_t* mount, const char* prefix) { goto out; } + ret = toml_bool_in(mount, "enable_recovery", /*defaultval=*/false, &mount_enable_recovery); + if (ret < 0) { + log_error("Cannot parse '%s.enable_recovery'", prefix); + ret = -EINVAL; + goto out; + } + if (!mount_path) { log_error("No value provided for '%s.path'", prefix); ret = -EINVAL; @@ -286,6 +304,7 @@ static int mount_one_nonroot(toml_table_t* mount, const char* prefix) { .path = mount_path, .uri = mount_uri, .key_name = mount_key_name, + .enable_recovery = mount_enable_recovery, }; ret = mount_fs(¶ms); @@ -459,6 +478,8 @@ static int mount_fs_at_dentry(struct libos_mount_params* params, struct libos_de mount->fs = fs; mount->data = mount_data; + mount->enable_recovery = params->enable_recovery; + /* Attach mount to mountpoint, and the other way around */ mount->mount_point = mount_point; @@ -706,9 +727,10 @@ BEGIN_CP_FUNC(mount) { new_mount->cpdata = (char*)base + cp_off; } - new_mount->data = NULL; - new_mount->mount_point = NULL; - new_mount->root = NULL; + new_mount->data = NULL; + new_mount->mount_point = NULL; + new_mount->root = NULL; + new_mount->enable_recovery = mount->enable_recovery; INIT_LIST_HEAD(new_mount, list); refcount_set(&new_mount->ref_count, 0); diff --git a/libos/src/fs/libos_fs_encrypted.c b/libos/src/fs/libos_fs_encrypted.c index 91e8220bec..14bb41414d 100644 --- a/libos/src/fs/libos_fs_encrypted.c +++ b/libos/src/fs/libos_fs_encrypted.c @@ -165,6 +165,8 @@ static int encrypted_file_internal_open(struct libos_encrypted_file* enc, PAL_HA int ret; char* normpath = NULL; + PAL_HANDLE recovery_file_pal_handle = NULL; + bool maybe_recovery_needed = !create && !pal_handle; if (!pal_handle) { enum pal_create_mode create_mode = create ? PAL_CREATE_ALWAYS : PAL_CREATE_NEVER; @@ -174,7 +176,32 @@ static int encrypted_file_internal_open(struct libos_encrypted_file* enc, PAL_HA log_warning("PalStreamOpen failed: %s", pal_strerror(ret)); return pal_to_unix_errno(ret); } + + if (enc->enable_recovery) { + const char* recovery_file_suffix = "_gramine_recovery"; + size_t uri_len = strlen(enc->uri); + size_t suffix_len = strlen(recovery_file_suffix); + size_t recovery_file_uri_len = uri_len + suffix_len + 1; + char* recovery_file_uri = malloc(recovery_file_uri_len); + if (!recovery_file_uri) { + ret = -ENOMEM; + goto out; + } + memcpy(recovery_file_uri, enc->uri, uri_len); + memcpy(recovery_file_uri + uri_len, recovery_file_suffix, suffix_len); + recovery_file_uri[uri_len + suffix_len] = '\0'; + + ret = PalStreamOpen(recovery_file_uri, PAL_ACCESS_RDWR, RECOVERY_FILE_PERM_RW, + PAL_CREATE_TRY, /*options=*/0, &recovery_file_pal_handle); + free(recovery_file_uri); + if (ret < 0) { + log_warning("PalStreamOpen failed: %s", pal_strerror(ret)); + ret = pal_to_unix_errno(ret); + goto out; + } + } } + assert(enc->enable_recovery == (recovery_file_pal_handle != NULL)); PAL_STREAM_ATTR pal_attr; ret = PalStreamAttributesQueryByHandle(pal_handle, &pal_attr); @@ -209,7 +236,7 @@ static int encrypted_file_internal_open(struct libos_encrypted_file* enc, PAL_HA goto out; } pf_status_t pfs = pf_open(pal_handle, normpath, size, PF_FILE_MODE_READ | PF_FILE_MODE_WRITE, - create, &enc->key->pf_key, &pf); + create, &enc->key->pf_key, recovery_file_pal_handle, &pf); unlock(&g_keys_lock); if (PF_FAILURE(pfs)) { log_warning("pf_open failed: %s", pf_strerror(pfs)); @@ -217,13 +244,61 @@ static int encrypted_file_internal_open(struct libos_encrypted_file* enc, PAL_HA goto out; } + if (maybe_recovery_needed) { + bool recovery_needed; + size_t node_size; + pfs = pf_get_recovery_info(pf, &recovery_needed, &node_size); + if (PF_FAILURE(pfs)) { + log_warning("get file recovery info failed: %s", pf_strerror(pfs)); + ret = -EACCES; + goto out; + } + + if (recovery_needed) { + if (!enc->enable_recovery) { + log_warning("%s: file recovery needed but feature disabled; please consider " + "setting 'enable_recovery = true' for the mount", normpath); + ret = -EACCES; + goto out; + } + + log_debug("%s: starting file recovery", normpath); + + ret = PalRecoverEncryptedFile(pal_handle, recovery_file_pal_handle, node_size); + if (ret < 0) { + log_warning("PalRecoverEncryptedFile failed: %s", pal_strerror(ret)); + ret = pal_to_unix_errno(ret); + goto out; + } + + pfs = pf_get_recovery_info(pf, &recovery_needed, /*out_node_size=*/NULL); + if (PF_FAILURE(pfs)) { + log_warning("get file recovery info: %s", pf_strerror(pfs)); + ret = -EACCES; + goto out; + } + + if (recovery_needed) { + log_warning("%s: file recovery attempted but failed", normpath); + ret = -EACCES; + goto out; + } + + log_debug("%s: file recovery completed", normpath); + } + } + enc->pf = pf; enc->pal_handle = pal_handle; + enc->recovery_file_pal_handle = recovery_file_pal_handle; ret = 0; out: free(normpath); - if (ret < 0) + if (ret < 0) { PalObjectDestroy(pal_handle); + if (recovery_file_pal_handle) + PalObjectDestroy(recovery_file_pal_handle); + } return ret; } @@ -251,11 +326,21 @@ static void encrypted_file_internal_close(struct libos_encrypted_file* enc) { pf_status_t pfs = pf_close(enc->pf); if (PF_FAILURE(pfs)) { log_warning("pf_close failed: %s", pf_strerror(pfs)); + /* `pf_close` may fail due to a recoverable flush error; keep the recovery file for + * potential recovery. */ + goto out; } + if (enc->recovery_file_pal_handle) + (void)PalStreamDelete(enc->recovery_file_pal_handle, PAL_DELETE_ALL); + +out: enc->pf = NULL; PalObjectDestroy(enc->pal_handle); enc->pal_handle = NULL; + if (enc->recovery_file_pal_handle) + PalObjectDestroy(enc->recovery_file_pal_handle); + enc->recovery_file_pal_handle = NULL; return; } @@ -446,7 +531,7 @@ void update_encrypted_files_key(struct libos_encrypted_files_key* key, const pf_ } static int encrypted_file_alloc(const char* uri, struct libos_encrypted_files_key* key, - struct libos_encrypted_file** out_enc) { + bool enable_recovery, struct libos_encrypted_file** out_enc) { assert(strstartswith(uri, URI_PREFIX_FILE)); if (!key) { @@ -467,14 +552,18 @@ static int encrypted_file_alloc(const char* uri, struct libos_encrypted_files_ke enc->use_count = 0; enc->pf = NULL; enc->pal_handle = NULL; + + enc->enable_recovery = enable_recovery; + enc->recovery_file_pal_handle = NULL; + *out_enc = enc; return 0; } int encrypted_file_open(const char* uri, struct libos_encrypted_files_key* key, - struct libos_encrypted_file** out_enc) { + bool enable_recovery, struct libos_encrypted_file** out_enc) { struct libos_encrypted_file* enc; - int ret = encrypted_file_alloc(uri, key, &enc); + int ret = encrypted_file_alloc(uri, key, enable_recovery, &enc); if (ret < 0) return ret; @@ -490,9 +579,9 @@ int encrypted_file_open(const char* uri, struct libos_encrypted_files_key* key, } int encrypted_file_create(const char* uri, mode_t perm, struct libos_encrypted_files_key* key, - struct libos_encrypted_file** out_enc) { + bool enable_recovery, struct libos_encrypted_file** out_enc) { struct libos_encrypted_file* enc; - int ret = encrypted_file_alloc(uri, key, &enc); + int ret = encrypted_file_alloc(uri, key, enable_recovery, &enc); if (ret < 0) return ret; @@ -510,6 +599,7 @@ void encrypted_file_destroy(struct libos_encrypted_file* enc) { assert(enc->use_count == 0); assert(!enc->pf); assert(!enc->pal_handle); + assert(!enc->recovery_file_pal_handle); free(enc->uri); free(enc); } @@ -762,6 +852,8 @@ BEGIN_CP_FUNC(encrypted_file) { new_enc = (struct libos_encrypted_file*)(base + off); new_enc->use_count = enc->use_count; + new_enc->enable_recovery = enc->enable_recovery; + DO_CP_MEMBER(str, enc, new_enc, uri); lock(&g_keys_lock); @@ -776,6 +868,12 @@ BEGIN_CP_FUNC(encrypted_file) { DO_CP(palhdl_ptr, &enc->pal_handle, &entry); entry->phandle = &new_enc->pal_handle; } + + if (enc->recovery_file_pal_handle) { + struct libos_palhdl_entry* entry; + DO_CP(palhdl_ptr, &enc->recovery_file_pal_handle, &entry); + entry->phandle = &new_enc->recovery_file_pal_handle; + } ADD_CP_FUNC_ENTRY(off); if (objp) diff --git a/libos/test/fs/manifest.template b/libos/test/fs/manifest.template index 612873c068..945735856c 100644 --- a/libos/test/fs/manifest.template +++ b/libos/test/fs/manifest.template @@ -11,7 +11,7 @@ fs.mounts = [ { path = "/mounted", uri = "file:tmp" }, { type = "encrypted", path = "/tmp/enc_input", uri = "file:tmp/enc_input" }, - { type = "encrypted", path = "/tmp/enc_output", uri = "file:tmp/enc_output" }, + { type = "encrypted", path = "/tmp/enc_output", uri = "file:tmp/enc_output", enable_recovery = true }, { type = "encrypted", path = "/mounted/enc_input", uri = "file:tmp/enc_input" }, { type = "encrypted", path = "/mounted/enc_output", uri = "file:tmp/enc_output" }, { type = "tmpfs", path = "/mnt-tmpfs" }, diff --git a/pal/include/host/linux-common/linux_utils.h b/pal/include/host/linux-common/linux_utils.h index 22127a9ea5..aa80f847b4 100644 --- a/pal/include/host/linux-common/linux_utils.h +++ b/pal/include/host/linux-common/linux_utils.h @@ -55,3 +55,5 @@ void file_attrcopy(PAL_STREAM_ATTR* attr, struct stat* stat); int create_reserved_mem_ranges_fd(void* reserved_mem_ranges, size_t reserved_mem_ranges_size); void probe_stack(size_t pages_count); + +int recover_encrypted_file(int file_fd, int recovery_file_fd, size_t node_size); diff --git a/pal/include/pal/pal.h b/pal/include/pal/pal.h index 0e2229f92a..9980af4056 100644 --- a/pal/include/pal/pal.h +++ b/pal/include/pal/pal.h @@ -1043,4 +1043,16 @@ void PalGetLazyCommitPages(uintptr_t addr, size_t size, uint8_t* bitvector); */ int PalFreeThenLazyReallocCommittedPages(void* addr, size_t size); +/*! + * \brief Recover an encrypted file. + * + * \param handle Handle to the file. + * \param recovery_file_handle Handle to the recovery file. + * \param node_size Size of the pf node. + * + * \returns 0 on success, negative error code on failure. + */ +int PalRecoverEncryptedFile(PAL_HANDLE file_handle, PAL_HANDLE recovery_file_handle, + size_t node_size); + #undef INSIDE_PAL_H diff --git a/pal/include/pal_internal.h b/pal/include/pal_internal.h index a8695e5448..e1dda3350e 100644 --- a/pal/include/pal_internal.h +++ b/pal/include/pal_internal.h @@ -318,3 +318,6 @@ extern int (*g_mem_bkeep_get_vma_info_upcall)(uintptr_t addr, pal_prot_flags_t* void _PalGetLazyCommitPages(uintptr_t addr, size_t size, uint8_t* bitvector); int _PalFreeThenLazyReallocCommittedPages(void* addr, uint64_t size); + +int _PalRecoverEncryptedFile(PAL_HANDLE file_handle, PAL_HANDLE recovery_file_handle, + size_t node_size); diff --git a/pal/src/host/linux-common/file_utils.c b/pal/src/host/linux-common/file_utils.c index 6a9c6569a7..3fd14d4c9f 100644 --- a/pal/src/host/linux-common/file_utils.c +++ b/pal/src/host/linux-common/file_utils.c @@ -168,3 +168,71 @@ out:; return ret; } + +/* The recovery file is in the format of pairs of 8-byte offsets (physical node numbers) and + * encrypted file nodes. This function reads each pair from the recovery file and applies the + * encrypted file nodes to the corresponding offsets in the main file. */ +int recover_encrypted_file(int file_fd, int recovery_file_fd, size_t node_size) { + long ret; + size_t recovery_node_size = sizeof(uint64_t) + node_size; + char* recovery_node = NULL; + + ret = DO_SYSCALL(lseek, recovery_file_fd, 0, SEEK_END); + if (ret < 0) { + log_error("lseek failed: %s", unix_strerror(ret)); + goto out; + } + + if (ret == 0 || ret % recovery_node_size != 0) { + log_error("recovery file size is not right [%lu]", ret); + ret = -EINVAL; + goto out; + } + + size_t nodes_count = ret / recovery_node_size; + + ret = DO_SYSCALL(lseek, recovery_file_fd, 0, SEEK_SET); + if (ret < 0) { + log_error("lseek failed: %s", unix_strerror(ret)); + goto out; + } + + recovery_node = malloc(recovery_node_size); + if (!recovery_node) { + ret = -ENOMEM; + goto out; + } + + for (size_t i = 0; i < nodes_count; i++) { + ret = read_all(recovery_file_fd, recovery_node, recovery_node_size); + if (ret < 0) { + log_error("read_all failed: %s", unix_strerror(ret)); + goto out; + } + + size_t offset = 0; + memcpy(&offset, recovery_node, sizeof(uint64_t)); + ret = DO_SYSCALL(lseek, file_fd, offset * node_size, SEEK_SET); + if (ret < 0) { + log_error("lseek failed: %s", unix_strerror(ret)); + goto out; + } + + ret = write_all(file_fd, recovery_node + sizeof(uint64_t), node_size); + if (ret < 0) { + log_error("write_all failed: %s", unix_strerror(ret)); + goto out; + } + } + + ret = DO_SYSCALL(fsync, file_fd); + if (ret < 0) { + log_error("fsync failed: %s", unix_strerror(ret)); + goto out; + } + + ret = 0; +out: + free(recovery_node); + return (int)ret; +} diff --git a/pal/src/host/linux-sgx/enclave_ocalls.c b/pal/src/host/linux-sgx/enclave_ocalls.c index 71e25c324b..47f18270bb 100644 --- a/pal/src/host/linux-sgx/enclave_ocalls.c +++ b/pal/src/host/linux-sgx/enclave_ocalls.c @@ -2426,3 +2426,33 @@ int ocall_edmm_remove_pages(uint64_t addr, size_t count) { sgx_reset_ustack(old_ustack); return ret; } + +int ocall_recover_encrypted_file(int fd, int recovery_fd, size_t node_size) { + int retval = 0; + struct ocall_recover_encrypted_file* ocall_recover_encrypted_file_args; + + void* old_ustack = sgx_prepare_ustack(); + ocall_recover_encrypted_file_args = sgx_alloc_on_ustack_aligned( + sizeof(*ocall_recover_encrypted_file_args), alignof(*ocall_recover_encrypted_file_args)); + if (!ocall_recover_encrypted_file_args) { + sgx_reset_ustack(old_ustack); + return -EPERM; + } + + COPY_VALUE_TO_UNTRUSTED(&ocall_recover_encrypted_file_args->fd, fd); + COPY_VALUE_TO_UNTRUSTED(&ocall_recover_encrypted_file_args->recovery_fd, recovery_fd); + COPY_VALUE_TO_UNTRUSTED(&ocall_recover_encrypted_file_args->node_size, node_size); + + do { + retval = sgx_exitless_ocall(OCALL_RECOVER_ENCRYPTED_FILE, + ocall_recover_encrypted_file_args); + } while (retval == -EINTR); + + if (retval < 0 && retval != -EAGAIN && retval != -EWOULDBLOCK && retval != -EBADF && + retval != -EINTR && retval != -EINVAL && retval != -EIO && retval != -EISDIR) { + retval = -EPERM; + } + + sgx_reset_ustack(old_ustack); + return retval; +} diff --git a/pal/src/host/linux-sgx/enclave_ocalls.h b/pal/src/host/linux-sgx/enclave_ocalls.h index 595731ef48..9885f27c08 100644 --- a/pal/src/host/linux-sgx/enclave_ocalls.h +++ b/pal/src/host/linux-sgx/enclave_ocalls.h @@ -140,3 +140,5 @@ int ocall_get_qe_targetinfo(sgx_target_info_t* qe_targetinfo); int ocall_edmm_restrict_pages_perm(uint64_t addr, size_t count, uint64_t prot); int ocall_edmm_modify_pages_type(uint64_t addr, size_t count, uint64_t type); int ocall_edmm_remove_pages(uint64_t addr, size_t count); + +int ocall_recover_encrypted_file(int fd, int recovery_fd, size_t node_size); diff --git a/pal/src/host/linux-sgx/host_ocalls.c b/pal/src/host/linux-sgx/host_ocalls.c index 7d36875dd6..0b88280e02 100644 --- a/pal/src/host/linux-sgx/host_ocalls.c +++ b/pal/src/host/linux-sgx/host_ocalls.c @@ -737,6 +737,11 @@ static long sgx_ocall_edmm_restrict_pages_perm(void* _args) { return edmm_restrict_pages_perm(args->addr, args->count, args->prot); } +static long sgx_ocall_recover_encrypted_file(void* _args) { + struct ocall_recover_encrypted_file* args = _args; + return recover_encrypted_file(args->fd, args->recovery_fd, args->node_size); +} + sgx_ocall_fn_t ocall_table[OCALL_NR] = { [OCALL_EXIT] = sgx_ocall_exit, [OCALL_MMAP_UNTRUSTED] = sgx_ocall_mmap_untrusted, @@ -788,6 +793,7 @@ sgx_ocall_fn_t ocall_table[OCALL_NR] = { [OCALL_EDMM_MODIFY_PAGES_TYPE] = sgx_ocall_edmm_modify_pages_type, [OCALL_EDMM_REMOVE_PAGES] = sgx_ocall_edmm_remove_pages, [OCALL_EDMM_RESTRICT_PAGES_PERM] = sgx_ocall_edmm_restrict_pages_perm, + [OCALL_RECOVER_ENCRYPTED_FILE] = sgx_ocall_recover_encrypted_file, }; static int rpc_thread_loop(void* arg) { diff --git a/pal/src/host/linux-sgx/pal_misc.c b/pal/src/host/linux-sgx/pal_misc.c index 9ee161f9e9..720c4552b2 100644 --- a/pal/src/host/linux-sgx/pal_misc.c +++ b/pal/src/host/linux-sgx/pal_misc.c @@ -897,3 +897,13 @@ int _PalFreeThenLazyReallocCommittedPages(void* addr, uint64_t size) { return 0; } + +int _PalRecoverEncryptedFile(PAL_HANDLE file_handle, PAL_HANDLE recovery_file_handle, + size_t node_size) { + int ret = ocall_recover_encrypted_file(file_handle->file.fd, recovery_file_handle->file.fd, + node_size); + if (ret < 0) + return unix_to_pal_error(ret); + + return 0; +} diff --git a/pal/src/host/linux-sgx/pal_ocall_types.h b/pal/src/host/linux-sgx/pal_ocall_types.h index d20bab3a19..fe6e027620 100644 --- a/pal/src/host/linux-sgx/pal_ocall_types.h +++ b/pal/src/host/linux-sgx/pal_ocall_types.h @@ -74,6 +74,7 @@ enum { OCALL_EDMM_RESTRICT_PAGES_PERM, OCALL_EDMM_MODIFY_PAGES_TYPE, OCALL_EDMM_REMOVE_PAGES, + OCALL_RECOVER_ENCRYPTED_FILE, OCALL_NR, }; @@ -363,4 +364,10 @@ struct ocall_edmm_remove_pages { size_t count; }; +struct ocall_recover_encrypted_file { + int fd; + int recovery_fd; + size_t node_size; +}; + #pragma pack(pop) diff --git a/pal/src/host/linux/pal_misc.c b/pal/src/host/linux/pal_misc.c index 44da9ccd21..fd72533010 100644 --- a/pal/src/host/linux/pal_misc.c +++ b/pal/src/host/linux/pal_misc.c @@ -101,3 +101,12 @@ int _PalFreeThenLazyReallocCommittedPages(void* addr, size_t size) { int ret = DO_SYSCALL(madvise, addr, size, MADV_DONTNEED); return ret < 0 ? unix_to_pal_error(ret) : 0; } + +int _PalRecoverEncryptedFile(PAL_HANDLE file_handle, PAL_HANDLE recovery_file_handle, + size_t node_size) { + int ret = recover_encrypted_file(file_handle->file.fd, recovery_file_handle->file.fd, node_size); + if (ret < 0) + return unix_to_pal_error(ret); + + return 0; +} diff --git a/pal/src/host/skeleton/pal_misc.c b/pal/src/host/skeleton/pal_misc.c index 839bcd9a40..7e09515c11 100644 --- a/pal/src/host/skeleton/pal_misc.c +++ b/pal/src/host/skeleton/pal_misc.c @@ -82,3 +82,11 @@ int _PalFreeThenLazyReallocCommittedPages(void* addr, uint64_t size) { __UNUSED(size); return PAL_ERROR_NOTIMPLEMENTED; } + +int _PalRecoverEncryptedFile(PAL_HANDLE file_handle, PAL_HANDLE recovery_file_handle, + size_t node_size) { + __UNUSED(file_handle); + __UNUSED(recovery_file_handle); + __UNUSED(node_size); + return 0; +} diff --git a/pal/src/pal_misc.c b/pal/src/pal_misc.c index 4aa3b9dac1..92e61408ba 100644 --- a/pal/src/pal_misc.c +++ b/pal/src/pal_misc.c @@ -67,3 +67,10 @@ int PalFreeThenLazyReallocCommittedPages(void* addr, size_t size) { return _PalFreeThenLazyReallocCommittedPages(addr, size); } + +int PalRecoverEncryptedFile(PAL_HANDLE file_handle, PAL_HANDLE recovery_file_handle, + size_t node_size) { + assert(file_handle && recovery_file_handle); + + return _PalRecoverEncryptedFile(file_handle, recovery_file_handle, node_size); +} diff --git a/pal/src/pal_symbols b/pal/src/pal_symbols index dc6365ca30..64704e410a 100644 --- a/pal/src/pal_symbols +++ b/pal/src/pal_symbols @@ -54,3 +54,4 @@ PalDebugLog PalGetPalPublicState PalGetLazyCommitPages PalFreeThenLazyReallocCommittedPages +PalRecoverEncryptedFile diff --git a/python/graminelibos/manifest_check.py b/python/graminelibos/manifest_check.py index 9fe5590812..a03bda779d 100644 --- a/python/graminelibos/manifest_check.py +++ b/python/graminelibos/manifest_check.py @@ -30,6 +30,7 @@ Required('type'): 'encrypted', Required('uri'): _uri, 'key_name': str, + 'enable_recovery': bool, }, { Required('type'): 'tmpfs', diff --git a/tools/common/pf_util/pf_util.c b/tools/common/pf_util/pf_util.c index eadd0323c3..6ddb63dbbd 100644 --- a/tools/common/pf_util/pf_util.c +++ b/tools/common/pf_util/pf_util.c @@ -330,7 +330,7 @@ int pf_encrypt_file(const char* input_path, const char* output_path, const pf_ke pf_handle_t handle = (pf_handle_t)&output; pf_status_t pfs = pf_open(handle, norm_output_path, /*size=*/0, PF_FILE_MODE_WRITE, - /*create=*/true, wrap_key, &pf); + /*create=*/true, wrap_key, /*recovery_file_handle=*/NULL, &pf); if (PF_FAILURE(pfs)) { ERROR("Failed to open output PF: %s\n", pf_strerror(pfs)); goto out; @@ -438,7 +438,7 @@ int pf_decrypt_file(const char* input_path, const char* output_path, bool verify } pf_status_t pfs = pf_open((pf_handle_t)&input, norm_input_path, input_size, PF_FILE_MODE_READ, - /*create=*/false, wrap_key, &pf); + /*create=*/false, wrap_key, /*recovery_file_handle=*/NULL, &pf); if (PF_FAILURE(pfs)) { ERROR("Opening protected input file failed: %s\n", pf_strerror(pfs)); goto out;