Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add custom-persist extension #650

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

Guiiix
Copy link
Member

@Guiiix Guiiix commented Jan 28, 2025

custom-persist extension (see QubesOS/qubes-issues#1006)

Adds a custom-persist feature. The service custom-persist must be activated to enable the feature.
When the feature is enabled, user custom persistent file and dirs must be configured with qvm-features. Example: qvm-features work custom-persist.var_test /var/test

@DemiMarie
Copy link
Contributor

@Guiiix Is there a companion PR to core-agent-linux?

@Guiiix
Copy link
Member Author

Guiiix commented Jan 29, 2025

Hi @DemiMarie, yes this one: QubesOS/qubes-core-agent-linux#551

I'll fix lint errors today.

@Guiiix Guiiix force-pushed the custom-persist branch 4 times, most recently from 68bcfbc to 2d64eae Compare February 2, 2025 10:18
Copy link

codecov bot commented Feb 2, 2025

Codecov Report

Attention: Patch coverage is 98.46154% with 1 line in your changes missing coverage. Please review.

Project coverage is 69.89%. Comparing base (c1cdf38) to head (78809ab).
Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
qubes/ext/custom_persist.py 98.46% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #650      +/-   ##
==========================================
+ Coverage   69.71%   69.89%   +0.18%     
==========================================
  Files          58       59       +1     
  Lines       12499    12607     +108     
==========================================
+ Hits         8714     8812      +98     
- Misses       3785     3795      +10     
Flag Coverage Δ
unittests 69.89% <98.46%> (+0.18%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@marmarek
Copy link
Member

marmarek commented Feb 3, 2025

Please add some unit test

@Guiiix Guiiix force-pushed the custom-persist branch 5 times, most recently from 1f0c3ec to 929a076 Compare February 7, 2025 11:05
@Guiiix
Copy link
Member Author

Guiiix commented Feb 7, 2025

Tests added

@marmarek
Copy link
Member

I'll also add an integration test. In the meantime, can you update documentation at https://www.qubes-os.org/doc/bind-dirs/ to include this feature too? Or maybe you see some better place for that?

@qubesos-bot
Copy link

qubesos-bot commented Feb 13, 2025

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025022709-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025021804-4.3&flavor=update

  • system_tests_basic_vm_qrexec_gui

  • system_tests_backupcompatibility

    • startup: unnamed test (unknown)

    • startup: Failed (test died)
      # Test died: no candidate needle with tag(s) 'bootloader, luks-prom...

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: qr/PS1="root# "/u...

  • system_tests_gui_interactive

    • clipboard_and_web: unnamed test (unknown)

    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'personal-firefox' mat...

    • clipboard_and_web: wait_serial (wait serial expected)
      # wait_serial expected: "lspci; echo 2E8vz-\$?-"...

    • clipboard_and_web: wait_serial (wait serial expected)
      # wait_serial expected: qr/2E8vz-\d+-/...

  • system_tests_guivm_gui_interactive

    • update_guivm: Failed (test died)
      # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...
  • system_tests_qrexec

    • TC_00_Qrexec_whonix-gateway-17: test_065_qrexec_exit_code_vm (failure)
      ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^... AssertionError: b'125\n' != b'0\n'
  • system_tests_network_updates

  • system_tests_backupdispvm

    • TC_10_RestoreInDispVM_debian-12-xfce: test_000_basic_backup (failure)
      test-inst-testhvm1 | StandaloneVM | n/a | ...

    • TC_10_RestoreInDispVM_fedora-41-xfce: test_000_basic_backup (failure)
      test-inst-testhvm1 | StandaloneVM | n/a | ...

    • TC_10_RestoreInDispVM_whonix-gateway-17: test_000_basic_backup (failure)
      test-inst-testhvm1 | StandaloneVM | n/a | ...

    • TC_10_RestoreInDispVM_whonix-workstation-17: test_000_basic_backup (failure)
      test-inst-testhvm1 | StandaloneVM | n/a | ...

  • system_tests_kde_gui_interactive

    • kde_install: Failed (test died)
      # Test died: command '(set -o pipefail; sudo qubes-dom0-update -y k...
  • system_tests_basic_vm_qrexec_gui_zfs

    • switch_pool: Failed (test died)
      # Test died: command 'qubes-dom0-update -y zfs' failed at /usr/lib/...
  • system_tests_guivm_vnc_gui_interactive

    • startup: unnamed test (unknown)
    • startup: Failed (test died)
      # Test died: no candidate needle with tag(s) 'bootloader, luks-prom...
  • system_tests_audio

  • system_tests_basic_vm_qrexec_gui_btrfs

  • system_tests_basic_vm_qrexec_gui_ext4

  • system_tests_basic_vm_qrexec_gui_xfs

  • system_tests_suspend@hw1

    • suspend: wait_serial (wait serial expected)
      # wait_serial expected: qr/z2_1S-\d+-/...
  • system_tests_basic_vm_qrexec_gui@hw7

  • system_tests_guivm_gpu_gui_interactive@hw13

    • update_guivm: Failed (test died)
      # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...

Failed tests

41 failures
  • system_tests_basic_vm_qrexec_gui

  • system_tests_backupcompatibility

    • startup: unnamed test (unknown)

    • startup: Failed (test died)
      # Test died: no candidate needle with tag(s) 'bootloader, luks-prom...

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: qr/PS1="root# "/u...

  • system_tests_gui_interactive

    • clipboard_and_web: unnamed test (unknown)

    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'personal-firefox' mat...

    • clipboard_and_web: wait_serial (wait serial expected)
      # wait_serial expected: "lspci; echo 2E8vz-\$?-"...

    • clipboard_and_web: wait_serial (wait serial expected)
      # wait_serial expected: qr/2E8vz-\d+-/...

  • system_tests_guivm_gui_interactive

    • update_guivm: Failed (test died)
      # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...
  • system_tests_qrexec

    • TC_00_Qrexec_whonix-gateway-17: test_065_qrexec_exit_code_vm (failure)
      ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^... AssertionError: b'125\n' != b'0\n'
  • system_tests_network_updates

  • system_tests_backupdispvm

    • TC_10_RestoreInDispVM_debian-12-xfce: test_000_basic_backup (failure)
      test-inst-testhvm1 | StandaloneVM | n/a | ...

    • TC_10_RestoreInDispVM_fedora-41-xfce: test_000_basic_backup (failure)
      test-inst-testhvm1 | StandaloneVM | n/a | ...

    • TC_10_RestoreInDispVM_whonix-gateway-17: test_000_basic_backup (failure)
      test-inst-testhvm1 | StandaloneVM | n/a | ...

    • TC_10_RestoreInDispVM_whonix-workstation-17: test_000_basic_backup (failure)
      test-inst-testhvm1 | StandaloneVM | n/a | ...

  • system_tests_kde_gui_interactive

    • kde_install: Failed (test died)
      # Test died: command '(set -o pipefail; sudo qubes-dom0-update -y k...
  • system_tests_basic_vm_qrexec_gui_zfs

    • switch_pool: Failed (test died)
      # Test died: command 'qubes-dom0-update -y zfs' failed at /usr/lib/...
  • system_tests_guivm_vnc_gui_interactive

    • startup: unnamed test (unknown)
    • startup: Failed (test died)
      # Test died: no candidate needle with tag(s) 'bootloader, luks-prom...
  • system_tests_audio

  • system_tests_basic_vm_qrexec_gui_btrfs

  • system_tests_basic_vm_qrexec_gui_ext4

  • system_tests_basic_vm_qrexec_gui_xfs

  • system_tests_suspend@hw1

    • suspend: wait_serial (wait serial expected)
      # wait_serial expected: qr/z2_1S-\d+-/...

    • suspend: Failed (test died + timed out)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...

  • system_tests_basic_vm_qrexec_gui@hw7

  • system_tests_guivm_gpu_gui_interactive@hw13

    • update_guivm: Failed (test died)
      # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/129058#dependencies

14 fixed
  • system_tests_whonix

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

  • system_tests_suspend

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died)
      # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...
  • system_tests_qrexec

  • system_tests_kde_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'clipboard-paste-notif...
  • system_tests_audio

  • system_tests_whonix@hw7

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

  • system_tests_basic_vm_qrexec_gui_btrfs

    • TC_03_QvmRevertTemplateChanges: test_000_revert_linux (error)
      subprocess.CalledProcessError: Command '['sha1sum', '/var/lib/qubes...
  • system_tests_suspend@hw1

Unstable tests

Performance Tests

Performance degradation:

27 performance degradations
  • debian-12-xfce_exec-data-simplex: 62.46 :small_red_triangle_up: ( previous job: 48.93, degradation: 127.66%)
  • debian-12-xfce_exec-data-duplex: 71.11 :small_red_triangle_up: ( previous job: 50.76, degradation: 140.08%)
  • debian-12-xfce_socket-data-duplex: 165.56 :small_red_triangle_up: ( previous job: 81.49, degradation: 203.17%)
  • fedora-41-xfce_exec-data-simplex: 60.55 :small_red_triangle_up: ( previous job: 49.65, degradation: 121.96%)
  • fedora-41-xfce_exec-data-duplex: 64.91 :small_red_triangle_up: ( previous job: 49.08, degradation: 132.26%)
  • fedora-41-xfce_exec-data-duplex-root: 111.28 :small_red_triangle_up: ( previous job: 81.65, degradation: 136.28%)
  • fedora-41-xfce_socket-data-duplex: 151.22 :small_red_triangle_up: ( previous job: 78.62, degradation: 192.33%)
  • whonix-gateway-17_exec-data-simplex: 79.21 :small_red_triangle_up: ( previous job: 48.76, degradation: 162.43%)
  • whonix-gateway-17_exec-data-duplex: 75.20 :small_red_triangle_up: ( previous job: 48.55, degradation: 154.88%)
  • whonix-gateway-17_exec-data-duplex-root: 103.48 :small_red_triangle_up: ( previous job: 70.13, degradation: 147.57%)
  • whonix-gateway-17_socket-data-duplex: 166.54 :small_red_triangle_up: ( previous job: 82.74, degradation: 201.28%)
  • whonix-workstation-17_socket-root: 9.07 :small_red_triangle_up: ( previous job: 8.20, degradation: 110.66%)
  • whonix-workstation-17_exec-data-simplex: 68.01 :small_red_triangle_up: ( previous job: 47.01, degradation: 144.65%)
  • whonix-workstation-17_exec-data-duplex: 70.43 :small_red_triangle_up: ( previous job: 49.48, degradation: 142.35%)
  • whonix-workstation-17_socket-data-duplex: 162.92 :small_red_triangle_up: ( previous job: 81.71, degradation: 199.38%)
  • dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 422132.00 :small_red_triangle_up: ( previous job: 486352.00, degradation: 86.80%)
  • dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 144860.00 :small_red_triangle_up: ( previous job: 276742.00, degradation: 52.34%)
  • dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 58998.00 :small_red_triangle_up: ( previous job: 423495.00, degradation: 13.93%)
  • dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 55533.00 :small_red_triangle_up: ( previous job: 185030.00, degradation: 30.01%)
  • dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 28775.00 :small_red_triangle_up: ( previous job: 100699.00, degradation: 28.58%)
  • dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 6295.00 :small_red_triangle_up: ( previous job: 8767.00, degradation: 71.80%)
  • fedora-41-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 339454.00 :small_red_triangle_up: ( previous job: 396586.00, degradation: 85.59%)
  • fedora-41-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 304641.00 :small_red_triangle_up: ( previous job: 343795.00, degradation: 88.61%)
  • fedora-41-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 2250.00 :small_red_triangle_up: ( previous job: 3785.00, degradation: 59.45%)
  • fedora-41-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 1531.00 :small_red_triangle_up: ( previous job: 3885.00, degradation: 39.41%)
  • fedora-41-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 41720.00 :small_red_triangle_up: ( previous job: 78093.00, degradation: 53.42%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1798.00 :small_red_triangle_up: ( previous job: 2693.00, degradation: 66.77%)

Remaining performance tests:

45 tests
  • debian-12-xfce_exec: 6.90 🟢 ( previous job: 7.15, improvement: 96.53%)
  • debian-12-xfce_exec-root: 28.85 :small_red_triangle_up: ( previous job: 27.97, degradation: 103.16%)
  • debian-12-xfce_socket: 8.55 :small_red_triangle_up: ( previous job: 8.33, degradation: 102.66%)
  • debian-12-xfce_socket-root: 8.67 :small_red_triangle_up: ( previous job: 8.20, degradation: 105.67%)
  • debian-12-xfce_exec-data-duplex-root: 70.17 :small_red_triangle_up: ( previous job: 64.91, degradation: 108.11%)
  • fedora-41-xfce_exec: 9.21 :small_red_triangle_up: ( previous job: 9.13, degradation: 100.85%)
  • fedora-41-xfce_exec-root: 64.39 :small_red_triangle_up: ( previous job: 61.17, degradation: 105.26%)
  • fedora-41-xfce_socket: 8.70 :small_red_triangle_up: ( previous job: 8.66, degradation: 100.49%)
  • fedora-41-xfce_socket-root: 8.86 :small_red_triangle_up: ( previous job: 8.61, degradation: 102.90%)
  • whonix-gateway-17_exec: 7.31 🟢 ( previous job: 7.87, improvement: 92.87%)
  • whonix-gateway-17_exec-root: 39.80 :small_red_triangle_up: ( previous job: 38.36, degradation: 103.74%)
  • whonix-gateway-17_socket: 7.58 :small_red_triangle_up: ( previous job: 7.54, degradation: 100.50%)
  • whonix-gateway-17_socket-root: 8.60 :small_red_triangle_up: ( previous job: 8.27, degradation: 104.00%)
  • whonix-workstation-17_exec: 7.73 🟢 ( previous job: 8.23, improvement: 93.98%)
  • whonix-workstation-17_exec-root: 54.50 :small_red_triangle_up: ( previous job: 52.56, degradation: 103.68%)
  • whonix-workstation-17_socket: 8.65 :small_red_triangle_up: ( previous job: 8.21, degradation: 105.31%)
  • whonix-workstation-17_exec-data-duplex-root: 84.35 :small_red_triangle_up: ( previous job: 79.93, degradation: 105.53%)
  • dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 5793.00 :green_circle: ( previous job: 3277.00, improvement: 176.78%)
  • dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 9396.00 :small_red_triangle_up: ( previous job: 10163.00, degradation: 92.45%)
  • dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 2842.00 :green_circle: ( previous job: 282.00, improvement: 1007.80%)
  • dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 511500.00 :green_circle: ( previous job: 475329.00, improvement: 107.61%)
  • dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 166872.00 :green_circle: ( previous job: 95209.00, improvement: 175.27%)
  • dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 432759.00 :small_red_triangle_up: ( previous job: 433474.00, degradation: 99.84%)
  • dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 192293.00 :green_circle: ( previous job: 164133.00, improvement: 117.16%)
  • dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 106402.00 :green_circle: ( previous job: 99808.00, improvement: 106.61%)
  • dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 7788.00 :green_circle: ( previous job: 7053.00, improvement: 110.42%)
  • dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 4432.00 :green_circle: ( previous job: 3868.00, improvement: 114.58%)
  • fedora-41-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 125914.00 :green_circle: ( previous job: 99783.00, improvement: 126.19%)
  • fedora-41-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 44557.00 :small_red_triangle_up: ( previous job: 44770.00, degradation: 99.52%)
  • fedora-41-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 86920.00 :green_circle: ( previous job: 86742.00, improvement: 100.21%)
  • fedora-41-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8375.00 :small_red_triangle_up: ( previous job: 8623.00, degradation: 97.12%)
  • fedora-41-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 1171.00 :green_circle: ( previous job: 1126.00, improvement: 104.00%)
  • fedora-41-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 373291.00 :small_red_triangle_up: ( previous job: 401907.00, degradation: 92.88%)
  • fedora-41-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 146606.00 :green_circle: ( previous job: 116848.00, improvement: 125.47%)
  • fedora-41-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 345722.00 :small_red_triangle_up: ( previous job: 357875.00, degradation: 96.60%)
  • fedora-41-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 67489.00 :green_circle: ( previous job: 41375.00, improvement: 163.12%)
  • fedora-41-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 91934.00 :green_circle: ( previous job: 87999.00, improvement: 104.47%)
  • fedora-41-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8303.00 :small_red_triangle_up: ( previous job: 8744.00, degradation: 94.96%)
  • fedora-41-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 1584.00 :small_red_triangle_up: ( previous job: 1613.00, degradation: 98.20%)
  • fedora-41-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 361453.00 :small_red_triangle_up: ( previous job: 392725.00, degradation: 92.04%)
  • fedora-41-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 158854.00 :green_circle: ( previous job: 139933.00, improvement: 113.52%)
  • fedora-41-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 288784.00 :small_red_triangle_up: ( previous job: 294875.00, degradation: 97.93%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 80667.00 :green_circle: ( previous job: 71108.00, improvement: 113.44%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 4403.00 :green_circle: ( previous job: 3959.00, improvement: 111.21%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 7811.00 :small_red_triangle_up: ( previous job: 8408.00, degradation: 92.90%)

This was referenced Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants