Synchronise yoga with upstream #116
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
It has been asserted that we should not be calling qemu-img info on untrusted files. That means we need to know if they have a backing_file, data_file or other unsafe configuration *before* we use qemu-img to probe or convert them. This grafts glance's format_inspector module into nova/images so we can use it to check the file early for safety. The expectation is that this will be moved to oslo.utils (or something) later and thus we will just delete the file from nova and change our import when that happens. NOTE: This includes whitespace changes from the glance version of format_inspector.py because of autopep8 demands. Conflicts: nova/conf/workarounds.py NOTE(elod.illes): conflict is due to the following patch that is only present in zed: Iab92124b5776a799c7f90d07281d28fcf191c8fe Change-Id: Iaefbe41b4c4bf0cf95d8f621653fdf65062aaa59 Closes-Bug: #2059809 (cherry picked from commit 9cdce71) (cherry picked from commit f07fa55) (cherry picked from commit 0acf5ee) (cherry picked from commit 67e5376) (cherry picked from commit da352ed)
There is an additional way we can be fooled into using a qcow2 file with a data-file, which is uploading it as raw to glance and then booting an instance from it. Because when we go to create the ephemeral disk from a cached base image, we've lost the information about the original source's format, we probe the image's file type without a strict format specified. If a qcow2 file is listed in glance as a raw, we won't notice it until it is too late. This brings over another piece of code (proposed against) glance's format inspector which provides a safe format detection routine. This patch uses that to detect the format of and run a safety check on the base image each time we go to use it to create an ephemeral disk image from it. This also detects QED files and always marks them as unsafe as we do not support that format at all. Since we could be fooled into downloading one and passing it to qemu-img if we don't recognize it, we need to detect and reject it as unsafe. Conflicts: nova/tests/unit/virt/libvirt/test_utils.py nova/virt/libvirt/utils.py NOTE(elod.illes): conflicts are due to patch to consolidate image creation functions (I111cfc8a5eae27b15c6312957255fcf973038ddf) is only introduced in zed. Change-Id: I4881c8cbceb30c1ff2d2b859c554e0d02043f1f5 (cherry picked from commit b1b88bf) (cherry picked from commit 8a0d5f2) (cherry picked from commit 0269234) (cherry picked from commit 9e10ac2) (cherry picked from commit 303c2c9)
This restores the vmdk_allowed_types checking in create_image() that was unintentionally lost by tightening the qemu-type-matches-glance code in the fetch patch recently. Since we are still detecting the format of base images without metadata, we would have treated a vmdk file that claims to be raw as raw in fetch, but then read it like a vmdk once it was used as a base image for something else. Conflicts: nova/tests/unit/virt/libvirt/test_utils.py nova/virt/libvirt/utils.py NOTE(elod.illes): conflicts are due to patch to consolidate image creation functions (I111cfc8a5eae27b15c6312957255fcf973038ddf) is only introduced in zed. Change-Id: I07b332a7edb814f6a91661651d9d24bfd6651ae7 Related-Bug: #2059809 (cherry picked from commit 08be7b2) (cherry picked from commit 11301e7) (cherry picked from commit 70a435f) (cherry picked from commit f732f84) (cherry picked from commit a2acb31)
Latest Zuul drops the following warnings: All regular expressions must conform to RE2 syntax, but an expression using the deprecated Perl-style syntax has been detected. Adjust the configuration to conform to RE2 syntax. The RE2 syntax error is: invalid perl operator: (?! This patch replaces the 'irrelevant-files' to 'files' with explicitly listing the pattern which files should be the tests run against. NOTE(elod.illes): this patch is extended with the removal of CentOS 8 Stream based jobs as that OS is End of Life, hence removed from available nodesets. Change-Id: If287e800fb9ff428dbe6f9c4c046627f22afe3df (cherry picked from commit 9b77bae) (cherry picked from commit 8223e6a) (cherry picked from commit 510a27b) (cherry picked from commit 197b14d) (cherry picked from commit ffc252e)
This commit is a direct port of the format inspector unit tests from glance as of commit 0d8e79b713bc31a78f0f4eac14ee594ca8520999 the only changes to the test are as follows "from glance.common import format_inspector" was updated to "from nova.image import format_inspector" "from glance.tests import utils as test_utils" was replaced with "from nova import test" "test_utils.BaseTestCase" was replaced with "test.NoDBTestCase" "glance-unittest-formatinspector-" was replaced with "nova-unittest-formatinspector-" This makes the test funtional in nova. TestFormatInspectors requries qemu-img to be installed on the host which would be a new depency for executing unit tests. to avoid that we skip TestFormatInspectors if qemu-img is not installed. TestFormatInspectorInfra and TestFormatInspectorsTargeted do not have a qemu-img dependency so no changes to the test assertions were required. Note for yoga backport:With older qemu installed one of the qemu-img create commands fails, let's skip it from unmaintained/yoga and below that. Change-Id: Ia34203f246f0bc574e11476287dfb33fda7954fe (cherry picked from commit 838daa3) (cherry picked from commit 66205be) (cherry picked from commit 497abea) (cherry picked from commit 58cd955) (cherry picked from commit d7e3d72)
This change adds a reproducer for the regression in iso file support when workarounds.disable_deep_image_inspection = False Change-Id: I56d8b9980b4871941ba5de91e60a7df6a40106a8 (cherry picked from commit b5a1d3b) (cherry picked from commit 3a6d9a0) (cherry picked from commit 000b435) (cherry picked from commit 1233d7b) (cherry picked from commit fb86ca6)
This change includes unit tests for the ISO format inspector using mkisofs to generate the iso files. A test for stashing qcow content in the system_area of an iso file is also included. This change modifies format_inspector.detect_file_format to evaluate all inspectors until they are complete and raise an InvalidDiskInfo exception if multiple formats match. Related-Bug: #2059809 Change-Id: I7e12718fb3e1f77eb8d1cfcb9fa64e8ddeb9e712 (cherry picked from commit b1cc398) (cherry picked from commit eeda7c3) (cherry picked from commit 24628ec) (cherry picked from commit 65f0789) (cherry picked from commit e8f0061)
while backporting Ia34203f246f0bc574e11476287dfb33fda7954fe We observed that several of the tests showed distro specific behavior depending on if qemu was installed in the test env, what version is installed and how it was compiled This change ensures that if qemu is present that it supprot the required formats otherwise it skips the test. Change-Id: I131996cdd7aaf1f52d4caac33b153753ff6db869 (cherry picked from commit cc2514d) (cherry picked from commit ae10fde) (cherry picked from commit bb2645e) (cherry picked from commit 673103f) (cherry picked from commit dae4230)
Some version of mkisofs does not properly handle if both the input and the output file of the command are the same. So this commit changes the unit tests depending on that binary to use a different files. Related-Bug: #2059809 Change-Id: I6924eb23ff5804c22a48ec6fabcec25f061906bb (cherry picked from commit c6d8c69) (cherry picked from commit a8783a7) (cherry picked from commit 02147b3) (cherry picked from commit 47428f6) (cherry picked from commit 11613e7)
When we moved the qemu-img command in fetch_to_raw() to force the format to what we expect, we lost the ability to identify and react to situations where qemu-img detected a file as a format that is not supported by us (i.e. identfied and safety-checked by format_inspector). In the case of some of the other VMDK variants that we don't support, we need to be sure to catch any case where qemu-img thinks it's something other than raw when we think it is, which will be the case for those formats we don't support. Note this also moves us from explicitly using the format_inspector that we're told by glance is appropriate, to using our own detection. We assert that we agree with glance and as above, qemu agrees with us. This helps us avoid cases where the uploader lies about the image format, causing us to not run the appropriate safety check. AMI formats are a liability here since we have a very hard time asserting what they are and what they will be detected as later in the pipeline, so there is still special-casing for those. Closes-Bug: #2071734 Change-Id: I4b792c5bc959a904854c21565682ed3a687baa1a (cherry picked from commit 8b4c522) (cherry picked from commit 8ef5ec9) (cherry picked from commit 45d9489) (cherry picked from commit fbe4290) (cherry picked from commit a541252)
Written by gibi, I just cleaned it up. Change-Id: I8386a846b3685b8d03c59334ccfb2efbd4afe427 Co-Authored-By: Balazs Gibizer <[email protected]> Related-Bug: #2025480 (cherry picked from commit 62300d4) (cherry picked from commit 477ff26) (cherry picked from commit 23c190a) (cherry picked from commit 7422642)
This patch concerns the time when a VM is being unshelved and the compute manager set the task_state to spawning, claimed resources of the VM and then called driver.spawn(). So the instance is in vm_state SHELVED_OFFLOADED, task_state spawning. If at this point a new update_available_resource periodic job is started that collects all the instances assigned to the node to calculate resource usage. However the calculation assumed that a VM in SHELVED_OFFLOADED state does not need resource allocation on the node (probably being removed from the node as it is offloaded) and deleted the resource claim. Given all this we ended up with the VM spawned successfully but having lost the resource claim on the node. This patch changes what we do in vm_state SHELVED_OFFLOADED, task_state spawning. We no longer delete the resource claim in this state and keep tracking the resource in stats. Change-Id: I8c9944810c09d501a6d3f60f095d9817b756872d Closes-Bug: #2025480 (cherry picked from commit f1dc4ec) (cherry picked from commit 4239d1f) (cherry picked from commit 683ecc0) (cherry picked from commit e41962f)
py39 jobs (on ubuntu-focal) started to fail due to recent setuptools release (20.26.4) on Yoga, because we have 'packaging==21.3' in this branch that is not compatible with the new setuptools [1]. setuptools is bundled in virtualenv, so it has to be capped via the virtualenv package. tox also needed to be capped (<4) as gate uses tox 3.28.0, but with capping virtualenv we pull in latest tox as well, which would cause other errors. [1] pypa/setuptools#4483 Change-Id: Ie3fe3060d05d2553c5eeb7fd75b41e8f04bf11a7
This job is pretty heavy and has been triggering OOMs that take out mysqld lately. This disables swift (and c-bak as a result) to try to reduce the runtime footprint. Losing coverage of these services should not be a problem for the goal of this job. Change-Id: Icc18ddd847465069aea34b226851afaeb94594fc (cherry picked from commit 16a463a)
This makes the ceph-multistore job use the MYSQL_REDUCE_MEMORY flag in devstack to try to address the frequent OOMs we see in that job. Conflicts: .zuul.yaml Change-Id: Ibc203bd10dcb530027c2c9f58eb840ccc088280d Closes-Bug: #1961068 (cherry picked from commit 84d1f25) (cherry picked from commit 6c9ebea)
Note that this includes seemingly-unrelated test changes because we were actually skipping the snapshot_running test for libvirt, which has been a bug for years. In that test case, when we went to look for image_meta.disk_format, that attribute was not set on the o.vo object, which raised a NotImplementedError. That error is also checked by the test to skip the test for drivers that do not support snapshot, which meant that for libvirt, we haven't been running that case beyond the point at which we create snapshot metadata and trip that exception. Thus, once removing that, there are other mocks not in place that are required for the test to actually run. So, this adds mocks for qemu_img_info() calls that actually try to read the file on disk, as well as the privsep chown() that attempts to run after. Change-Id: Ie731045629f0899840a4680d21793a16ade9b98e (cherry picked from commit d5a631b) (cherry picked from commit 8c5929f) (cherry picked from commit d2d3b2c) (cherry picked from commit 77dfa4f) (cherry picked from commit e6f4503)
priteau
approved these changes
Oct 7, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains a snapshot of yoga from upstream unmaintained/yoga.