Implement first phase of two-phase conversion

This commit implements the first phase of the two-phase conversion, basically
everything unrelated to the actual output mode.

This also adds parsing of one more piece of information (paths to created
overlays -- currently an implementation detail which is not supposed to be
exposed and will be refactored later) into `log_parser.py`.

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>

Author: nertpinx

.travis.yml

@@ -19,6 +19,7 @@ addons:
     packages:
       - python-pip
       - libgnutls.dev  # required by pycurl
+      - libvirt.dev
 
 install:
   - pip install tox

Pipfile

@@ -7,6 +7,12 @@ name = "pypi"
 ovirt-engine-sdk-python = "*"
 pycurl = "*"
 six = "*"
+libvirt-python = "*"
+pyvmomi = "*"
+packaging = "*"
+# TODO: Uncomment this when it becomes available in pypi and also do the same in
+# setup.py, add it to requirements.txt and remove the mocked nbd.py
+# libnbd-python = ">1.0.0"
 
 [dev-packages]
 yamllint = "*"

docs/Virt-v2v-wrapper.md

@@ -154,10 +154,10 @@ Example:
 State file is a JSON file. Its content changes as the conversion goes through
 various stages. With it also the keys present in the file.
 
-If two-phase conversion is selected there will be a key `pre_copy` with various
-information related to the first (pre-copy) phase of the conversion.  The
-particular keys and information is, at the time of this writing, subject to
-change, but whould be as self-explanatory as possible.
+the output differs based on the type of conversion that is running:
+
+<a id="anchor_onephase"></a>
+### One-Phase Conversion
 
 Once virt-v2v is executed the state file is created with the following keys:
 
@@ -203,6 +203,15 @@ Right before the wrapper terminates it updates the state with:
 
 * `failed`: with value `true` if the conversion process failed.
 
+### Two-Phase Conversion
+
+* `disks`: This key includes a list of disks and for each one there is
+  information about the progress of copying disks before running virt-v2v.  The
+  data are pulled from the source server and they are being updated depending on
+  the progress.
+  
+Once virt-v2v is started all keys from the [One-Phase Conversion](#anchor_onephase)
+except `disks` and `disk_count` are updated in a similar fashion.
 
 ## LUKS encrypted devices
 

requirements.txt

@@ -6,3 +6,6 @@ tox
 flake8
 pep8-naming
 pylint
+libvirt-python
+pyvmomi
+packaging

setup.py

@@ -23,7 +23,14 @@
     },
     install_requires=[
         'pycurl',
-        'six'
+        'six',
+        'libvirt-python',
+        'pyvmomi',
+        'packaging',
+        # TODO: Uncomment this when it becomes available in pypi and also do
+        # the same in Pipfile, add it to requirements.txt and remove the mocked
+        # nbd.py
+        # 'libnbd-python',
     ],
     extras_require={
         'ovirt': 'ovirt-engine-sdk-python',

tox.ini

@@ -7,6 +7,8 @@ skip_missing_interpreters=True
 exclude = .git,.eggs,.tox,__pycache__,venv,build,dist
 
 [testenv]
+setenv =
+    PYTHONPATH={toxinidir}/wrapper/tests/mocks/
 usedevelop = True
 deps =
     -r{toxinidir}/requirements.txt

v2v-conversion-host.spec.in

@@ -38,6 +38,10 @@ BuildArch: noarch
 Requires: libcgroup-tools
 Requires: python3
 Requires: %{py3_dist pycurl}
+Requires: %{py3_dist packaging}
+Requires: %{py3_dist pyvmomi}
+Requires: %{py3_dist libnbd}
+Requires: %{py3_dist libvirt}
 
 %description wrapper
 Daemonizing wrapper for virt-v2v.

wrapper/log_parser.py

@@ -34,6 +34,8 @@ class OutputParser(object):
         br' \'?virt_v2v_disk_index=(?P<volume>[0-9]+)/[0-9]+.*'
         br' \'?(?P<uuid>[a-fA-F0-9-]*)\'?$')
     SSH_VMX_GUEST_NAME = re.compile(br'^displayName = "(.*)"$')
+    OVERLAY_PATH_RE = re.compile(
+        br'virt-v2v: Overlay saved as (?P<path>/.*\.qcow2) ')
 
     def __init__(self, duplicate=False):
         # Wait for the log files to appear
@@ -47,6 +49,7 @@ def __init__(self, duplicate=False):
         self._current_disk = None
         self._current_path = None
         self._duplicate = duplicate
+        self._current_overlay_disk = 0
 
     def __del__(self):
         self._log.close()
@@ -81,6 +84,22 @@ def parse_line(self, line):
             STATE.vm_id = vm_id
             logging.info('Created VM with id=%s', vm_id)
 
+        if STATE.pre_copy:
+            # Ovelays to commit in two_phase mode
+            m = self.OVERLAY_PATH_RE.match(line)
+            if m is not None:
+                path = m.group('path').decode('utf-8')
+                disks = STATE.pre_copy.disks
+                if self._current_overlay_disk >= len(disks):
+                    error('Disk list mismatch when getting overlay data')
+                disks[self._current_overlay_disk].overlay = path
+                logging.debug('Attaching overlay path "%s" to disk "%d"',
+                              path, self._current_overlay_disk)
+                self._current_overlay_disk += 1
+
+            # There is nothing else to parse for two-phase conversion
+            return
+
         m = self.COPY_DISK_RE.match(line)
         if m is not None:
             try: