Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes some failing tests on windows #775

Merged
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Change Log


## 2.47.10 (2019-10-26)
[Source](https://github.com/repos/nerdvegas/rez/tree/2.47.10) | [Diff](https://github.com/repos/nerdvegas/rez/compare/2.47.9...2.47.10)

Notes

* corrects failing tests in `rez/tests/test_packages.py` on windows
* adds %SYSTEMROOT% envvar to Python(ActionInterpreter) so `rez/tests/test_context.py` tests pass on windows


## 2.47.9 (2019-10-25)
[Source](https://github.com/repos/nerdvegas/rez/tree/2.47.9) | [Diff](https://github.com/repos/nerdvegas/rez/compare/2.47.8...2.47.9)

Expand Down
55 changes: 55 additions & 0 deletions src/rez/rex.py
Original file line number Diff line number Diff line change
Expand Up @@ -591,6 +591,7 @@ def apply_environ(self):
"interpreter before using it.")

self.target_environ.update(self.manager.environ)
self.target_environ = self.adjust_env_for_platform(self.target_environ)

def get_output(self, style=OutputStyle.file):
self.apply_environ()
Expand Down Expand Up @@ -633,6 +634,7 @@ def error(self, value):
def subprocess(self, args, **subproc_kwargs):
if self.manager:
self.target_environ.update(self.manager.environ)
self.target_environ = self.adjust_env_for_platform(self.target_environ)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would drop the return value, and avoid the unnecessary dict copy. adjust_env_for_platform should just alter self.target_environ in-place, asthe previous code already does

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!


shell_mode = isinstance(args, basestring)
return Popen(args,
Expand Down Expand Up @@ -684,6 +686,59 @@ def get_key_token(self, key):
# here because the API requires it.
return "${%s}" % key

def adjust_env_for_platform(self, env):
""" Make required platform-specific adjustments to env.
"""
if sys.platform.startswith('win'):
env = self._add_systemroot_to_env_win32(env)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, in-place rather than copy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

return env.copy()

def _add_systemroot_to_env_win32(self, env):
""" Sets ``%SYSTEMROOT%`` environment variable, if not present.

Args:
env (dict): desired environment variables

Notes:
on windows, python-3.6 startup fails within an environment
where it ``%PATH%`` includes python3, but ``%SYSTEMROOT%`` is not
present.

for example.

.. code-block:: python

from subprocess import Popen
cmds = ['python', '--version']

# successful
Popen(cmds)
Popen(cmds, env={'PATH': 'C:\\Python-3.6.5',
'SYSTEMROOT': 'C:\Windows'})

# failure
Popen(cmds, env={'PATH': 'C:\\Python-3.6.5'})

#> Fatal Python Error: failed to get random numbers to initialize Python

"""
# 'SYSTEMROOT' only relevant on windows
if not sys.platform.startswith('win'):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this check is already done, and this is a non-public func, so this can be removed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

thank you, I'm never sure how much I should worry about accidents. this seems like a good rule of thumb.

return env
# 'SYSTEMROOT' unecessary unless 'PATH' is set.
if env is None:
return env
# leave SYSTEMROOT alone if set by user
if 'SYSTEMROOT' in env:
return env
# not enough info to set SYSTEMROOT
if 'SYSTEMROOT' not in os.environ:
return env

new_env = env.copy()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as above, in-place rather than copy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

new_env['SYSTEMROOT'] = os.environ['SYSTEMROOT']
return new_env


#===============================================================================
# String manipulation
Expand Down
31 changes: 22 additions & 9 deletions src/rez/tests/test_packages.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@
"""
from rez.packages_ import iter_package_families, iter_packages, get_package, \
create_package, get_developer_package
from rez.package_resources_ import package_release_keys
from rez.package_repository import create_memory_package_repository
from rez.package_py_utils import expand_requirement
from rez.package_repository import create_memory_package_repository
from rez.package_resources_ import package_release_keys
from rez.tests.util import TestBase, TempdirMixin
from rez.utils.formatting import PackageRequest
from rez.utils.platform_ import platform_
from rez.utils.sourcecode import SourceCode
import unittest
from rez.vendor.version.version import Version
Expand Down Expand Up @@ -119,21 +120,27 @@ def test_2(self):

def test_3(self):
"""check package contents."""
def _format_platformpath(path):
""" `FileSystemPackageRepository` lower-cases paths for case-insensitive filesystems.
"""
if platform_.has_case_sensitive_filesystem:
return path
return path.lower()

# a py-based package
package = get_package("versioned", "3.0")
expected_data = dict(
name="versioned",
version=Version("3.0"),
base=os.path.join(self.py_packages_path, "versioned", "3.0"),
base=_format_platformpath(os.path.join(self.py_packages_path, "versioned", "3.0")),
commands=SourceCode('env.PATH.append("{root}/bin")'))
data = package.validated_data()
self.assertDictEqual(data, expected_data)

# a yaml-based package
package = get_package("versioned", "2.0")
expected_uri = os.path.join(self.yaml_packages_path,
"versioned", "2.0", "package.yaml")
expected_uri = _format_platformpath(os.path.join(self.yaml_packages_path,
"versioned", "2.0", "package.yaml"))
self.assertEqual(package.uri, expected_uri)

# a py-based package with late binding attribute functions
Expand All @@ -142,7 +149,7 @@ def test_3(self):

# a 'combined' type package
package = get_package("multi", "1.0")
expected_uri = os.path.join(self.yaml_packages_path, "multi.yaml<1.0>")
expected_uri = _format_platformpath(os.path.join(self.yaml_packages_path, "multi.yaml<1.0>"))
self.assertEqual(package.uri, expected_uri)
expected_data = dict(
name="multi",
Expand All @@ -153,7 +160,7 @@ def test_3(self):

# a 'combined' type package, with version overrides
package = get_package("multi", "1.1")
expected_uri = os.path.join(self.yaml_packages_path, "multi.yaml<1.1>")
expected_uri = _format_platformpath(os.path.join(self.yaml_packages_path, "multi.yaml<1.1>"))
self.assertEqual(package.uri, expected_uri)
expected_data = dict(
name="multi",
Expand All @@ -164,7 +171,7 @@ def test_3(self):

# check that visibility of 'combined' packages is correct
package = get_package("multi", "2.0")
expected_uri = os.path.join(self.py_packages_path, "multi.py<2.0>")
expected_uri = _format_platformpath(os.path.join(self.py_packages_path, "multi.py<2.0>"))
self.assertEqual(package.uri, expected_uri)

def test_4(self):
Expand Down Expand Up @@ -309,11 +316,17 @@ def preprocess(this, data):

def test_6(self):
"""test variant iteration."""
# ensure that differing case doesn't get interpreted as different repos
# on case-insensitive platforms (eg windows)
base = os.path.join(self.py_packages_path, "variants_py", "2.0")
if not platform_.has_case_sensitive_filesystem:
base = base.lower()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should move _format_platformpath and reuse here instead

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this sort of code already exists in another place, see https://github.com/nerdvegas/rez/blob/master/src/rezplugins/package_repository/filesystem.py#L473.

It would be good to add a utility function to rez.utils.filesystem, something like so:

def canonical_path(path, platform=None):
    if platform is None:
        from rez.utils.platform_ import platform_
        platform = platform_

    path = os.path.realpath(path)
    if not platform.has_case_sensitive_filesystem:
        path = path.lower()
    return path

I think this may also fix a potential bug currently, where two package repositories could be the same, but if one uses symlinks to the same path on disk, rez will think they are two different repos.

To give some context: The lower() is there to fix a previous bug that was caused by differing cases of the same path, on windows (non-case-sensitive), which then led rez to believe that there were two different packages that were in fact the same package ('foo' vs 'Foo' for eg).

At least if there's just the one utility function, this can be documented in one place.

Copy link
Contributor Author

@willjp willjp Oct 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I agree with this idea.

The FileSystemPackageRepository that you provided a link to is where I copied the logic from, and was the source of the test-failure under windows.

I wrote canonical_path in addition to tests for it.

There are some frustrating pain points here for testing however.

# python-3.6.5, windows-10
ntpath.realpath('/a/b/c')      #>> 'C:\\a\\b\\c'
posixpath.realpath('C:/a/b/c') #>> 'C:\\Users\\willjp/C:/a/b/c'

# python-3.7.4, windows-10
ntpath.realpath('/a/b/c')  #>> 'C:\\a\\b\\c'
posixpath.realpath('C:/a/b/c')  #>> 'C:\\Users\willjp/C:/a/b/c'
# python-3.7.4, linux
ntpath.realpath('/a/b/c')     #>> '\\a\\b\\c'
posixpath.realpath('/a/b/c')  #>> '/a/b/c'

This change in python is not represented in the docs, so I am uncertain when they were introduced: https://docs.python.org/3/library/os.path.html?highlight=realpath#os.path.realpath


expected_data = dict(
name="variants_py",
version=Version("2.0"),
description="package with variants",
base=os.path.join(self.py_packages_path, "variants_py", "2.0"),
base=base,
requires=[PackageRequest("python-2.7")],
commands=SourceCode('env.PATH.append("{root}/bin")'))

Expand Down
2 changes: 1 addition & 1 deletion src/rez/utils/_version.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@


# Update this value to version up Rez. Do not place anything else in this file.
_rez_version = "2.47.9"
_rez_version = "2.47.10"


# Copyright 2013-2016 Allan Johns.
Expand Down