Skip to content

Fix unable to build Pandas with xlc on z/OS #35829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
May 26, 2021
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.3.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -913,6 +913,7 @@ Other
- Bug in :func:`pandas.testing.assert_series_equal`, :func:`pandas.testing.assert_frame_equal`, :func:`pandas.testing.assert_index_equal` and :func:`pandas.testing.assert_extension_array_equal` incorrectly raising when an attribute has an unrecognized NA type (:issue:`39461`)
- Bug in :meth:`DataFrame.equals`, :meth:`Series.equals`, :meth:`Index.equals` with object-dtype containing ``np.datetime64("NaT")`` or ``np.timedelta64("NaT")`` (:issue:`39650`)
- Bug in :func:`pandas.util.show_versions` where console JSON output was not proper JSON (:issue:`39701`)
- Fixed Pandas not being able to compile on z/OS when using `xlc <https://www.ibm.com/products/xl-cpp-compiler-zos>`_ (:issue:`35826`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small personal preference, but I'd rephrase as "Let pandas compile with xlc in z/OS" or similar. I don't think it was broken, and now it's fixed, since that was never supported.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've swapped up the wording for this similar to as you've put it

- Bug in :meth:`DataFrame.convert_dtypes` incorrectly raised ValueError when called on an empty DataFrame (:issue:`40393`)
- Bug in :meth:`DataFrame.clip` not interpreting missing values as no threshold (:issue:`40420`)

Expand Down
12 changes: 12 additions & 0 deletions pandas/_libs/src/headers/cmath
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,18 @@ namespace std {
__inline int isnan(double x) { return _isnan(x); }
__inline int notnan(double x) { return x == x; }
}
#elif defined(__MVS__)
#include <cmath>

#define _signbit signbit
#undef signbit
#undef isnan

namespace std {
__inline int notnan(double x) { return x == x; }
__inline int signbit(double num) { return _signbit(num); }
__inline int isnan(double x) { return isnan(x); }
}
#else
#include <cmath>

Expand Down
24 changes: 22 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ def is_platform_mac():
return sys.platform == "darwin"


def is_platform_zos():
return sys.platform == "zos"


min_cython_ver = "0.29.21" # note: sync with pyproject.toml

try:
Expand Down Expand Up @@ -569,15 +573,31 @@ def srcpath(name=None, suffix=".pyx", subdir="src"):
include = data.get("include", [])
include.append(numpy.get_include())

extra_comp_args = extra_compile_args.copy()
comp_macros = data.get("macros", macros)
undef_macros = []

if is_platform_zos():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why we need this function, why not simply if sys.platform == "zos":?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had added it to be consistent with how the other platforms handled it. I've minimized the change as suggested.

language = data.get("language", None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None is already the default. Also, since you aren't using the language more than in the if, I'd simply write if data.get('language') == 'c++': and make this more compact.

This file is huge, so would prefer to not make this unnecessarily longer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, thanks!

if language == "c++":
compiler = os.environ.get("CXX", "/bin/xlc++")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we do assume that if CXX is not in env, the compiler is xlc++?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On z/OS, xlc/xlc++ is the default compiler that is provided, so most users will have it.

compiler_name = os.path.basename(compiler)

if (compiler_name == "xlc") or (compiler_name == "xlc++"):
Copy link
Member

@datapythonista datapythonista May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels that all these if could be just a single condition:

if sys.platform == "zos" and data.get('language') == 'c++' and os.path.basename(os.environ.get('CXX')) in ('xlc', 'xlc++'):

Does it make sense?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup - I've made this change. Thanks

comp_macros.append(("__s390__", "1"))
extra_comp_args.append("-qlanglvl=extended0x:nolibext")
undef_macros.append("_POSIX_THREADS")

obj = Extension(
f"pandas.{name}",
sources=sources,
depends=data.get("depends", []),
include_dirs=include,
language=data.get("language", "c"),
define_macros=data.get("macros", macros),
extra_compile_args=extra_compile_args,
define_macros=comp_macros,
extra_compile_args=extra_comp_args,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is extra_compile_args being used elsewhere now? Not easy to see in the diff, but just in case you can simply append to extra_compile_args and not make a copy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra_compile_args isn't used elsewhere. I've removed the copy and just appended directly to it now.

extra_link_args=extra_link_args,
undef_macros=undef_macros,
)

extensions.append(obj)
Expand Down