Skip to content

Commit

Permalink
Additional compilers (#349)
Browse files Browse the repository at this point in the history
* Moved OBJECT_ARCHIVES from constants to ArtefactSet.

* Moved PRAGMAD_C from constants to ArtefactSet.

* Turned 'all_source' into an enum.

* Allow integer as revision.

* Fixed flake8 error.

* Removed specific functions to add/get fortran source files etc.

* Removed non-existing and unneccessary collections.

* Try to fix all run_configs.

* Fixed rebase issues.

* Added replace functionality to ArtefactStore, updated test_artefacts to cover all lines in that file.

* Started to replace artefacts when files are pre-processed.

* Removed linker argument from linking step in all examples.

* Try to get jules to link.

* Fixed build_jules.

* Fixed other issues raised in reviews.

* Try to get jules to link.

* Fixed other issues raised in reviews.

* Simplify handling of X90 files by replacing the X90 with x90, meaning only one artefact set is involved when running PSyclone.

* Make OBJECT_ARCHIVES also a dict, migrate more code to replace/add files to the default build artefact collections.

* Fixed some examples.

* Fix flake8 error.

* Fixed failing tests.

* Support empty comments.

* Fix preprocessor to not unnecessary remove and add files that are already in the output directory.

* Allow find_soure_files to be called more than once by adding files (not replacing artefact).

* Updated lfric_common so that files created by configurator are written in build (not source).

* Use c_build_files instead of pragmad_c.

* Removed unnecessary str.

* Documented the new artefact set handling.

* Fixed typo.

* Make the PSyclone API configurable.

* Fixed formatting of documentation, properly used ArtefactSet names.

* Support .f and .F Fortran files.

* Removed setter for tool.is_available, which was only used for testing.

* #3 Fix documentation and coding style issues from review.

* Renamed Categories into Category.

* Minor coding style cleanup.

* Removed more unnecessary ().

* Re-added (invalid) grab_pre_build call.

* Fixed typo.

* Renamed set_default_vendor to set_default_compiler_suite.

* Renamed VendorTool to CompilerSuiteTool.

* Also accept a Path as exec_name specification for a tool.

* Move the check_available function into the base class.

* Fixed some types and documentation.

* Fix typing error.

* Added explanation for meta-compiler.

* Improved error handling and documentation.

* Replace mpiifort with mpifort to be a tiny bit more portable.

* Use classes to group tests for git/svn/fcm together.

* Fixed issue in get_transformation script, and moved script into lfric_common to remove code duplication.

* Code improvement as suggested by review.

* Fixed run config

* Added reference to ticket.

* Updated type information.

* More typing fixes.

* Fixed typing warnings.

* As requested by reviewer removed is_working_copy functionality.

* Issue a warning (which can be silenced) when a tool in a toolbox is replaced.

* Fixed flake8.

* Fixed flake8.

* Fixed failing test.

* Addressed issues raised in review.

* Removed now unnecessary operations.

* Updated some type information.

* Fixed all references to APIs to be consistent with PSyclone 2.5.

* Added api to the checksum computation.

* Fixed type information.

* Added test to verify that changing the api changes the checksum.

* Make compiler version a tuple of integers

* Update some tests to use tuple versions

* Explicitly test handling of bad version format

* Fix formatting

* Tidying up

* Make compiler raise an error for any invalid version string

Assume these compilers don't need to be hashed.
Saves dealing with empty tuples.

* Check compiler version string for compiler name

* Fix formatting

* Add compiler.get_version_string() method

Includes other cleanup from PR comments

* Add mpi and openmp settings to BuildConfig, made compiler MPI aware.

* Looks like the circular dependency has been fixed.

* Revert "Looks like the circular dependency has been fixed." ...
while it works with the tests, a real application still triggered it.

This reverts commit 150dc37.

* Don't even try to find a C compiler if no C files are to be compiled.

* Updated gitignore to ignore (recently renamed) documentation.

* Fixed failing test.

* Return from compile Fortran early if there are no files to compiles. Fixed coding style.

* Add MPI enables wrapper for intel and gnu compiler.

* Fixed test.

* Automatically add openmp flag to compiler and linker based on BuildConfig.

* Removed enforcement of keyword parameters, which is not supported in python 3.7.

* Fixed failing test.

* Support more than one tool of a given suite by sorting them.

* Use different version checkout for each compiler vendor with mixins

* Refactoring, remove unittest compiler class

* Fix some mypy errors

* Use 'Union' type hint to fix build checks

* Added option to add flags to a tool.

* Introduce proper compiler wrapper, used this to implement properly wrapper MPI compiler.

* Fixed typo in types.

* Return run_version_command to base Compiler class

Provides default version command that can be overridden for other compilers.
Also fix some incorrect tests
Other tidying

* Add a missing type hint

* Added (somewhat stupid) 'test' to reach 100% coverage of PSyclone tool.

* Simplified MPI support in wrapper.

* More compiler wrapper coverage.

* Removed duplicated function.

* Removed debug print.

* Removed permanently changing compiler attributes, which can cause test failures later.

* More test for C compiler wrapper.

* More work on compiler wrapper tests.

* Fixed version and availability handling, added missing tests for 100% coverage.

* Fixed typing error.

* Try to fix python 3.7.

* Tried to fix failing tests.

* Remove inheritance from mixins and use protocol

* Simplify compiler inheritance

Mixins have static methods with unique names,
overrides only happen in concrete classes

* Updated wrapper and tests to handle error raised in get_version.

* Simplified regular expressions (now tests cover detection of version numbers with only a major version).

* Test for missing mixin.

* Use the parsing mixing from the compiler in a compiler wrapper.

* Use setattr instead of assignment to make mypy happy.

* Simplify usage of compiler-specific parsing mixins.

* Minor code cleanup.

* Updated documentation.

* Simplify usage of compiler-specific parsing mixins.

* Test for missing mixin.

* Fixed test.

* Added missing openmp_flag property to compiler wrapper.

* Don't use isinstance for consistency check, which does not work for CompilerWrappers.

* Fixed isinstance test for C compilation which doesn't work with a CompilerWrapper.

* Use a linker's compiler to determine MPI support. Removed mpi property from CompilerSuite.

* Added more tests for invalid version numbers.

* Added more test cases for invalid version number, improved regex to work as expected.

* Fixed typo in test.

* Fixed flake/mypy errors.

* Combine wrapper flags with flags from wrapped compiler.

* Made mypy happy.

* Fixed test.

* Split tests into smaller individual ones, fixed missing asssert in test.

* Parameterised compiler version tests to also test wrapper.

* Added missing MPI parameter when getting the compiler.

* Fixed comments.

* Order parameters to be in same order for various compiler classes.

* Remove stray character

* Added getter for wrapped compiler.

* Fixed small error that would prevent nested compiler wrappers from being used.

* Added a cast to make mypy happy.

* Add simple getter for linker library flags

* Add getter for linker flags by library

* Fix formatting

* Add optional libs argument to link function

* Reorder and clean up linker tests

* Make sure `Linker.link()` raises for unknown lib

* Add missing type

* Fix typing error

* Add 'libs' argument to link_exe function

* Try to add documentation for the linker libs feature

* Use correct list type in link_exe hint

* Add silent replace option to linker.add_lib_flags

* Fixed spelling mistake in option.

* Clarified documentation.

* Removed unnecessary functions in CompilerWrapper.

* Fixed failing test triggered by executing them in specific order (tools then steps)

* Fixed line lengths.

* Add tests for linker LDFLAG

* Add pre- and post- lib flags to link function

* Fix syntax in built-in lib flags

* Remove netcdf as a built-in linker library

Bash-style substitution is not currently handled

* Configure pre- and post-lib flags on the Linker object

Previously they were passed into the Linker.link() function

* Use more realistic linker lib flags

* Formatting fix

* Removed mixing, use a simple regex instead.

* Added support for ifx/icx compiler as intel-llvm class.

* Added support for nvidia compiler.

* Add preliminary support for Cray compiler.

* Added Cray compiler wrapper ftn and cc.

* Made mpi and openmp default to False in the BuildConfig constructor.

* Removed white space.

* Follow a more consistent naming scheme for crays, even though the native compiler names are longer (crayftn-cray, craycc-cray).

* Changed names again.

* Support compilers that do not support OpenMP.

* Added documentation for openmp parameter.

* Renamed cray compiler wrapper to be CrayCcWrapper and CrayFtnWrapper, to avoid confusion with Craycc.

* Fixed incorrect name in comments.

---------

Co-authored-by: jasonjunweilyu <[email protected]>
Co-authored-by: Luke Hoffmann <[email protected]>
Co-authored-by: Luke Hoffmann <[email protected]>
  • Loading branch information
4 people authored Nov 12, 2024
1 parent ae04f0b commit 7a2eb59
Show file tree
Hide file tree
Showing 10 changed files with 624 additions and 141 deletions.
19 changes: 13 additions & 6 deletions source/fab/tools/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,11 @@

from fab.tools.ar import Ar
from fab.tools.category import Category
from fab.tools.compiler import (CCompiler, Compiler, FortranCompiler, Gcc,
Gfortran, GnuVersionHandling, Icc, Ifort,
IntelVersionHandling)
from fab.tools.compiler_wrapper import CompilerWrapper, Mpicc, Mpif90
from fab.tools.compiler import (CCompiler, Compiler, Craycc, Crayftn,
FortranCompiler, Gcc, Gfortran, Icc,
Icx, Ifort, Ifx, Nvc, Nvfortran)
from fab.tools.compiler_wrapper import (CompilerWrapper, CrayCcWrapper,
CrayFtnWrapper, Mpicc, Mpif90)
from fab.tools.flags import Flags
from fab.tools.linker import Linker
from fab.tools.psyclone import Psyclone
Expand All @@ -32,20 +33,26 @@
"CompilerWrapper",
"Cpp",
"CppFortran",
"Craycc",
"CrayCcWrapper",
"Crayftn",
"CrayFtnWrapper",
"Fcm",
"Flags",
"FortranCompiler",
"Fpp",
"Gcc",
"Gfortran",
"Git",
"GnuVersionHandling",
"Icc",
"Icx",
"Ifort",
"IntelVersionHandling",
"Ifx",
"Linker",
"Mpif90",
"Mpicc",
"Nvc",
"Nvfortran",
"Preprocessor",
"Psyclone",
"Rsync",
Expand Down
272 changes: 182 additions & 90 deletions source/fab/tools/compiler.py

Large diffs are not rendered by default.

31 changes: 27 additions & 4 deletions source/fab/tools/compiler_wrapper.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,9 @@ def __init__(self, name: str, exec_name: str,
name=name, exec_name=exec_name,
category=self._compiler.category,
suite=self._compiler.suite,
version_regex=self._compiler._version_regex,
mpi=mpi,
availability_option=self._compiler.availability_option)
# We need to have the right version to parse the version output
# So we set this function based on the function that the
# wrapped compiler uses:
setattr(self, "parse_version_output", compiler.parse_version_output)

def __str__(self):
return f"{type(self).__name__}({self._compiler.name})"
Expand Down Expand Up @@ -196,3 +193,29 @@ class Mpicc(CompilerWrapper):
def __init__(self, compiler: Compiler):
super().__init__(name=f"mpicc-{compiler.name}",
exec_name="mpicc", compiler=compiler, mpi=True)


# ============================================================================
class CrayFtnWrapper(CompilerWrapper):
'''Class for the Cray Fortran compiler wrapper. We add 'wrapper' to the
class name to make this class distinct from the Crayftn compiler class.
:param compiler: the compiler that the ftn wrapper will use.
'''

def __init__(self, compiler: Compiler):
super().__init__(name=f"crayftn-{compiler.name}",
exec_name="ftn", compiler=compiler, mpi=True)


# ============================================================================
class CrayCcWrapper(CompilerWrapper):
'''Class for the Cray C compiler wrapper. We add 'wrapper' to the class
name to make this class distinct from the Craycc compiler class
:param compiler: the compiler that the mpicc wrapper will use.
'''

def __init__(self, compiler: Compiler):
super().__init__(name=f"craycc-{compiler.name}",
exec_name="cc", compiler=compiler, mpi=True)
33 changes: 24 additions & 9 deletions source/fab/tools/tool_repository.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,13 @@
from fab.tools.tool import Tool
from fab.tools.category import Category
from fab.tools.compiler import Compiler
from fab.tools.compiler_wrapper import (CrayCcWrapper, CrayFtnWrapper,
Mpif90, Mpicc)
from fab.tools.linker import Linker
from fab.tools.versioning import Fcm, Git, Subversion
from fab.tools import (Ar, Cpp, CppFortran, Craycc, Crayftn,
Gcc, Gfortran, Icc, Icx, Ifort, Ifx,
Nvc, Nvfortran, Psyclone, Rsync)


class ToolRepository(dict):
Expand Down Expand Up @@ -57,26 +62,36 @@ def __init__(self):

# Add the FAB default tools:
# TODO: sort the defaults so that they actually work (since not all
# tools FAB knows about are available). For now, disable Fpp:
# We get circular dependencies if imported at top of the file:
# pylint: disable=import-outside-toplevel
from fab.tools import (Ar, Cpp, CppFortran, Gcc, Gfortran,
Icc, Ifort, Psyclone, Rsync)

for cls in [Gcc, Icc, Gfortran, Ifort, Cpp, CppFortran,
Fcm, Git, Subversion, Ar, Psyclone, Rsync]:
# tools FAB knows about are available). For now, disable Fpp (by not
# adding it). IF someone actually uses it it can added.
for cls in [Craycc, Crayftn,
Gcc, Gfortran,
Icc, Icx, Ifort, Ifx,
Nvc, Nvfortran,
Cpp, CppFortran,
Ar, Fcm, Git, Psyclone, Rsync, Subversion]:
self.add_tool(cls())

from fab.tools.compiler_wrapper import Mpif90, Mpicc
# Now create the potential mpif90 and Cray ftn wrapper
all_fc = self[Category.FORTRAN_COMPILER][:]
for fc in all_fc:
mpif90 = Mpif90(fc)
self.add_tool(mpif90)
# I assume cray has (besides cray) only support for Intel and GNU
if fc.name in ["gfortran", "ifort"]:
crayftn = CrayFtnWrapper(fc)
print("NEW NAME", crayftn, crayftn.name)
self.add_tool(crayftn)

# Now create the potential mpicc and Cray cc wrapper
all_cc = self[Category.C_COMPILER][:]
for cc in all_cc:
mpicc = Mpicc(cc)
self.add_tool(mpicc)
# I assume cray has (besides cray) only support for Intel and GNU
if cc.name in ["gcc", "icc"]:
craycc = CrayCcWrapper(cc)
self.add_tool(craycc)

def add_tool(self, tool: Tool):
'''Creates an instance of the specified class and adds it
Expand Down
4 changes: 3 additions & 1 deletion tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@
@pytest.fixture(name="mock_c_compiler")
def fixture_mock_c_compiler():
'''Provides a mock C-compiler.'''
mock_compiler = CCompiler("mock_c_compiler", "mock_exec", "suite")
mock_compiler = CCompiler("mock_c_compiler", "mock_exec", "suite",
version_regex="something")
mock_compiler.run = mock.Mock()
mock_compiler._version = (1, 2, 3)
mock_compiler._name = "mock_c_compiler"
Expand All @@ -32,6 +33,7 @@ def fixture_mock_fortran_compiler():
'''Provides a mock Fortran-compiler.'''
mock_compiler = FortranCompiler("mock_fortran_compiler", "mock_exec",
"suite", module_folder_flag="",
version_regex="something",
syntax_only_flag=None, compile_flag=None,
output_flag=None, openmp_flag=None)
mock_compiler.run = mock.Mock()
Expand Down
1 change: 0 additions & 1 deletion tests/unit_tests/steps/test_archive_objects.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,6 @@ def test_for_library(self):
def test_incorrect_tool(self, tool_box):
'''Test that an incorrect archive tool is detected
'''

config = BuildConfig('proj', tool_box)
cc = tool_box.get_tool(Category.C_COMPILER, config.mpi, config.openmp)
# And set its category to be AR
Expand Down
Loading

0 comments on commit 7a2eb59

Please sign in to comment.