Cross complilation of bundle libevent for windows uses wrong compiler

[dirk-zimoch]

Bug Description

I am cross compiling on RHEL8 for windows-x64 using the MSVC 2019 compiler in wine.

However, when building the bundled libevent using make -C pvxs/bundle libevent.windows-x64, the build uses the host gcc instead of the Microsoft compiler in wine as configured in EPICS base:

$ make -C pvxs/bundle libevent.windows-x64
make: Entering directory '/home/zimoch/epics/epics-base-7.0/modules/pvxs/bundle'
mkdir -p /home/zimoch/epics/epics-base-7.0/modules/pvxs/bundle/O.windows-x64
make -C /home/zimoch/epics/epics-base-7.0/modules/pvxs/bundle/O.windows-x64 -f /home/zimoch/epics/epics-base-7.0/modules/pvxs/bundle/Makefile TOP=../.. T_A=windows-x64 libevent.windows-x64
make[1]: Entering directory '/home/zimoch/epics/epics-base-7.0/modules/pvxs/bundle/O.windows-x64'
cmake  -DCMAKE_POLICY_VERSION_MINIMUM=3.10 -DCMAKE_INSTALL_PREFIX:PATH="/home/zimoch/epics/epics-base-7.0/bundle/usr/windows-x64" -DEVENT__DISABLE_OPENSSL=ON -DEVENT__DISABLE_MBEDTLS=ON -DEVENT__DISABLE_REGRESS=ON -DEVENT__DISABLE_SAMPLES=ON -DEVENT__DISABLE_TESTS=ON -DEVENT__DISABLE_BENCHMARK=ON -DCMAKE_MACOSX_RPATH=ON -DCMAKE_SKIP_INSTALL_RPATH=OFF -DCMAKE_MODULE_PATH:DIR="/home/zimoch/epics/epics-base-7.0/modules/pvxs/bundle/cmake" -DCMAKE_MAKE_PROGRAM="make" -DCMAKE_BUILD_TYPE=RELEASE -DEVENT__LIBRARY_TYPE=SHARED  ../libevent
-- The C compiler identification is GNU 8.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Git: /usr/bin/git (found version "2.43.5") 
[...]

The produced include files are unusable for the cross build, leading to the following compiler error:

cl -EHsc -GR           -DUSE_TYPED_RSET -wd4800 -wd4275 -DPVXS_API_BUILDING -DPVXS_ENABLE_EXPERT_API -DNOMINMAX -D_WIN32_WINNT=_WIN32_WINNT_VISTA     -nologo -FC -D__STDC__=0 -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE    -Ox -GL -Oy-   -W3 -w44355 -w44344 -w44251 -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING          -MD -DEPICS_BUILD_DLL -DEPICS_CALL_DLL -TP   -I. -I../O.Common -I. -I. -I../os/WIN32 -I../os/default -I.. -I/home/zimoch/epics/epics-base-7.0/include/compiler/msvc -I/home/zimoch/epics/epics-base-7.0/include/os/WIN32 -I/home/zimoch/epics/epics-base-7.0/include -I/home/zimoch/epics/epics-base-7.0/include/compiler/msvc -I/home/zimoch/epics/epics-base-7.0/include/os/WIN32 -I/home/zimoch/epics/epics-base-7.0/include        -I/home/zimoch/epics/epics-base-7.0/bundle/usr/windows-x64/include -c ../describe.cpp
describe.cpp
Z:\home\zimoch\epics\epics-base-7.0\bundle\usr\windows-x64\include\event2\event.h(198): fatal error C1083: Cannot open include file: 'sys/time.h': No such file or directory

Also, the libraries installed in bundle/usr/windows-x64/lib/ are *.so files, clearly for Linux and not for Windows.

Expected behavior

The bundle build should use the compiler as defined in EPICS base for the cross architecture in use, not the host compiler.

Information (please complete the following):

• PVXS Version or Git commit ID: commit e8f6b5c00
• EPICS Base Version: 7.0.9
• libevent Version: from bundle
• EPICS_HOST_ARCH: RHEL8-x86_64 (a.k.a. linux-x86_64)
• EPICS Target Arch. : windows-x64
• Host OS: RHEL8
• Target OS: Windows
• Compiler version: Should be "Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30148 for x64" but isn't.

Additional context
My CONFIG.Linux.windows-x64, included by CONFIG.RHEL8-x86_64.windows-x64:

include $(CONFIG)/os/CONFIG.windows-x64.windows-x64

VALID_BUILDS = Ioc Command

MSVC_VERSION ?= 2019
PATH := /opt/wine-msvc-$(MSVC_VERSION)/bin/x64:$(PATH)
export WINE ?= wine64
export WINEPREFIX = $(HOME)/.wine-$(EPICS_HOST_ARCH)
export WINEDEBUG := err-all,warn-all,fixme-all$(WINEDEBUG:%=,%)
export WINEDLLOVERRIDES := mscoree,mshtml=$(WINEDLLOVERRIDES:%=,%)
export WINEPATH := $(WINEPATH:%=%;)$(abspath $(INSTALL_BIN))

We have installed MSVC in /opt/wine-msvc-$(MSVC_VERSION) and have wrapper scripts calling wine for the compiler cl, the linker, etc.

[dirk-zimoch]

I checked if the existing bundle/libevent/CMakeLists.txt contributes to the problem by removing it, but that does not help. Still the build produces host files, not cross files. And without it, I can no longer cross compile the bundled libevent for RTEMS.

[mdavidsaver]

So far with bundle/ I handled cross builds on Linux as special cases for RTEMS (should cover all BSPs) and windows-x64-mingw. So windows-x64 is not (yet) supported.

What "support" means in the context of cross building with cmake is a toolchain file referenced by $(CMAKE_TOOLCHAIN_$(T_A)).

pvxs/bundle/Makefile

Lines 58 to 61 in e32a752

	ifneq (,$(filter linux-%,$(EPICS_HOST_ARCH)))
	# cross mingw hosted on linux
	CMAKE_TOOLCHAIN_windows-x64-mingw ?= x86_64-w64-mingw32
	endif

The mingw version is:

pvxs/bundle/cmake/x86_64-w64-mingw32.cmake

Lines 1 to 9 in e32a752

	set(CMAKE_SYSTEM_NAME Windows)
	set(CMAKE_SYSTEM_PROCESSOR AMD64)
	set(CMAKE_C_COMPILER x86_64-w64-mingw32-gcc)
	set(CMAKE_CXX_COMPILER x86_64-w64-mingw32-g++)
	set(CMAKE_RC_COMPILER x86_64-w64-mingw32-windres)
	set(CMAKE_FIND_ROOT_PATH /usr/x86_64-w64-mingw32 )
	set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
	set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
	set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

So one simple way to start would be to copy this file (toolchain file names are not important) and set CMAKE_TOOLCHAIN_windows-x64 ?= in bundle/Makefile just after the line quoted above. Then change the executable names and target root path.

[dirk-zimoch]

I handled cross builds on Linux as special cases

Long term, that is not maintainable. I do not think that it is a good idea to mix cmake into the EPICS build system. Consder using the EPICS build system for the bundle too, so that one does not need to have "special cases" for everything.
Also, the assumption that all Linux host architecture names match linux-% restricts the use of other conventions. For example, we (historically) had SL6_x86, RHEL7-x86_64, RHEL8-x86_64, RHEL9-x86_64 as host architectures, which allows us to install them all into the same NFS location. IMHO, if it works for base, it should work for pvxs too.

[mdavidsaver]

Long term, that is not maintainable.

I agree.

Consder using the EPICS build system for the bundle

I believe that a plain Makefile build is infeasible. libevent makes extensive use of autoconf style probing (cf. event-config.h.cmake). This is not idle speculation on my part. As part of building the pvxslibs python package, setup.py in PVXS contains a replacement build for libevent with probing using setuptools-dso. The fragility of that code is a major reason why the bundled version of libevent is not updated more often.

What I think would be feasible is more comprehensive generation of cmake toolchain files.

Of perhaps it would be better to give up on the idea of bundle/? I have already decided that openssl will not be included (I don't want to chase their CVEs). And I do not see a straight foward solution to the bootstrapping problems when building PVXS as a sub-module of Base which were pointed out during the recent core developer meeting.

match linux-%

Thinking about it now, a more appropriate test would be ifeq ($(BUILD_CLASS),CROSS) ...

[dirk-zimoch]

Can we assume that all Windows systems can share the same config, as well as all Linux systems, and all Macs? In that case, you could provide pre-built config files per OS_CLASS. At least for the not too exotic ones. The fragility is probably mainly due to changes in libevent, but as long as you keep the version in the bundle constant, it should not break.

[dirk-zimoch]

I tried it with a toolchain file that I used for another project. Unfortunately it fails when building sha1.c.obj:

[ 47%] Building C object CMakeFiles/event_extra_shared.dir/sha1.c.obj
sha1.c
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\stdio.h(1788): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\stdio.h(1964): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(104): error C2143: syntax error: missing ')' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(104): error C2143: syntax error: missing '{' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(104): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(104): error C2059: syntax error: ')'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(123): error C2143: syntax error: missing ')' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(123): error C2143: syntax error: missing '{' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(123): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(123): error C2059: syntax error: ')'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(386): error C2143: syntax error: missing ')' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(386): error C2143: syntax error: missing '{' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(386): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(386): error C2059: syntax error: ')'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(422): error C2143: syntax error: missing ')' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(422): error C2143: syntax error: missing '{' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(422): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\corecrt_wstring.h(422): error C2059: syntax error: ')'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(95): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(134): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(233): error C2143: syntax error: missing ')' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(233): error C2143: syntax error: missing '{' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(233): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(233): error C2059: syntax error: ')'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(427): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(455): error C2143: syntax error: missing ')' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(455): error C2143: syntax error: missing '{' before 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(455): error C2059: syntax error: 'constant'
z:\opt\wine-msvc-2019\kits\10\include\10.0.19041.0\ucrt\string.h(455): error C2059: syntax error: ')'
z:/home/zimoch/epics/epics-base-7.0/modules/pvxs/bundle/libevent/sha1.c(47): fatal error C1189: #error:  "Endianness not defined!"
make[4]: *** [CMakeFiles/event_extra_shared.dir/build.make:132: CMakeFiles/event_extra_shared.dir/sha1.c.obj] Error 2

Maybe I will try on Windows natively...

[dirk-zimoch]

Found the problem. As this is a cross build, cmake did not assume its native byte order. And (as configured) did not bother to run the test for it. That caused a wrong compiler flag -D=1 (define a macro with an empty name, neither LITTLE_ENDIAN nor BIG_ENDIAN, and value 1) which then confused the standard include files. I had to set(CMAKE_C_BYTE_ORDER LITTLE_ENDIAN) in my toolchain file.
Now libevent compiles and I can build pvxs.

[mdavidsaver]

... my toolchain file.

Can you attached this file?

[mdavidsaver]

Can we assume that all Windows systems can share the same config, as well as all Linux systems, and all Macs?

Yes, I think these three OSs should have similar toolchain files. RTEMS will be the outlier as this is my doing (see bundle/cmake/Platform/).

... As this is a cross build, cmake did not assume its native byte order ...

This looks to me like a logic hole in the CMakeLists.txt for libevent, or maybe a documentation bug with [cmake](https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_BYTE_ORDER.html_

Which version of cmake did you use?

[dirk-zimoch]

This is my file. A colleague made it for me for another project. I do not understand the reasoning for anything in it. I just added the BYTE_ORDER setting.

set(CMAKE_SYSTEM_NAME Windows)
set(BUILD_SHARED_LIBS ON)
set(LIBTYPE SHARED)
set(CMAKE_CXX_COMPILER_ID MSVC)
set(CMAKE_CXX_PLATFORM_ID Windows)
set(CMAKE_C_COMPILER_ID MSVC)
set(BASE /opt/wine-msvc-2019)
set(CMAKE_C_COMPILER ${BASE}/bin/x64/cl)
set(CMAKE_CXX_COMPILER ${BASE}/bin/x64/cl)
set(CMAKE_LINKER ${BASE}/bin/x64/link)

set(CMAKE_C_COMPILER_FORCED YES)
set(CMAKE_CXX_COMPILER_FORCED YES)
set(CMAKE_C_BYTE_ORDER LITTLE_ENDIAN)

set(CMAKE_CXX_LINK_EXECUTABLE "${BASE}/bin/x64/link <FLAGS> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS>  -o <TARGET> <LINK_LIBRARIES>")
set(CMAKE_BUILD_TYPE Release CACHE STRING "Debug|Release|RelWithDebInfo|MinSizeRel")
set(CMAKE_CROSS_COMPILING ON) # Workaround for http://www.cmake.org/Bug/view.php?id=14075

The set(BASE /opt/wine-msvc-2019) is of course specific to our installation.

[dirk-zimoch]

My cmake version is 3.26.5

[dirk-zimoch]

The "logic hole" is in libevent/CMakeLists.txt lines 988-996. It does not consider cross compilation.

[mdavidsaver]

The "logic hole" is in libevent/CMakeLists.txt lines 988-996. It does not consider cross compilation.

This endianness test? How does it fail?

if("${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION}" LESS 3.20)
  include(TestBigEndian)
  TEST_BIG_ENDIAN(IS_BIG_ENDIAN)
  if(IS_BIG_ENDIAN)
    set(CMAKE_C_BYTE_ORDER BIG_ENDIAN)
  else()
    set(CMAKE_C_BYTE_ORDER LITTLE_ENDIAN)
  endif()
endif()

My cmake version is 3.26.5

The condition above should not match.

I think the automatic derivation of CMAKE_C_BYTE_ORDER should use only the target compiler.

[mdavidsaver]

This is my file. A colleague made it for me for another project.
...
set(CMAKE_CROSS_COMPILING ON) # Workaround for http://www.cmake.org/Bug/view.php?id=14075

The bug referenced was reported for cmake 2.8.10.2 back in 2013, and fixed long ago. Support for cross-compiling in cmake has improved significantly since. You might start again with bundle/cmake/x86_64-w64-mingw32.cmake as an example, just fill in the plain executable names and rely on vcvarsall.bat to prepare the environment.