libclang does not get the header search correct

[llvmbot]

Bugzilla Link	17776
Version	trunk
OS	Linux
Reporter	LLVM Bugzilla Contributor

Extended Description

For instance, if we run one of the included python tests using the raw build directory from cmake+ninja:

$ LD_LIBRARY_PATH=/scratchl/jgg/llvm/build/lib/ PYTHONPATH=/scratchl/jgg/llvm/tools/clang/bindings/python
python /scratchl/jgg/llvm/tools/clang/bindings/python/examples/cindex/cindex-dump.py t.c -v

Here! ../tools/clang/lib/Driver/Driver.cpp:68 Dir=
clang version 3.4
Target: x86_64-unknown-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/4.7
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/4.7.3
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7.3
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7
Here! ../tools/clang/lib/Frontend/ASTUnit.cpp:2038 ResourceFilesPath=/scratchl/jgg/llvm/build/lib/clang/3.4
ignoring nonexistent directory "../lib/clang/3.4/include"

The 'ignoring nonexistent directory' line shows the wrong path, it should be prefixed with the bin dir. Eg when running clang natively that doesn't print and the search path properly includes:

/scratchl/jgg/llvm/build/bin/../lib/clang/3.4/include

Which is missing from the libclang version.

This problem causes parses through libclang to fail, typically with cannot find stddef.h warnings.

I've edited the source (using GIT as of today) and added some Here! prints to show some of the flow.

Analysis:

libclang uses the code in CIndexer::getClangResourcesPath() to locate the installation relative to the shared library location. This code is working fine, the 2nd Here! above prints the result of that function as it shows up in ASTUnit::LoadFromCommandLine as ResourceFilesPath

However, the value ResourceFilesPath is no longer used on Linux because InitHeaderSearch::AddDefaultIncludePaths exits early. On Linux the include files are found via the Driver constructor's ClangExecutable argument, which is wired to 'clang' when libclang is being used.

Which is because of this code in clang::createInvocationFromCommandLine

// FIXME: We shouldn't have to pass in the path info.
driver::Driver TheDriver("clang", llvm::sys::getDefaultTargetTriple(),
"a.out", *Diags);

The driver should probably be created with something like ResourceFilesPath + "../bin/clang" until the FIXME is solved properly..

[jnikula]

@jgunthorpe is this your bug originally? Was there any resolution? Do you have a workaround?

I'm struggling with libclang header search paths. I can parse clang -E -Wp,-v - or clang -x c++ -E -Wp,-v - output and pass the paths with -I option to libclang (more specifically its Python bindings), and it has worked for me on native Linux and GitHub workflows. However, nothing seems to work with GitHub workflows on the macos-latest image. I've also tried the -stdlib=libstdc++ and -stdlib=libc++ options with -x c++.

I keep hitting the error messages referenced in https://reviews.llvm.org/D131441 i.e. apparently llvm C++ headers find the wrong C headers first.

[jnikula]

@ldionne I'm guessing you authored https://reviews.llvm.org/D131441. It sounds like libclang should figure out the header search path automatically, but that doesn't seem to always be the case.

[ldionne]

The patch you linked to catches instances where the search paths are in the incorrect order. It doesn't provide anything for when the headers are not found at all, which can't be solved from the library side.

[jnikula]

@ldionne sorry for pulling you into something that can barely be called a conversation due to the lag on my side, but please bear with me...

The patch you linked to catches instances where the search paths are in the incorrect order. It doesn't provide anything for when the headers are not found at all [...]

Okay, that's fair.

[...] when the headers are not found at all, which can't be solved from the library side.

What I'm seeing in many cases is that the headers are there, properly installed, but libclang fails to locate them.

clang -E -Wp,-v -xc /dev/null on the command-line finds and shows the header paths just fine, but the equivalent passed to libclang does not. The latter is typically missing the prefix path, and the suffix alone might work by coincidence as a relative path if run in the appropriate directory. Just like in the original description of this report.

See for example jnikula/hawkmoth#262 (comment)

The questions are:

1. Is libclang normally expected to get the header search path right?

2. If yes, whose fault is it that so many distro libclang installations don't get it right? Where should this be fixed?

3. If not, how should I figure it out in an automated fashion, and what should I pass to libclang to make it find the headers relevant to that specific libclang version/installation?

[jnikula]

Everything works in Debian, but they carry this change against upstream: https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/blob/18/debian/patches/fix-clang-path-and-build.diff

Should upstream libclang have that? Should all the distros have that?

[ldionne]

@jnikula Sorry for the much delayed response :-)

From my perspective, libclang should indeed get the header search paths right, and yes that should be handled in upstream. But I don't contribute to libclang and I contribute only bits and pieces to Clang, so I'm not sure who should own that. I do have a feeling that this kind of consistency across platforms doesn't have much of an official owner in Clang, which has resulted in a lot of these kinds of situations. But I'm speculating a bit here.

[jnikula]

@ldionne No worries!

I think in the big picture the question is, how is one supposed to use libclang properly to parse code with #include of standard headers, especially for C++?

libclang obviously doesn't get the header search path right, while clang looks like it does. Lacking better ideas, we ended up running clang -xc -E -Wp,-v /dev/null or clang -xc++ -E -Wp,-v /dev/null depending on the language, parsing the results, and passing the paths to libclang with -nostdinc and -I. This has worked out reasonably well.

Except there's a catch. Even clang lists the GCC libstdc++ include directories before the LLVM libc++ directories on most if not all distros. Even that's fine in most cases, but with GCC 15 in Fedora Rawhide, the libstd++ headers result in a bunch of compiler warnings with libclang.

Things like:

  + WARNING: c++/15/bits/functexcept.h:83: '__format__' attribute argument not supported: gnu_printf
  + WARNING: c++/15/type_traits:1458: builtin __has_trivial_destructor is deprecated; use __is_trivially_destructible instead
  + WARNING: c++/15/type_traits:1848: keyword '__make_unsigned' will be made available as an identifier for the remainder of the translation unit
  + WARNING: c++/15/type_traits:2008: keyword '__make_signed' will be made available as an identifier for the remainder of the translation unit
  + WARNING: c++/15/bits/cpp_type_traits.h:359: keyword '__is_arithmetic' will be made available as an identifier for the remainder of the translation unit
  + WARNING: c++/15/ext/numeric_traits.h:72: keyword '__is_signed' will be made available as an identifier for the remainder of the translation unit
  + WARNING: c++/15/new:139: unknown attribute '__externally_visible__' ignored

So if I write a tool that's based on libclang, it's not a great experience if I can't make it work out of the box, and the user has to do some autotools type things to provide the header search path even for standard headers. I'd like to make it Just Work. But I'm not sure how anymore.

[jnikula]

I can suppress the warnings by using -isystem instead of -I, but that doesn't really change the overall issue.

[MaskRay]

When using the Clang library, including the C API libclang, the header search paths might not be expected, and it might be unfixable.
Certain default search paths are relative to getProgramPaths(), which is derived from the main executable’s location when using clang.
When using clang libraries, we might want the path of the clang library or a system clang used to compile the user project.
More complexity: The Clang library itself—whether located in its build directory or an installation directory (like CMAKE_PREFIX_PATH)—doesn’t inherently know its own location.

The correct search paths depend on context that the library can’t always determine without additional input from the user. The result is a variety of potential scenarios, each requiring specific configuration.
In my case, I use my language server, ccls, and address this by capturing CLANG_RESOURCE_DIR at build time (https://github.com/MaskRay/ccls/blob/master/CMakeLists.txt).
Then, I append -resource-dir=$CLANG_RESOURCE_DIR to the command line, ensuring the paths are correctly resolved.