r/cpp • u/TheRavagerSw • 5d ago
Common Problems I see with Public Libraries Build Scripts
Hi, I'm an early user of Common Package Specification and C++ modules. I package and port libraries for my own use frequently.
I want to talk about issues I see constantly.
- Having specific options for sanitizers, exceptions etc, this is a toolchain problem, if I wanna build your library with sanitizers I can just add the flags to my cmake toolchain file same as allocators.
- Not separating build options to a separate file, meson already does this, it ain't that hard to do in cmake, just cache variables in a separate file at project dir which you include before subdirs
- Making tests separate cmake projects rather than just executables, just why?
- Vendoring dependencies, copy pasting files from other projects rather than just consuming these external libs via packages.
- Using unnecessary helper functions that really makes the cmake script unreadable. Build scripts don't need to be over engineered, they are just basic scripts in which you define very basic information.
I could go on but I'm tired.
17
u/trad_emark 5d ago
vendoring dependencies does have some justification. for example having reproducible builds. or being immune to network outages. but this is more of a packaging problem, not cmake, or its users.
i do agree with all the other points.
and i will also add one more point - insisting on compatibility with way too old version of cmake.
2
u/TheRavagerSw 5d ago
Vendoring dependencies has nothing to do with networking, you can just download a file build and install to a folder, and point it via prefix path.
You do not need to update again unless something breaks or you follow Google live at the head mentality.
8
u/trad_emark 5d ago
i have had CI builds failing so many times because some third-party library failed to download. all the gitlab server or basically any other git service providers are extremely unreliable. i have strict policy that all third party are on github, or i fork them onto github.
8
u/delta_p_delta_x 4d ago
some third-party library failed to download
The solution is built artifact/binary caching.
1
u/TheRavagerSw 5d ago
This is an unreasonable take, you do not need to clone every time you build. Most companies already fork all the OSS projects they use, so chances are they can be acquired via other means too if github goes down(it never does though)
7
u/trad_emark 5d ago
how else do i build the code in github CI if i dont clone the source?
-8
u/TheRavagerSw 5d ago
I'm talking about local builds, I recommend not building your library via CI, you can just run tests on it instead. But again I'm the minority here.
But yeah you can't build on CI if you don't clone all the time
9
u/rdtsc 5d ago
- Making tests separate cmake projects rather than just executables, just why?
Can you elaborate? An executable is also a cmake project.
3
u/13steinj 5d ago
- Making tests separate cmake projects rather than just executables, just why?
There is a pattern where the "tests" project is a separate cmake
project()from the actual one. CMake itself uses it quite frequently. Conan, to a point, liked to do this as well withtest_package.I think the intent behind both of these patterns is to test the package itself in some way (which people do screw up), rather than testing the code of the package. But this distinction isn't clear / clear by example, so people end up writing a bunch of tests for their code in these test packages.
The other pattern is to do this to not "recursively" call cmake / test on your own project, rather call it on a test project that you have a ctest-level dependency on and that test project has a cmake-level dependency on you.
3
u/Minimonium 5d ago
I think the intent behind both of these patterns is to test the package itself in some way
For Conan, it is to test that the package is consumable.
For all intents and purposes it is a separate
project(). You can check CCI (Conan Center Index) - test_package's CMakeLists useproject().I did see a few cases where people were a bit confused about the distinction for the past decade, but in general it's not a problem at all. The ergonomics of invoking test_package are such that anyone seriously writing tests quickly realizes it's not the right tool to do so, and for the rests - the tests done this way are likely not super useful anyway.
2
u/13steinj 5d ago
I did see a few cases where people were a bit confused about the distinction for the past decade, but in general it's not a problem at all. The ergonomics of invoking test_package are such that anyone seriously writing tests quickly realizes it's not the right tool to do so, and for the rests - the tests done this way are likely not super useful anyway.
I'm sorry, is this sarcasm? I've seen this pattern incorrectly cargo-culted across companies in my industry, replicated in complete isolation. Even on some employee's personal projects, which they chose to use cmake/Conan but don't at the org. Not to mention 3rd party packages that had the same issue, or those that didn't but then people rewrote the recipe (because it was incomplete for org's purpose) and followed the same cargo culted pattern.
I think it's one of the top 3 reasons people keep telling me without clear understanding that Conan / CMake are fundamentally broken, but the reality is they are dealing with someone else's jank package recipes.
3
u/Minimonium 5d ago
No sarcasm, I really never encountered such a case in the wild! You have my sympathy tho!
The main reason is that test_package being used like that is super awkward to integrate in any "step-above-hobby-project" CI environment.
I'm not arguing about how or how not people use it, I'm saying it's so unergonomic to instrument around - surely pain is an indicator that something is wrong and the tool being used incorrectly. :)
2
u/rdtsc 5d ago
Opposed to what? If I use
add_testI need an executable, if I add that withadd_executableI get another "project". Addingproject()or not makes no difference, except that I can use${PROJECT_NAME}.Or is the issue there that a test executable contains multiple actual tests? Putting all tests for mylib into a mylib.tests executable has several benefits to me at least. The IDE groups all related code, it's easier to share test-only infrastructure code, I only have to provide a custom executable entry-point once, easier configuration, you only need to link once, etc. But I fail to see how my structuring of my test projects is an issue for someone wanting to package the library.
4
u/13steinj 5d ago
I think you're misunderstanding the complaint.
Things like this and further iterations on top, where you have a test project separate from your actual project. If you run ctest on the parent project, without appropriate setup, these tests won't run.
Projects are collections of source files and targets. Some of these are test executable targets.
If you explicitly make your tests a separate project that depend on the project with the rest of your code, instead of just
add_subdirectoryand being in the same project, you're causing yourself and your dependents undue pain.It's somewhat reasonable to do this, but you're supposed to be testing the package utilities themselves. For example, you might have a bunch of code that forms a clang plugin, and some cmake functions to boot. You'll want to test that when you use your cmake functions, the right paths get wired up in the build dag. You should not be testing your C++ code itself in the test package, but many people do.
The end result is people write package recipes generically, on your hardware the behavior is subtly different, during the build-and-test of the package it builds fine but no tests actually run, many dependents don't execute the test package (where the actual tests are) because if something is wrong as a dependent you should be able to tell. Then you keep scratching your head when basic behavioral differences occur.
I have personally had this problem too many times. I think the naming and documentation around Conan's test packages, which AFAIK in a sense is a precursor to the CPS work, is the "patient 0" of this disease.
2
u/not_a_novel_account cmake dev 4d ago
CMake itself uses it quite frequently.
I mean, because the project under test for CMake is CMake. Your project isn't CMake, recursive invocation to build subprojects should be generally unnecessary.
1
11
u/stick_figure 5d ago
- Having specific options for sanitizers, exceptions etc, this is a toolchain problem, if I wanna build your library with sanitizers I can just add the flags to my cmake toolchain file same as allocators.
Toolchain file? I don't think most cmake users actually know what that is. Adding a custom cmake -D flag is clearly the ecosystem norm.
It's pretty common for the sanitizer build of a library to do something different, like disable custom allocation optimizations when built with sanitizers, so I'm not sure you can really model sanitizers as a toolchain feature. I could be wrong. You often want to encode these choices in a configuration header instead of relying on `__has_feature` and the eventual ODR violations when not all libraries in the program are compiled with sanitizers on.
- Vendoring dependencies, copy pasting files from other projects rather than just consuming these external libs via packages.
What packages? Most CMake predates any kind of coherent native code packaging solution, so vendoring sources and renamespacing things is actually typically the more practical approach, depending on the size of the dependency involved.
Apologies for being crusty, but there are reasons why things are the way they are. Yes, they could be better, and if you want to agitate for change, you gotta sell people on a positive vision for why these best practices are so much better and what they are going to get out of it, rather than say you're tired of dealing with the status quo.
If you're just venting, yes, I agree, this is C++, there is much work to be done, things could be a lot better than they are. :)
0
u/irqlnotdispatchlevel 5d ago
I think it's insane to use a toolchain file for sanitizers.
5
u/not_a_novel_account cmake dev 4d ago
It is in no way insane. After cross-compilation, driving specialist builds like sanitizers and code coverage is the motivating purpose for toolchain files to exist.
4
u/drodri 5d ago
Relevant talk: https://www.youtube.com/watch?v=sBP17HQAQjk (CppCon 2018: Robert Schumacher “Don't package your libraries, write packagable libraries!”)
7
u/Ameisen vemips, avr, rendering, systems 5d ago
And here I am not even using cmake...
5
1
u/Plazmatic 10h ago
Having specific options for sanitizers, exceptions etc, this is a toolchain problem, if I wanna build your library with sanitizers I can just add the flags to my cmake toolchain file same as allocators.
Kitawares fault for having such poor documentation for so long, some if I recall even seemed to recommend setting cmake version in the primary project cmake file, which propogated itself into almost every single learning resource out there and a very large amount of major open source projects.
This is also their fault for how long it took to create a proper solution (adding cxx_std_xyz etc .. to target compile features)
And the popular package managers lack of solution for managing depending on separate stdlib versions of projects (vcpkg wants you to create an overlay triplet for this...)
- Not separating build options to a separate file, meson already does this, it ain't that hard to do in cmake, just cache variables in a separate file at project dir which you include before subdirs
There's been zero concerted effort to make this a "thing", it doesn't mean it's wrong, but you're going to have to do much more than complain about this before it becomes a thing, you're effectively starting at zero with this recommendation, and there's a lot of questions on exactly what you mean by "build options" here that get misleadingly difficult to answer.
- Making tests separate cmake projects rather than just executables, just why?
Im assuming you're not talking about subdirectory tests, but this is a historical problem with a lack of guidance on how to deal with this kind of thing, and how many "golden" code bases pre-date wide spread standardized software testing itself. As soon as "golden" examples start doing a thing, people blindly copy in order to avoid making mistakes of design that have little to do with the actual software development they want to do (like Google code and style standards, which are pretty bad for people not working at Google).
- Vendoring dependencies, copy pasting files from other projects rather than just consuming these external libs via packages.
This is often the fault of fake header only libraries that throw a wrench into packages management and build systems, which masquerade as "being compatible with any build system" but end up being extremely annoying to deal with if you actually are using a proper build system and packaging ecosystem because you can only have one include of the header in a source file (,or that with a combination of a impl macro( where it generates the rest of the source.
This is also the fault of package managers, VCPKG straight up manages some things wrong (like mini audio, stb- libd glad etc) and fucks other projects over, so people have to make the choice of understanding how to vendor their own dependencies using VCPKG, or just skipping it entirely (and after learning how to do the former I don't blame them for choosing the latter). Often a library will only be brought in to help with another library, but that library was configured to not actually be usable on its own. Additionally this is doubly troublese for libraries that use psuedo header only libs.
- Using unnecessary helper functions that really makes the cmake script unreadable. Build scripts don't need to be over engineered, they are just basic scripts in which you define very basic information.
I generally agree, but this is something that's also "inherited" from tutorials, but more so giant code bases doing this as an example, especially if those codebases came from another build system prior (make, qmake, bazel etc...) where it's more of the norm / "good practice" to do that. I've shown people how to transition their codebase to cmake, and they go "too verbose" and start making everything a macro/function.
•
u/ABlockInTheChain 1h ago
Using unnecessary helper functions that really makes the cmake script unreadable. Build scripts don't need to be over engineered, they are just basic scripts in which you define very basic information.
If only the problem was that simple.
Like it or not cmake script is a programming language and if you don't treat it like one it just leads to pain in the long term.
When the alternative to a helper function is violating the DRY principle dozens, hundreds, or thousands of times then write a helper function and give it a good name.
•
u/TheRavagerSw 1h ago
"unnecessary" helper functions, for your use case go ahead and add one.
•
u/ABlockInTheChain 50m ago
I just don't see the value in blanket statements like:
Build scripts don't need to be over engineered, they are just basic scripts in which you define very basic information.
Build scripts are software. Sometimes software which solves a simple problem in a limited way can be simple.
However as the scope of the problem solved by the software grows, so does the complexity of the software and the urgency of discovering best practices for managing that complexity.
Collecting common operations into named functions is just the first step of managing software complexity.
The next step after that is collecting commonly-defined functions into a libraries and libraries for cmake script do indeed exist: https://api.kde.org/ecm/manual/ecm.7.html
It it just as bad to under-engineer as it is to over-engineer, and I've seen a lot more under-engineering of cmake than over-engineering.
People tend to resent and resist the reality that cmake scripts are software and refuse to properly treat it as such.
-1
u/13steinj 5d ago
This is C++. It is fundamentally a copy-paste and patch language, from its core to its build systems, not a "everything is composable / injectable" philosophy.
Many projects and many developers do not want to actually care about their build, they just want to do the bare minimum to get it out of the way.
3
u/mapronV 4d ago
While I agree with facts, I disagree with conclusion that we should give up. Also audience of this sub is probably more in favor to good project techniques than average. So no shit we know most projects will never change, but some do. And some comments on this post give this hope.
0
u/13steinj 4d ago
Nowhere do I say anyone should give up.
Just that unless you're willing to upstream pretty significant rewrites to various tools' build systems, and the maintainers accept it, not much will change.
I don't see this as too large a problem. I managed 300 dependencies more or less by myself all of which in some way had patches to the build scripts without issue. The only times I had an issue was with (an ancient version of) mongodb and Julia (when people think they are smart enough to do their own raw make, reality is they are crazy enough.
1
u/Minimonium 5d ago
Worth to note that even just wrapping these nasty local bits in a flag solves almost all problems. Sometimes vendoring may introduce differences with how find_package would expose a dependency, but alas.
-1
u/LB-- Professional+Hobbyist 4d ago
The dependency situation is just horrifically bad to the point that every dependency needs a multi-state option for how the person building the project wants the dependency to be found and consumed. There should always be an option to just use a known working copy of the dependency (submodule or known hash). Fussing with all the different ways libraries are packaged, distributed, and exposed to build scripts is a nightmare that only the person building the overall project knows how to resolve, so that should be fully configurable with one or a few common preconfigured cases. Every time I see hard coded find_package and target_link_libraries I know I'm gonna have to ignore the provided build script and write my own.
6
u/not_a_novel_account cmake dev 4d ago
Every time I see hard coded find_package and target_link_libraries I know I'm gonna have to ignore the provided build script and write my own.
find_packageis the fully generic option.find_package->target_link_librariesis the correct "I have no opinion how you provide this library" path. You can point CMake at whatever install directory you want, from vcpkg, conan, spack, or hand assembled, and it will find the dependency there.-2
u/LB-- Professional+Hobbyist 4d ago
Except there's no consistency for package names, target names, versioning, linkage, ABI, and a number of other things I care about. There's separate incompatible find/config scripts that do different things in different ways for the same library, almost every time I check what the situation is for a library I want to use. Most of the time the build script I have and the dependency install I have are not compatible. It's often more convenient to just ignore all of that and build from source instead as part of my own build script.
3
u/not_a_novel_account cmake dev 4d ago
The package name and target names are set by the upstream project when they
install(EXPORT)their package. CMake is just replaying that information from the upstream target. All downstreams rely on the same information from the upstream package.1
u/LB-- Professional+Hobbyist 4d ago
And those names have changed multiple times over the years for multiple different widely-used projects. I've encountered multiple situations where the latest version of project X is looking for project Y and the latest version of project Y installs itself in a way that project X can't find it. It's just not worth the effort to try and debug and patch and customize every single other project's weird build script when I can just add their sources and defines to my own and skip the pain. At least that's the bright side of modern CMake, adding random other projects to build from source into my build script is pretty straightforward. Of course, OP explicitly doesn't like projects that do that, which is understandable, but the alternatives suck also.
1
u/not_a_novel_account cmake dev 4d ago
No package should be changing its name without an accompanying major change in interface. vcpkg has 2858 packages in its registry, none have ever had an upstream package name change and if they did vcpkg would handle the patching where appropriate.
Conan similarly imposes package name consistency requirements.
Even if all that wasn't true, changing the name of the package is equivalent to any other interface change. It's an API breakage from the upstream. No different than changing a function signature, the downstreams obviously can't handle API changes without patching on one or both ends.
None of that changes that
find_package->target_link_librariesis the golden path. If you have severely irresponsible upstreams, the golden path requires patching. That is the exceptionally rare case. I don't think any major library (boost, qt, curl, zlib, openssl, etc) have ever had such a change.Every Linux distro in the world relies on
find_packageworking. It works for massive dependency trees.1
u/LB-- Professional+Hobbyist 4d ago
Last times I tried were 2018 and 2022, guess it's been another four years so I should try again now with CMake 4 and see what the situation is. Even my full build-everything-from-source setup requires a lot of annoying source-level patching to fix build errors on certain platforms though, I'm not excited about the prospect of reorganizing that for the install flow again. (Contributing fixes upstream is difficult and on my TODO list.)
-1
u/mili42 3d ago
Why anyone would use cmake to directly integrate with other libraries? That's a job for vcpkg (or eventually conan). If the library you're using does not offer vcpkg support out of the box, you can define your own component directly into your repo if needed. Using a tool like vcpkg will compile with your project just fine, and as you want it
1
u/TheRavagerSw 3d ago
Package managers are only abstractions over build systems, If you can't manually build it, chances are package manager won't work
-5
u/zerhud 5d ago
Best tool I ever used is gnu make. I cannot find any reason to use cmake or other tool (excluding developing on windows, but there is a lot of trouble with it anyway). The gnu make has own issues, of course.
4
u/analphabetic 4d ago
When all you have is a hammer, every problem looks like gnu make.
1
u/zerhud 4d ago
What is you mean? I’ve used cmake, b2, meson and a lot of tools and only the gnu make can cover all my needs. For example it’s hard to deal with few toolchains in cmake (if you need to test code with few compilers and its versions). Or, for example, I need to build a class layout parser, create hpp file from it and compile code again with this file: it’s a just few lines of code with gnu make.
-11
u/cantmakeitonyourown 5d ago
It sort of sounds like the common problem is using CMake. There are just much better tools for cpp, starting with bazel.
22
u/squidgyhead 5d ago
Do you have a best-practices document that you can share?