diff --git a/site/blog/2025-07-20-no-gnulib.md b/site/blog/2025-07-20-no-gnulib.md deleted file mode 100644 index 7b33edf..0000000 --- a/site/blog/2025-07-20-no-gnulib.md +++ /dev/null @@ -1,236 +0,0 @@ ---- -title: Say NO to Gnulib -description: ...and Debian. ---- - -If you are a GNU maintainer and are not willing to be criticized by people sick -of your bullshit software, do not read this post. - -Also, obligatory notice on every post that touches GNU: No, nothing in this post -is political. I am perfectly fine with using GNU licensed software, and find it -more favourable than MIT licensed software as it protects user freedoms. (My -personal choice however is MPL, as it doesn't deem your software dead on arrival -at places that actually make money and feed hungry mouths) - ---- - -# Say NO to Gnulib - -The commonly overlooked but also very important argument for dropping GNU -coreutils (or any other GNU tool that depends on Gnulib) for anything other than -the most core-level bootstrapping needs is that it depends on Gnulib. - -Gnulib is a huge blob of C and 80 thousand lines of m4 that has crusted over 30+ -years and is almost impossible to build correctly and hermetically, has to be -manually patched in every single program that embeds it (if trying to be -hermetic, and correct), [^Gnulib is not a library, but a collection of source -files that you are supposed to embed into your program, so you can expect to be -pulling your hair out as it is well-rooted into most programs that embed it. -Have fun packaging it all!] rewards bad OSes, makes good OSes shrivel in pain, -makes packagers go bald & makes issues hard to diagnose and debug. - -It's so incredibly easy to build it wrong and create a shitty non-reproducible -distro (and it is built non-hermetically by default). At least in Rust (or in -other ecosystems, such as Go) and the general ecosystem of Rust, `Cargo.toml` is -pretty well-defined and `build.rs` scripts don't do anything that insane. (Hell, -even the [C compilation tools used inside crates](https://lib.rs/cc) are shared -deps and are well-defined). - -I don't trust the average distro to build any toolchain made by GNU properly, -and I do not trust them to produce a proper set of system tools either because -of Gnulib and other bad build tools. - -I do however trust the average distro (not Debian, they are lower than average -and suck at packaging [(Yes, really.)](#debian-sucks-at-packaging)) to build -Uutils tools & any other Rust tool correctly, because it is pretty relatively -straightforward compared to hundreds of thousands of lines of hard to audit & -ancient m4. It also won't misbehave or segfault when you interact with a -misbehaving NTFS partition. - -I hope Uutils coreutils & Uutils findutils and so on achieve near perfect -compliance so I do not need to serve GNU tools to my users. - -> Also, keep in mind that it doesn't have to be Rust or uutils or anything that -> is shilled by the "Rust Strike Force" in particular. I would also welcome the -> GNU people fixing up their shitty build tooling & making their software behave -> correctly. But that isn't happening any time soon, so we are starting from -> scratch & fixing (quite) a few itches on the way. - -Read more about this on the -[Sortix wiki.](https://gitlab.com/sortix/sortix/-/wikis/Gnulib) - -# Debian sucks at packaging - -Yes, really. - -From -[Phoronix, on bcachefs-tools being "impossible to maintain in a package collection":](https://www.phoronix.com/news/Debian-Orphans-Bcachefs-Tools) - -> So, back in April the Rust dependencies for bcachefs-tools in Debian didn’t at -> all match the build requirements. I got some help from the Rust team who says -> that the common practice is to relax the dependencies of Rust software so that -> it builds in Debian. So errno, which needed the exact version 0.2, was relaxed -> so that it could build with version 0.4 in Debian, udev 0.7 was relaxed for -> 0.8 in Debian, memoffset from 0.8.5 to 0.6.5, paste from 1.0.11 to 1.08 and -> bindgen from 0.69.9 to 0.66. -> -> I found this a bit disturbing, but it seems that some Rust people have lots of -> confidence that if something builds, it will run fine. And at least it did -> build, and the resulting binaries did work, although I’m personally still not -> very comfortable or confident about this approach (perhaps that might change -> as I learn more about Rust). -> -> **With that in mind, at this point you may wonder how any distribution could -> sanely package this. The problem is that they can’t. Fedora and other -> distributions with stable releases take a similar approach to what we’ve done -> in Debian, while distributions with much more relaxed policies (like Arch) -> include all the dependencies as they are vendored upstream.** - -Incredibly foolish. You are not supposed to package every single crate manually, -and you should not be anyway. - -The way you should package any programming language that has a widely used and -generally well-defined and static build system is to generate (unlisted) package -definitions from packages using a script or tool (such as -`cargo metadata -> parse json -> puke out package manifests`), and only add -extra configuration to packages that depend on anything external (such as a C -library, or CMake, or perl, or Go for some godforsaken reason (Why, -`aws-lc-sys`, why?)). - -It's also not hard to use a dependency solver algorithm to try and deduplicate -all crates required in the whole package repository using a pre-made library, -such as [`lib.rs/pubgrub`](https://lib.rs/crates/pubgrub). You can have the best -of all worlds. - -In general - Debian is a distro stuck in the 90s that assumes every language -ecosystem is as fragmented, differing and inconsistent as C's. That's not the -case anymore, Debian maintainers should wake up from their slumber and modernize -their tools, automating way more. The future is not C[^Nor is it Rust, but -that's the best we have right now & it is pretty damn good!], and a good distro -cannot assume that. - -> And so, my adventure with bcachefs-tools comes to an end. I’d advise that if -> you consider using bcachefs for any kind of production use in the near future, -> you first consider how supportable it is long-term, and whether there’s really -> anyone at all that is succeeding in providing stable support for it. - -It's trivial to support! -[Here is what Nixpkgs, the biggest Nix package -collection, Nixpkgs, does](https://github.com/NixOS/nixpkgs/blob/6e987485eb2c77e5dcc5af4e3c70843711ef9251/pkgs/by-name/bc/bcachefs-tools/package.nix) - -look, it's all 140 lines of code! - -## But it can be even better - -Nixpkgs has [`pkgs.buildRustCrate`](https://noogle.dev/f/pkgs/buildRustCrate), -to build crates without Cargo, but currently doesn't use it for most packages, -so it doesn't have crate-level incremental rebuilds. This may change in the -future, and when it does, compiling Rust programs will take a fraction of the -time because you aren't building dependencies over and over and over and over -again, and can utilize `cache.nixos.org`, or any other cache. - -It will also decrease the amount of lines you'll have to write in Nixpkgs -package specifications, because you'll no longer have to specify all external -dependencies for a program. Why? Because external, non-Cargo managed -dependencies will be configured in a -[`per-crate basis,`](https://github.com/NixOS/nixpkgs/blob/f101cc2c243f0f3869f9a214d71b736c66b5317a/pkgs/build-support/rust/default-crate-overrides.nix) -so a top-level Rust program that uses a crate that requires `liburing` won't -actually see `liburing` when being compiled. - -And here's another great part: Since the `pkgs` view is not a 1:1 map of all the -derivations (build units, but in a fancy Nix / Functional Software Deployment -Model way), these generated definitions won't pollute search or interfere with -users. It's just a plus in almost every way imagineable.[^A negative would be -the sandbox creation overhead of `nix-daemon`, but that shouldn't be significant -for most of Rust builds from scratch, and for those that are, the already-cached -crates will mitigate the losses pretty easily.] - -
- -You can actually use this right now with -[`cargo2nix`](https://github.com/cargo2nix/cargo2nix) or -[`crate2nix`](https://github.com/nix-community/crate2nix), though expect some -headaches. (I recently spent a few hours getting `crate2nix` to work on a huge -codebase with a lot of native dependencies, and `aws-lc-sys` was the biggest -headache on darwin) - -- -## Takeaway...? - -**Package management and build systems aren't hard, your tools are just bad.** - -That's why I'm working on a new system named "Cull", which will hopefully solve -a lot of these problems (and thus fix the mistakes of Nix). Stay tuned! - -It will also be cross platform -(Linux/BSDs/Darwin/Windows/\--The extra crate overrides I ended up with, if it helps.
- -```nix -{ pkgs, ... }: -{ - openssl = old: { - buildInputs = old.buildInputs or [ ] ++ [ - pkgs.openssl - ]; - }; - - openssl-sys = old: { - RUSTFLAGS = "--cfg ossl111 --cfg ossl110 --cfg ossl101"; - - "${pkgs.stdenv.buildPlatform.rust.cargoEnvVarTarget}_OPENSSL_DIR" = "${pkgs.openssl.dev}"; - OPENSSL_NO_VENDOR = "1"; - OPENSSL_STATIC = "0"; - }; - - aws-lc-sys = old: { - nativeBuildInputs = old.nativeBuildInputs or [ ] ++ [ - pkgs.cmake - pkgs.go - pkgs.libclang - pkgs.perl - pkgs.nasm - ]; - env.CFLAGS = "-D_DARWIN_C_SOURCE"; # Hours of my life wasted to figure this out. - }; - - librocksdb-sys = old: { - buildInputs = old.buildInputs or [ ] ++ [ - pkgs.zlib - pkgs.zstd - pkgs.bzip2 - pkgs.lz4 - ]; - }; - - tokio = old: { - extraRustcOpts = old.extraRustcOpts or [ ] ++ [ - "--cfg tokio_unstable" - ]; - }; - - uvm_syn = old: { - extraRustcOpts = old.extraRustcOpts or [ ] ++ [ - "-Awarnings" - ]; - }; - - pyo3_build_config = old: { - buildInputs = old.buildInputs or [ ] ++ [ - pkgs.python313 - ]; - }; -} -``` - -I also had to patch the `Cargo.nix` generator because it was not handling or -even exposing a way to enable extra cfg flags separately, but maybe this is -fixed in `cargo2nix`? - -Anyway, we do not talk about the general state of Nix/Nixpkgs/NixOS tooling. -(But it's still better than every other distro out there) - -