diff --git a/site/blog/2025-07-20-no-gnulib.md b/site/blog/2025-07-20-no-gnulib.md new file mode 100644 index 0000000..7b33edf --- /dev/null +++ b/site/blog/2025-07-20-no-gnulib.md @@ -0,0 +1,236 @@ +--- +title: Say NO to Gnulib +description: ...and Debian. +--- + +If you are a GNU maintainer and are not willing to be criticized by people sick +of your bullshit software, do not read this post. + +Also, obligatory notice on every post that touches GNU: No, nothing in this post +is political. I am perfectly fine with using GNU licensed software, and find it +more favourable than MIT licensed software as it protects user freedoms. (My +personal choice however is MPL, as it doesn't deem your software dead on arrival +at places that actually make money and feed hungry mouths) + +--- + +# Say NO to Gnulib + +The commonly overlooked but also very important argument for dropping GNU +coreutils (or any other GNU tool that depends on Gnulib) for anything other than +the most core-level bootstrapping needs is that it depends on Gnulib. + +Gnulib is a huge blob of C and 80 thousand lines of m4 that has crusted over 30+ +years and is almost impossible to build correctly and hermetically, has to be +manually patched in every single program that embeds it (if trying to be +hermetic, and correct), [^Gnulib is not a library, but a collection of source +files that you are supposed to embed into your program, so you can expect to be +pulling your hair out as it is well-rooted into most programs that embed it. +Have fun packaging it all!] rewards bad OSes, makes good OSes shrivel in pain, +makes packagers go bald & makes issues hard to diagnose and debug. + +It's so incredibly easy to build it wrong and create a shitty non-reproducible +distro (and it is built non-hermetically by default). At least in Rust (or in +other ecosystems, such as Go) and the general ecosystem of Rust, `Cargo.toml` is +pretty well-defined and `build.rs` scripts don't do anything that insane. (Hell, +even the [C compilation tools used inside crates](https://lib.rs/cc) are shared +deps and are well-defined). + +I don't trust the average distro to build any toolchain made by GNU properly, +and I do not trust them to produce a proper set of system tools either because +of Gnulib and other bad build tools. + +I do however trust the average distro (not Debian, they are lower than average +and suck at packaging [(Yes, really.)](#debian-sucks-at-packaging)) to build +Uutils tools & any other Rust tool correctly, because it is pretty relatively +straightforward compared to hundreds of thousands of lines of hard to audit & +ancient m4. It also won't misbehave or segfault when you interact with a +misbehaving NTFS partition. + +I hope Uutils coreutils & Uutils findutils and so on achieve near perfect +compliance so I do not need to serve GNU tools to my users. + +> Also, keep in mind that it doesn't have to be Rust or uutils or anything that +> is shilled by the "Rust Strike Force" in particular. I would also welcome the +> GNU people fixing up their shitty build tooling & making their software behave +> correctly. But that isn't happening any time soon, so we are starting from +> scratch & fixing (quite) a few itches on the way. + +Read more about this on the +[Sortix wiki.](https://gitlab.com/sortix/sortix/-/wikis/Gnulib) + +# Debian sucks at packaging + +Yes, really. + +From +[Phoronix, on bcachefs-tools being "impossible to maintain in a package collection":](https://www.phoronix.com/news/Debian-Orphans-Bcachefs-Tools) + +> So, back in April the Rust dependencies for bcachefs-tools in Debian didn’t at +> all match the build requirements. I got some help from the Rust team who says +> that the common practice is to relax the dependencies of Rust software so that +> it builds in Debian. So errno, which needed the exact version 0.2, was relaxed +> so that it could build with version 0.4 in Debian, udev 0.7 was relaxed for +> 0.8 in Debian, memoffset from 0.8.5 to 0.6.5, paste from 1.0.11 to 1.08 and +> bindgen from 0.69.9 to 0.66. +> +> I found this a bit disturbing, but it seems that some Rust people have lots of +> confidence that if something builds, it will run fine. And at least it did +> build, and the resulting binaries did work, although I’m personally still not +> very comfortable or confident about this approach (perhaps that might change +> as I learn more about Rust). +> +> **With that in mind, at this point you may wonder how any distribution could +> sanely package this. The problem is that they can’t. Fedora and other +> distributions with stable releases take a similar approach to what we’ve done +> in Debian, while distributions with much more relaxed policies (like Arch) +> include all the dependencies as they are vendored upstream.** + +Incredibly foolish. You are not supposed to package every single crate manually, +and you should not be anyway. + +The way you should package any programming language that has a widely used and +generally well-defined and static build system is to generate (unlisted) package +definitions from packages using a script or tool (such as +`cargo metadata -> parse json -> puke out package manifests`), and only add +extra configuration to packages that depend on anything external (such as a C +library, or CMake, or perl, or Go for some godforsaken reason (Why, +`aws-lc-sys`, why?)). + +It's also not hard to use a dependency solver algorithm to try and deduplicate +all crates required in the whole package repository using a pre-made library, +such as [`lib.rs/pubgrub`](https://lib.rs/crates/pubgrub). You can have the best +of all worlds. + +In general - Debian is a distro stuck in the 90s that assumes every language +ecosystem is as fragmented, differing and inconsistent as C's. That's not the +case anymore, Debian maintainers should wake up from their slumber and modernize +their tools, automating way more. The future is not C[^Nor is it Rust, but +that's the best we have right now & it is pretty damn good!], and a good distro +cannot assume that. + +> And so, my adventure with bcachefs-tools comes to an end. I’d advise that if +> you consider using bcachefs for any kind of production use in the near future, +> you first consider how supportable it is long-term, and whether there’s really +> anyone at all that is succeeding in providing stable support for it. + +It's trivial to support! +[Here is what Nixpkgs, the biggest Nix package +collection, Nixpkgs, does](https://github.com/NixOS/nixpkgs/blob/6e987485eb2c77e5dcc5af4e3c70843711ef9251/pkgs/by-name/bc/bcachefs-tools/package.nix) - +look, it's all 140 lines of code! + +## But it can be even better + +Nixpkgs has [`pkgs.buildRustCrate`](https://noogle.dev/f/pkgs/buildRustCrate), +to build crates without Cargo, but currently doesn't use it for most packages, +so it doesn't have crate-level incremental rebuilds. This may change in the +future, and when it does, compiling Rust programs will take a fraction of the +time because you aren't building dependencies over and over and over and over +again, and can utilize `cache.nixos.org`, or any other cache. + +It will also decrease the amount of lines you'll have to write in Nixpkgs +package specifications, because you'll no longer have to specify all external +dependencies for a program. Why? Because external, non-Cargo managed +dependencies will be configured in a +[`per-crate basis,`](https://github.com/NixOS/nixpkgs/blob/f101cc2c243f0f3869f9a214d71b736c66b5317a/pkgs/build-support/rust/default-crate-overrides.nix) +so a top-level Rust program that uses a crate that requires `liburing` won't +actually see `liburing` when being compiled. + +And here's another great part: Since the `pkgs` view is not a 1:1 map of all the +derivations (build units, but in a fancy Nix / Functional Software Deployment +Model way), these generated definitions won't pollute search or interfere with +users. It's just a plus in almost every way imagineable.[^A negative would be +the sandbox creation overhead of `nix-daemon`, but that shouldn't be significant +for most of Rust builds from scratch, and for those that are, the already-cached +crates will mitigate the losses pretty easily.] + +
+ +You can actually use this right now with +[`cargo2nix`](https://github.com/cargo2nix/cargo2nix) or +[`crate2nix`](https://github.com/nix-community/crate2nix), though expect some +headaches. (I recently spent a few hours getting `crate2nix` to work on a huge +codebase with a lot of native dependencies, and `aws-lc-sys` was the biggest +headache on darwin) + +
+The extra crate overrides I ended up with, if it helps. + +```nix +{ pkgs, ... }: +{ + openssl = old: { + buildInputs = old.buildInputs or [ ] ++ [ + pkgs.openssl + ]; + }; + + openssl-sys = old: { + RUSTFLAGS = "--cfg ossl111 --cfg ossl110 --cfg ossl101"; + + "${pkgs.stdenv.buildPlatform.rust.cargoEnvVarTarget}_OPENSSL_DIR" = "${pkgs.openssl.dev}"; + OPENSSL_NO_VENDOR = "1"; + OPENSSL_STATIC = "0"; + }; + + aws-lc-sys = old: { + nativeBuildInputs = old.nativeBuildInputs or [ ] ++ [ + pkgs.cmake + pkgs.go + pkgs.libclang + pkgs.perl + pkgs.nasm + ]; + env.CFLAGS = "-D_DARWIN_C_SOURCE"; # Hours of my life wasted to figure this out. + }; + + librocksdb-sys = old: { + buildInputs = old.buildInputs or [ ] ++ [ + pkgs.zlib + pkgs.zstd + pkgs.bzip2 + pkgs.lz4 + ]; + }; + + tokio = old: { + extraRustcOpts = old.extraRustcOpts or [ ] ++ [ + "--cfg tokio_unstable" + ]; + }; + + uvm_syn = old: { + extraRustcOpts = old.extraRustcOpts or [ ] ++ [ + "-Awarnings" + ]; + }; + + pyo3_build_config = old: { + buildInputs = old.buildInputs or [ ] ++ [ + pkgs.python313 + ]; + }; +} +``` + +I also had to patch the `Cargo.nix` generator because it was not handling or +even exposing a way to enable extra cfg flags separately, but maybe this is +fixed in `cargo2nix`? + +Anyway, we do not talk about the general state of Nix/Nixpkgs/NixOS tooling. +(But it's still better than every other distro out there) + +
+
+ +## Takeaway...? + +**Package management and build systems aren't hard, your tools are just bad.** + +That's why I'm working on a new system named "Cull", which will hopefully solve +a lot of these problems (and thus fix the mistakes of Nix). Stay tuned! + +It will also be cross platform +(Linux/BSDs/Darwin/Windows/\), +and cacheable at the expression level. No waiting for your system closure to +evaluate for 5 minutes. diff --git a/site/dump/posix/sysadmin-copypasta.md b/site/dump/posix/sysadmin-copypasta.md new file mode 100644 index 0000000..8c3df64 --- /dev/null +++ b/site/dump/posix/sysadmin-copypasta.md @@ -0,0 +1,27 @@ +--- +date: 2025-07-20 +--- + +You will never be a platform engineer. You have no tooling, no namespacing, no +developer agency. You are a sysadmin twisted by buzzwords and job titles into a +crude mockery of [BOFH's](https://bofh.bjash.com/) perfection. All the +"validation" you get is two-faced and half-hearted. Behind your back developers +mock you. Your boss is disgusted and ashamed of you, your "cloud support agent" +laughs at your ghoulish infrastructure behind closed doors. Developers are +utterly repulsed by you. Decades of tooling advancements have allowed developers +to sniff out frauds with incredible efficiency. Even "DevOps Engineers" who +"pass" look uncanny and unnatural to a developer. Your decrepit Git repos are a +dead giveaway. And even if you manage to convince a developer to follow your +processes, he'll turn tail and bolt the second he gets a whiff of your diseased, +infected Terraform modules. You will never be happy. You wrench out a fake smile +every single morning and tell yourself it's gonna be ok, but deep inside you +feel the AWS costs creeping up like a weed, ready to crush you under the +unbearable weight. Eventually it'll be too much to bear - you'll turn in your 2 +weeks' notice, cry in the bathroom, `terraform apply` to production, and plunge +into the cold abyss. Developers will find you, heartbroken but relieved that +they no longer have to live with the unbearable processes and procedures. +They'll fire you with cause under 'incompetence', and every recruiter for the +rest of eternity will know you're a shitty sysadmin. Your body will decay and go +back to the dust, and all that will remain of your legacy is an archived repo +that has your full name attached to it. This is your fate. This is what you +chose. There is no turning back.