koine devlog part 4

In which I am reminded I don't actually know how to use Nix that well

Check this shit out:

As discussed in the Pele wrapper library post from the other day, I've been busy trying to add Mermaid diagram rendering to koine by way of the recently-released Pele rendering library. (At this point I've actually beaten Pele making it into general-access Obsidian, the v1.15.0 release that includes it is still in Catalyst member preview. Which, like, I did pay for a Catalyst license, but I ain't logging in to any account in Obsidian, get real.)

I said the other day that I wanted to finish up the wrapper library before I added the features to koine. I changed my mind. I was watching the latest episode of the BEAM There, Done That podcast with Davide Bettio, the creator of AtomVM, as well as Giacomo Cavalieri from the Gleam core team; at one point Giacomo discusses his philosophy for writing Gleam wrappers for external libraries, which he boils down to "wrap the features you need, and once you have a decent thing going, then extract it to a library."

(As an aside, AtomVM is a super cool project, and I would love to do something with it at some point. It lets you run the BEAM on embedded devices!)

I was thinking about this--this is how my Ohm wrapper came together--and also about how I didn't really know how I'd want to use Pele's 'mount' or 'zoom' features, so how was I supposed to design APIs for them? They won't be in koine since they're for running Pele client-side. I considered a lot of this up front before I even started the wrapper and decided I wasn't going to worry about it, but hearing Giacomo talk about this sort of flipped a switch in my brain and I decided maybe I should take my own (and others' advice) and stop doing the thing that felt like kind of a silly thing to do. Ah well, it was fun while it lasted.

(As another aside, Giacomo is a great speaker. I especially enjoyed his talks about his SQL->Gleam codegen library 'squirrel' and his snapshot testing library 'birdie'. The moment I have something to put in a database or, uh, test, I'll be there. And don't even get me started on his SVG chart rendering library, 'gerbil'!)

Anyway, I went ahead and integrated Pele into koine by just vendoring the wrapper into the koine repository, warts and all. You can check it out if you want to see the aggressive number of 'panic's in main execution paths.

That was the easy part, though. The actual hard part was getting koine to build and run again after I'd done it.

Did you think I forgot about the subtitle, no, I didn't

I updated the koine Flake to use the 'pele_gleam' local dependency and change the target to JavaScript, built, ran the new binary, and...

$ result/bin/koine --help
node:internal/modules/esm/resolve:989
    throw error;
    ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'pele' imported from ...
...

Oh. Duh. I then proceeded to embarrass myself to a nonexistent audience of Nix aficionados, writing things like this:

...
postInstall = ''
  ${pkgs.nodejs}/bin/npm install -g pele
'';
...

You know how the whole deal with Nix is that builds are hermetic and you have to get files into the Nix store if you want to be able to access them during your build and you can't just make network calls? I know that, except apparently when I'm trying to write hooks for a package build for the first time, and then I start taking random actions with no theory behind them whatsoever. And that was after I understood what was going on with the runHooks to begin with.

As it stands now, the relevant part of the koine Flake looks like this:

...
{
  packages = forEachSystem (system: let
    pkgs = import nixpkgs { inherit system; };
    pnpm = pkgs.pnpm_12; 
    pname = "koine"; 1️⃣
    version = "0.0.1";
    src = ./.;
  in
  {
    default = nix-gleam.packages.${system}.buildGleamApplication {
      inherit pname version src;
      localPackages = [./vendor/glemtext ./vendor/pele_gleam];
      target = "javascript";
      nativeBuildInputs = [ pkgs.pnpmConfigHook pnpm ]; 2️⃣
      pnpmDeps = pkgs.fetchPnpmDeps {
        inherit pname version src pnpm;
        fetcherVersion = 4;
        hash = "sha256-hQE/LTysuu6EpTCUO9C/t74XJGkG2txkhjJ5aw0hxkk="; 3️⃣
      };
      postBuild = '' 4️⃣
        ${pnpm}/bin/pnpm install --node-linker hoisted 5️⃣
        mv node_modules build/dev/javascript/
      '';
    };
  });
}
...

A few features of note:

• 1️⃣: The 'fetchPnpmDeps' function, which retrieves the dependencies in your lockfile and adds them to the Nix store (i.e., does the thing we need to have happen), expects the pname/version/src of your package. 'buildGleamApplication' calculates these itself, but since we don't have 'finalAttrs' or any other way to get recursive access to package properties as far as I know I just gave in and hoisted them out of the definition.
• 2️⃣: The 'pnpmConfigHook' "package" does some kind of black magic I don't totally understand that configures pnpm so that when you run 'install' normally it uses the files from the Nix store instead of trying to go to the network. I guess it's probably setting up a store dir and moving stuff into it behind the scenes, but I also don't get how me including it in 'nativeBuildInputs' means it gets to run code in the build. Ah, well, average nixpkgs symbol behavior.
• 3️⃣: The example in the nixpkgs docs gave 'hash = "..."', which seemed like it couldn't be right, but I typed that anyway. Sure enough I got an error telling me that wasn't a valid hash, so I deleted it and went with the empty string. *That* got me a different error that also spat back out this hash, so I copied it into the file. I think it's the hash of my package.json, or maybe of the package.json and the lockfile somehow? There's apparently a tool you could use to generate this without having to run a failed 'nix build', but I'll cross that bridge if I ever come to it.
• 4️⃣: runHooks! Thankfully the 'nix-gleam' flake is well-written and provides all the normal package hooks, plus also lets you add the necessary attrs to configure them. As you saw above I started out trying the postInstall hook, but due to the way 'buildGleamApplication' constructs the final package filesystem it was easiest to get the 'node_modules' folder into the JavaScript build directory right after the build.
• 5️⃣: And, finally, good old pnpm 'node_modules' filesystem nonsense. The 'fetchPnpmDeps'/'pnpmConfigHook' duo claims to accept a 'pnpmInstallFlags' argument, which clearly does something, but when I provided '[ "--node-linker" "hoisted" ]' as an argument I was getting errors that showed that clearly the two separate list items were getting joined into one arg (i.e., the arg was '--node-linker hoisted', all one token), so I gave up and put the args directly in the hook script. I'm not sure if 'hoisted' is necessary, but it looks like Nix was eating the symlinks, so this seemed like the easiest fix.

While it was a little time-consuming (at a point where I didn't so much have the time to be consumed) getting this build worked out (I had originally intended to do more development on pele_gleam tonight), in the end I enjoyed having to work through the Nix troubleshooting. I've done a lot of script kiddie copy-pasting in my Nix career so far, or asking LLMs for help, so piecing this together from the docs was very good practice. If you have any suggestions for improvements, please, I'm all ears.

(Also, one last final tragedy: html-tidy, the formatting library I *was* using to keep the journcy.net HTML nice and readable if that's your thing, was doing nasty stuff to certain bits of the SVG HTML. It doesn't seem to respect the 'xml:space="preserve"' attributes Pele uses and was mucking the diagrams all up. So, I had to take the formatting out of the build. May it return one day soon.)

Now I just need to come up with something I actually need to write a Mermaid diagram about.

The kanji of the day is 尽

• Its Kodansha Kanji Learner's Course keyword is EXHAUST.
• Its readings are つ, づ, ず, ことごと, ジン, and サン.

Guy leaning on his shovel with some sweat dripping off him (because he's so exhausted). It's interesting to me how despite being so similar to 冬/WINTER (Wiki thinks the two strokes at the bottom of both are actually a variant of the two strokes we met before in 次/NEXT, meaning ICE, which is interesting), this kanji barely rates for me. It's okay, I guess? Shovel guy just leaves me wanting.