Brioche

Brioche Project Update - November 2024

Published on

By Kyle Lacy

The last month (and change) has been one of rabbit holes!

Package updates

Since the last update, there were 11 new packages:

Python was a big milestone! It was quite fun to puzzle out how package builds look when using Python. You can see asciinema for what a Python-based build looks like (although projects using Poetry/uv/pyenv/etc will probably end up being quite different— asciinema just uses pure Python and pip)

moreutils proved to be a surprising challenge— it’s simple on the surface but it turned out to be hard specifically due to how docs are built. I needed to figure out what an “XML catalog” was, which I used to create this monstrosity beautiful inline XML file directly in the package definition. (next up: adding JSX support for defining inline XML files more easily in Brioche?)

std updates

There were a few disperate changes in std since last update. Full changes in the changelog.

There was a breaking change around the std.process() function— the unsafe options have been reorganized to be more composable. The rust package is now taking advantage of this as of brioche-dev/brioche-packages#123, where it passes through all unsafe options directly to the underlying std.process().

There were also two new functions used to create “runnables” (this also included a minor breaking change around process template symbols). The new functions std.withRunnable() and std.addRunnable() provide a nice alternative to the very bare-bones std.withRunnableLink() function, and the heavy-handed std.bashRunnable template function that pulls in Bash as a dependency.

Finally, there were a few minor changes around the toolchain to sand off a few more sharp edges:

Brioche core updates

Okay, here’s where things get exciting (and also a little complicated!)

This cycle, I’ve been interested in improving developer ergonomics in preparation to start on cross-platform builds, so I wanted to revisit some of the minor pain points in the flow for building and debugging packages before embarking on that journey.

New console output

I was getting pretty tired of the “pretty” output that the CLI currently uses… so I redesigned it!

Here’s a side-by-side between the current version in Brioche v0.1.3 and the new one as of brioche-dev/brioche#137:

Before
After

(You can probably guess why I added the asciinema package recently!)

I find the new output much easier to scan personally. The new output also uses a higher framerate, which makes it “feel” faster— to my eyes at least.

New process logging

When a build fails in Brioche v0.1.3, you get a very ugly error message:

Terminal window
Error: process failed, view full output from these paths:
- /home/user/.local/share/brioche/process-temp/01JCYBMMGZ5QJD61R5Z0H7WZCQ/stdout.log
- /home/user/.local/share/brioche/process-temp/01JCYBMMGZ5QJD61R5Z0H7WZCQ/stderr.log: process exited with status code exit status: 1

The file paths listed though are pretty helpful! You can see the full stdout/stderr of the failed process (and pro tip: the directories containing those files contain the build’s working directory, making it easy to find things like config.log files). Lots of process write to both stdout and stderr, but since they get written to separate files, these logs are lossy, since you lose how the messages get interleaved. In practice, this just makes trawling through failed logs harder than it should be.

As of brioche-dev/brioche#138, the error message is… still pretty ugly, but more useful!

Terminal window
Error: process failed, view full output by runing `brioche jobs logs /home/user/.local/share/brioche/process-temp/01JCYC3PN9QHFJDD6R41KRTDMB/events.bin.zst`: process exited with status code exit status: 1

As hinted by the error message, the separate stdout and stderr files have been replaced with a new mysterious events.bin.zst file, ooh! This is a pretty simple (binary) file format that includes both stdout and stderr, but with lots of fancy bells and whistles:

If we run the suggested brioche jobs logs command, we get a pretty detailed output:

Terminal window
$ brioche jobs logs /home/user/.local/share/brioche/process-temp/01JCYC3PN9QHFJDD6R41KRTDMB/events.bin.zst
process stack trace:
- file:///home/user/.local/share/brioche/projects/97beeb18f169ced9073a4f542f4195ef675ca4b1407170a247485c21545987bf/extra/run_bash.bri:42:14
- file:///home/user/example-project/project.bri:4:21


[0.00s] [spawned process with pid 109069, preparation took 0.01s]
[0.01s] uh oh, build failed
[0.01s] [process exited with code 1]

There’s lots of untapped potential with the new process event format, but I’m pretty happy with this as a first pass! brioche jobs is an entirely new subcommand, and I think it’ll be possible to add lots of handy new utilities for diving in when a build fails.

Compression woes

Oh, also, you may have noticed the filename above was events.bin.zst, implying zstd (Zstandard) compression. This is obviously good! If you have a long-running and verbose build that dumps a ton of output to stdout and stderr, compression can save a lot of disk space.

…but, I wanted to have my cake and eat it too. The new brioche jobs logs command also supports the flag --reverse, allowing you to print events starting from the end. For a super long build log, it’d be annoying to have to wait for the entire file to decompress just to show the last 100 or so log lines!

That’s when I learned about the zstd seekable format, which basically chunks a zstd stream into multiple frames, and allows efficient seeking by only needing to decompress a particular frame: in the case of reading only a few lines starting from the end, you might only need to read the last frame for example (currently ~1MB of data). It’s not a magic bullet though: dividing a stream into multiple frames does harm the compression ratio. But, in this case, I thought the trade-off was worth it.

Unfortunately, the zstd seekable format isn’t yet supported in either the zstd or the async-compression Rust crates (it should be decodable by any compliant decoder, but it wouldn’t allow seeking and it wouldn’t be possible to encode the seekable format without explicit support). So, I ended up writing the zstd-framed crate to handle the compression and decompression with the seekable format. It should also gracefully handle edge cases, like if the process dies after having written a partial file, where it’ll allow decoding even if the file wasn’t closed properly during encoding. Good if, I dunno, your computer shuts down during a long-running build!

Anyway, the choice to use compression ended up stretching out development of the new process event format by several weeks, but I’m really happy with the end result!

Extension updates

@asheliahut just published v0.2.0 of the brioche-vscode extension, and added a new GitHub Actions workflow to simplify publishing of future versions! The new release doesn’t have any feature changes, but bumps the versions of various dependencies (and additionally now requires a newer version of VS Code, hence why it was a bump to v0.2.0).

Also, we published the VS Code extension in the Open VSX Registry (although at the time of writing, our namespace is not yet verified, track this issue for progress). This means you can now use Brioche in other VS Code-based editors, such as VSCodium!

Infrastructure updates

In our homelab, we have an M1 Mac mini that has gone largely unused since we got it. I finally decided to provision it with Asahi Linux, which means I now have a powerful machine for doing native ARM64 Linux builds locally! This represents the first tangible work towards getting Brioche supporting cross-platform builds— first up will be supporting native ARM64 (aarch64) Linux builds, followed by supporting ARM64-to-x86_64 and x86_64-to-ARM64 cross builds. The Asahi-ified Mac mini will be how I’ll work on builds, and it’ll eventually become a GitHub Actions runner for the brioche-packages repo.

Coming soon

Here’s my shortlist of features that I’m semi-planning to work on soon: