Brioche

Brioche Project Update - February 2025

Published on

By Kyle Lacy

Some highlights this month have been a major overhaul to build caching and more progress towards automatic updates!

Status report

Major overhaul (and speedup) for cached builds

Last month, I talked about progress on speeding up synced builds, which I still felt was a big win, but it involved some manual work at the packaging level. I wanted to come up with a way to improve syncing across the board without any extra manual work… which led to #179.

The problem is that even moderately complicated directory structures would take a long time to fetch from the registry— each file and each subdirectory required a separate request. Things were downloaded in parallel to make this somewhat tolerable, but the sheer number of requests made fetching from the registry pretty slow.

Now, the new cache stores each artifact as a custom archive format. For small artifacts, that’s basically it: fetch the entire archive, unpack it, and you’re done. But, once the total data in the archive crosses some threshold (2 MB currently), we break the archive up into chunks using a content-defined chunking algorithm. Doing so means that two similar artifacts can share some underlying data, reducing the total amount of data stored in the the cache. Actually, the new cache uses less storage space than the data used by the registry for the same set of artifacts!

That said, the registry isn’t going away either— it’s still used for mapping project names to project hashes. That is, when you run brioche run -r hello_world, the registry is used to determine which files to fetch from the cache for the name hello_world. But, if you were running a custom registry, you’ll either want to migrate to a cache in parallel, or move entirely to using the new cache system.

Anyway, I plan on writing more in-depth on how the new cache works and especially the details of the custom archive format soon

Cache migration

Oh, and to follow up on the cache, I migrated all existing artifacts/bakes/projects from the existing registry to the new cache, which is already configured to be used by default when building Brioche from main today. The new code will also ensure all brioche-packages builds will be uploaded both to the legacy registry and to the new cache in parallel.

For the next release, migrating over to the new cache should be completely seamless

Support for unarchiving zip files

In #176, @paricbat updated Brioche’s unarchive recipes to support unarchiving zip files! These work just like tar files, except the option to additionally decompress the file first isn’t supported (since zip files natively handle compression internally).

The changes on the std side will be published following the next Brioche release, so the following will work:

const source = Brioche.download(
  "https://github.com/brioche-dev/brioche/archive/refs/tags/v0.1.4.zip",
).unarchive("zip");

LSP fixes and tweaks

While working on the new cache, I ended up causing a regression in the LSP code…

That was enough motivation to sit down and finally write some integration tests for the LSP server. Along the way, I found and fixed some other LSP bugs while fixing the regression, which all landed in #188. There were quite a few fixes, so check the PR for details.

I also made a minor tweak to the LSP: the LSP server will no longer remove unused dependencies from the lockfile on save. The main motivation is that I felt the current behavior could be surprising if you commented then uncommented part of a file. You can read more about the change in #192.

brioche fmt fix

@asheliahut opened #190 to fix a bug with brioche fmt. If you didn’t explicitly specify a project path with -p, the command would just exit without doing anything! The PR fixed this, defaulting to the current directory if no project path is provided.

More work on auto-updates and tests

Since last month, @jaudiger has started adding a ton more test and autoUpdate scripts to tons of packages. Some of the newly-added packages have these functions too, so as we make progress on #94 and #165, a lot of packages should be ready to go on day one for automated tests and auto-updates!

Recursive import troubles

While working on adding libpsl support to curl, @jaudiger hit a case where two packages would need to import each other, which straight-up doesn’t work today.

After a lot of trial and error, we came to the conclusion that there wouldn’t be a satisfying way to work around this problem. So, I opened #175 to track work on adding circular project dependencies. I haven’t gotten very far with the implementation yet, but it’s my highest priority currently, and I’m hoping to get a PR up in short order.

Housekeeping

New packages

Since the last update, there were 9 new packages:

Brioche core updates