Sprue Encoding

@makoconstruct.bsky.social

It's common that a server will need to transmit a tree of many small objects that are nested deeply inside each other.

The stupidest way of doing this would be to just send the root object, and wait for the client to say, "I don't have any of the objects referred to within this object. Obviously I need those. Send me those." And then send those objects, and then wait for the client to say that there are further dependencies they need, and then send those. This is many network round trips (as many as there are levels in the tree) and would be extremely slow.

So a good server should batch all of the objects that the client probably doesn't have and send them on the same trip. If the client already has some of those objects, no harm done (as long as you're not sending big stuff like images).

But this can be a bit space-inefficient, primarily due to the fact that the content-addresses of content-addressed objects will sometimes be longer than the object structure specification itself. Content-addresses are long, and also totally uncompressable.

But we don't need to transmit the content-address of each object, because the client can just compute them on their end. But you do need to leave the addresses within the objects, right?

Actually no, there's an alternative to sending those addresses too:

Sprue transmission!

I name sprue transmission after the casting sprue forms that you may know from plastic model building kits.

A sprue sends data that consists of many nested objects all connected together as a single nested structure. To assemble/recompute the proper structure, the client is supposed to cut the objects apart themselves, leaf-first, at the sprue joins, to figure out the content addresses needed to put together the canonical object representation. sprues are a much more compact encoding for batched nested structures.

I notice we could also use a simpler batch format that uses local ids instead of nested structure, where a local name of an object is its index within the batch list. This is slightly less efficient than sprue encoding by n_objects*2 extra bytes. So I'll probably go with sprue encoding.

makoconstruct.bsky.social
mako

@makoconstruct.bsky.social

philosophy of information systems (applied)
https://aboutmako.makopool.com

alt for foodposts: https://bsky.app/profile/makoconstruct-food.bsky.social

Post reaction in Bluesky

*To be shown as a reaction, include article link in the post or add link card

Reactions from everyone (0)