- AI

Yet another mp4 parser (& serializer)

2026-04-06T00:00:00+00:00

Introduction</h2>
Let me introduce `mp4-edit</code></a>, a Rust crate I’ve been working on.`
`While it’s not yet stable, I do intent to keep working on it until, well, is anything ever done? And it is usable if you’re okay with the API probably changing quite a bit (I’m not satisfied with where it’s at) and it maybe panicking in some cases (I haven’t done any fuzz testing` `yet</a>).`
My goals with it are to be lossless when round-tripping files (no data is implicitly dropped, and output bytes equal input bytes unless data is changed for the most part), and to provide tools for transforming mp4s as efficiently as possible. Currently that includes streaming metadata edits while maintaining fast-start, adding a chapter list, and trimming without re-encoding via dropping samples (this is still experimental and only works when retaining a single AAC audio track). It’s actually not that much slower than ffmpeg</code> when using the release build for these tasks (on my machine anyways 🙃).

Back story</h2> I’ve been working on a side project to play around with managing my audiobooks. While I could just use some python scripts and something like audiobookshelf</a> and be done with it, there’s a part of me that’s deeply unsatisfied by that. My brain has been hooked on Rust for some time. There’s something about having such an expressive and yet very strict type system to work with that makes everything fun. So I decided to see how hard it would be to build the entire project in pure Rust. Not so much out of practicality, of which there is some, but because I don’t have any deadlines and I’d like to explore. Speaking of exploration, this post is about a side quest I went on that led to yet another mp4 parser/serializer on crates.io. When I started the project I didn’t really know much of anything about mp4s. I’ve used them a lot. I knew audiobooks tend to be in this format (with the .m4b</code> extension). And since ffmpeg</code>/libav</code> feels like a bit much to depend on when I’m not planning on doing any re-encoding (and violates my goal of pure Rust for the project) I set about probing the ecosystem to see what’s out there. There are a number of crates that can parse mp4s, and even a few that can write them back out. But the ones I tried choked on some inputs I fed them, and also discarded atoms (also called boxes) they didn’t support. After messing around with one of them a bit, I did get it to parse all the files I fed it, but I wasn’t getting a clear enough view into them to understand what was happening — and I really wanted to know about everything inside of them — so I set out to see how hard it’d be to build my own mp4 parser. This isn’t an admonishment of any of the crates I looked at, and I may very well have been better off using one of them as a starting point. I somehow missed mp4-atom</a> in my initial evaluation and from the looks of things that could have saved me some time and effort; It may become a dependency at some point in the future. The process</h2> The first hurtle I ran into is the lack of documentation on the public internet. There was of course the existing parsers. But since it was 2025 and I also wanted to answer the question of “just how capable are these LLMs?” (aka “what problems can be solved with next token prediction?”), I utilized some free credits and vibe coded something that could let me see inside the mp4 files to understand them better (turns out getting access to information not on the public internet is something these things can be useful for? 👀). But don’t worry too much, I’ve re-written basically all of it myself at this point. I think it’s important to note here that — aside from all the associated harms1</a> — one can’t simply trust the output of an LLM. No matter how many of the outputs turn out to be true, all of them need to be checked. At the start of this I was just using a single audiobook file and critical thinking as my reality check: i.e. does the vibe coded parser parse the file, and does the output of that look plausible enough to be correct? There were of course things I didn’t catch right away because I didn’t know enough about the domain I was working on or have any definitive source I could reference beyond looking at other codebases. The commit history is a testament to that. I’ve learned a lot working on this project, and not just about mp4s. It’s allowed me to grow my skills as a Rust developer, and to have a better understanding of what LLMs can be useful for vs where they get in the way. When I started this project I let the LLMs loose in small bursts and not a whole lot of course correction. Do this specific thing. Then this one. etc. And there were enough mp4 parsers in the training data for it to make something that sort of worked, but not something I’d want to ship to production. I could have spent more time cleaning this up incrementally, but was more interested in other aspects of the project. Once I’d gained enough understanding of the domain and decided it was worth continuing, I went back and started cleaning things up. It was more appealing at this point to rewrite all the atom parsers myself than to refactor them, but I still used LLMs for access to the spec. I found this worked much better. And while I was able to get fairly consistent output for most of the atoms, I did need to reference other parsers in some cases, and am still unsure of correctness. There’s only so much the tests can ease my mind — I will always doubt information I got from a probability machine. I think I’d be a lot happier if I’d just read the actual spec. There were a number of strategies I used to work towards correctness. At first it was “does the parser blow up when I give it this file?”. But once I got far enough with that, I added in round trip tests for each atom, and then snapshot testing with full files. And of course, “does the output file play?” and “is it’s chapter track usable?”. Writing this post prompted me to review the full commit log. I’d done a quick pass of that already before making the repo public, but not in as much detail as I’d have liked to. The commit history is quite messy and I found a few artifacts that shouldn’t have been there, which I had to rewrite the history to remove (oops!). When I started this project I didn’t intent to ship anything — I thought I’d be transferring what I’d learned back to working with one of the other crates (or a fork thereof) — but it worked out to be easier to just continue on with this one. The process would look very different if I was starting from the beginning again today (even if I somehow deleted everything I know about mp4s from my brain). The interesting bits</h2> As fun as it was writing the parsing logic — and I did genuinely have fun doing that aside from all the LLM wrangling — there are two pieces of the puzzle that I found especially interesting, and they both have to do with how mp4s map time to byte offsets. First, if you’re not already familiar with the structure of an mp4, suffice it to say there’s a series of chunks called atoms or boxes (both mean the same thing). Each atom starts with a header advertising it’s size and type (four bytes known as a fourcc</code>), followed by size</code> number of bytes which are to be parsed according to it’s type</code> (which may be another series of atoms). The actual audio/video/subtitle/etc data of the file will be in the mdat</code> atom, which is either located before or after all the other (metadata) ones. I’ll cover more details in the following sections, but let’s get an overview of how time maps to byte offsets. Nested under a few levels of containers, each trak</code> in the mp4 metadata will have an stbl</code> (sample table) atom that contains atoms to do with navigating samples for the track. The common ones are stsd</code>, stts</code>, stsc</code>, stsz</code>, and stco</code>/co64</code>, but there are others such as stss</code> and sttp</code> for dealing with keyframes (e.g. in video tracks). The ones we’ll be discussing here are: stts</code> — Time to sample mapping, each entry contains the duration of the next n</code> samples.</li> stsc</code> — Sample to chunk mapping, each entry contains the first chunk it applies to and how many samples are in each chunk until the next entry.</li> stsz</code> — Sample size mapping, each entry is simply the size of the sample at the same index. There’s also a shorthand size and count for if they’re all the same size.</li> stco</code> — Chunk offset mapping, each entry is an absolute byte offset in the file that each chunk starts at.</li> co64</code> — Exactly the same as stco</code> except that each offset is 64 bit vs 32 bit.</li> </ul> Adjusting chunk offsets</h3> As you’ve probably noticed, if we have our metadata at the start of the file — which is necessary to be able to start playing a partially loaded file (e.g. from a network stream) — then any change to the metadata that doesn’t output the exact same number of bytes will corrupt the file unless the chunk offset mapping is also updated accordingly. There’s a few approaches here, and most use multiple passes on the file. For example, one might write the metadata at the end of the file so chunk offsets can be relative to the start of the file; And if you care about it being fast-start, then another pass would move the metadata ahead of mdat</code>. Then you don’t have to care about keeping track of the byte delta of the change(s) you’re making to the metadata. But is it possible to do that in a single pass? I really wanted to enable editing while streaming file bytes over the network and have it be immediately playable on the other end. We just need to know the byte delta for each change we make each metadata atom and apply that to the chunk offsets. I decided to keep things simple here and waste a bit of memory and CPU. Looking at the size of the metadata in the files I’d been testing with, I noticed that the largest thing in there is the thumbnail image (in an ilst</code> entry), and I’m already loading the entirety of it into memory. So rather than meticulously keeping track of the size of every change, I decided to serialize it all two extra times to determine it’s size and then apply that to the chunk offsets (the reason for the second time is if stco</code> becomes co64</code>) before serializing again and writing to the stream/file. I’ll probably explore a more efficient approach at some point so we’re not wasting compute, but everything will break if there’s just one bit out of place. The fun part of this was using the power of Rust iterators to calculate the size of each chunk from stsc</code> and stsz</code>, and then apply that given the mdat</code> offset (which we now know). Not that difficult of a problem, but Rust makes everything fun! (I get a lot of dopamine making the compiler happy, okay?!) The next step was to take into account the original chunk ordering, and again it’s not the most efficient solution possible, but it works. Go check it out</a> if you’re curious what the code looks like. Trimming audio samples</h3> This piece of the puzzle was probably the most fun for me, and at first glance seems very simple. Even more so if we only care about the audio track (in my audiobook project I strip out all other tracks since I rebuild the chapter track and don’t care about any of the others that may be present). All we need to do is compare a series of trim ranges to a series of samples, and remove the samples that reside inside of a trim range. It’s not a perfect cut, but works pretty well, and doesn’t require any re-encoding (AAC audio samples fortunately don’t depend on each other, so we can just drop the ones we don’t want and it’ll still play just fine). What isn’t so simple though is that stts</code> entries are compressed and, while it would be fairly simple to decompress them, that approach works out to be pretty slow (around an order of magnitude slower than iterating through chunks). Instead I opted for the more meticulous, but much more efficient route of calculating what sample indices are in each entry while iterating through them. And perhaps more challenging is then sorting out which chunks need to be dropped or adjusted based on the sample indices that got removed. As with building chunk offsets in the previous section, the power of Rust iterators came to the rescue to help break the problem down into more manageable pieces. Again, go read the code</a> if you’re curious (but you’ll probably want to be using an editor with rust-analyzer</code> to help you jump around for this one). There’s still some missing test cases for this, one or two edge cases where things break (e.g. an empty trim range), and I’d like to support more than just the single audio track case (see #1</a>). You might be asking why I didn’t just use an edit list to tell the player to skip the ranges being trimmed? I did consider that — and to support multiple tracks I may still need to implement that where the samples being trimmed don’t fully align for all of them — but I really don’t like the idea of carrying around data that’s not being used, especially for larger trims, and from what I understand not all players even fully support that feature</a>. What’s next</h2> As this project moves away from being a prototype towards a dependable and maintainable library, there’s some more refactoring that needs to take place in addition to completing the trimming feature. And while this isn’t meant to be the most optimal mp4 parser out there, there’s definitely room for performance improvements. Network speeds end up being more of a limiting factor though, so I’m not too worried about it at the moment. The main thing I’d like to address though is API ergonomics. It’s currently a bottom and a middle and needs a top. It takes over 100 lines just to copy a file,</li> You need to know if a transformation should apply to the input metadata or the output metadata,</li> It’s kinda awkward to do anything other than copy samples forward (e.g. interleaving a new track would be quite involved currently),</li> And if you do anything in the wrong order everything breaks.</li> </ul> I’d like the top level API to “just work” regardless of your knowledge of mp4s, and I have some ideas around what that might look like (stay tuned). Conclusion</h2> We’ve covered a lot of ground, and yet, so little. There’s a lot that’s been left unsaid and we haven’t seen any code examples, but I hope it’s been informative nonetheless. It’s nice to have a fun side-project like this where I can experiment and really just play around with things without any hard deadlines. And while I know that can be a slippery slope towards never finishing, I think the iterative approach I’ve been taking of getting something working and then refactoring helps me not get stuck in the weeds too much. It also helps that I choose projects I care about finishing. While I do enjoy a good puzzle, the challenge alone feels empty. This also contributed to me learning that using an LLM makes things less fun for me; Tedious even; Though it did help that I initially didn’t care too much about the quality of code output beyond it working well enough. While I think it could be interesting to see how these tools have evolved since the summer of 2025, I’m not willing to pay for that. I do enjoy collaborating with humans, and even compilers, but there’s something about LLMs that suck that joy right out of me. Anyways, I did a thing, I wrote about it, and now you’ve read about it — thanks for taking the time! If you have any questions or comments, I’d love to hear about them! Please feel free to send me an email (link in the footer), drop a like or comment on whatever platform you found this on, or open an issue on the mp4-edit repo</a>. You can also subscribe via RSS if you’d like (either to everything, or any of the tags, links in <head></code> so your browser/extension should pick them up). There are many harms associated with the training and use of LLMs. I don’t want to shame anyone who are using these tools. And I don’t want to dive into it in this post (maybe soon I will). Here’s a</a> few</a> viewpoints</a> to consider if you’re interested in more discussion around that. ↩</a> </li> </ol> </section>

Recursive type state in Rust 2025-10-10T00:00:00+00:00 TL;DR; It’s possible to implement a binary tree using Rust’s trait solver. I didn’t set out to implement a binary tree. It started with a requirement to store one or more implementors of a trait in a struct. Dynamic dispatch seemed like the obvious solution, but it quickly became clear that said trait wasn’t dyn-compatible/object safe1</a>. Next up was considering using an enum with all the possible trait implementors, but then I’d have to match the concrete type for every operation which very quickly gets too complex. The use case involved composing all the implementors of the aforementioned trait to act as a single implementor, so that’s where I started—creating a struct that composes two implementors into a single one. Now I can use this structure to compose two or more implementors (and can handle the one case with an empty struct and a no-op implementation of the trait). Problem solved, right? Well, there was just one issue. I needed to construct the tree before I actually had any instances of the types implementing the trait, and then set each one later on. I would however know upfront which types could be added to the tree. Simple, right? Just wrap each type in an Option</code>, and ensure I can get a mutable reference to it when the time comes to add each instance. If only I could settle for storing each type in a well-known location in the tree, we could say problem solved. But I couldn’t settle, and was hungry for an adventure, so down the rabbit hole we went! The tree needed a setter method that would propagate down the tree until it found the right node. So, add a trait and use type state to determine which node we’re setting. Here’s my first attempt: #[derive(Default)] struct MaybeNode<T> { value: Option<T>, } impl<T> SetNode<NodeRight, T> for MaybeNode<T> { fn set_node(&mut self, value: T) { self.value = Some(value) } } impl<T> SetNode<NodeLeft, T> for MaybeNode<T> { fn set_node(&mut self, value: T) { self.value = Some(value) } } struct EitherNode<A, B> { right: A, left: B, } struct NodeRight; struct NodeLeft; pub trait SetNode<Node, T> { fn set_node(&mut self, value: T); } impl<A1, A2, B> SetNode<NodeRight, A2> for EitherNode<A1, B> where A1: SetNode<NodeRight, A2>, { fn set_node(&mut self, value: A2) { SetNode::<NodeRight, A2>::set_node(&mut self.right, value); } } impl<A, B1, B2> SetNode<NodeLeft, B2> for EitherNode<A, B1> where B1: SetNode<NodeLeft, B2>, { fn set_node(&mut self, value: B2) { SetNode::<NodeLeft, B2>::set_node(&mut self.left, value); } } fn main() -> Result<(), Box<dyn std::error::Error>> { let mut c = EitherNode { right: MaybeNode::<String>::default(), left: EitherNode { right: MaybeNode::<usize>::default(), left: EitherNode { right: MaybeNode::<u32>::default(), left: EitherNode { right: MaybeNode::<u128>::default(), left: MaybeNode::<u64>::default(), }, }, }, }; c.set_node("Hello string".to_owned()); // c.set_slot(0usize); // Error c.set_node(0u64); // c.set_slot(0u32); // Error Ok(()) } </code></pre> Notice that while we can successfully set nodes that follow a consistent path (e.g. right.right</code>, or left.left.left</code>), this simply won’t work if we try to set a path that has any deviation from that. Oops! The reason this failed is because the trait bound is too restrictive, encoding which node should match, locking us in to either left</code> or right</code> the whole way down. Naively attempting to loosen that bound would result in coherence2</a> issues, which is why we’re using type state in the first place. I found myself thinking that maybe Rust just isn’t capable of representing this sort of thing. I’m a bit embarrassed to admit that at this point I pasted my code into Claude3</a> to see if any insight could be gained from that. While it’d be easy to think there’s no harm in that, I’ve become aware of many reasons one might want to consider—which may be the discussion of another blog post. One such reason is that it can erode ones ability to think. It took me a bit of time to filter through all the confidently given and completely wrong answers it gave. While most of them were obviously wrong, I still got tripped up a bit on one that seemed like it might be true—that I’d need to define typestate for each possible path, similar to if we were working with a tuple of varying sizes4</a>. But, several (more targeted) prompts and a few threads later, there was one piece of output that eventually caught my eye which—after doing a quick search— looks like it was pulled from Sgeo</a>’s hlist crate</a>5</a>, and as it turned out, pointed me right at a solution (but perhaps a well placed nap could have done the same?—we’ll never know): // ... pub struct PathA<Next>(PhantomData<Next>); pub struct PathB<Next>(PhantomData<Next>); pub struct PathEnd; impl<T> SetNode<PathEnd, T> for MaybeNode<T> { // ... } impl<A, B, Next, T> SetNode<PathA<Next>, T> for EitherNode<A, B> where A: SetNode<Next, T>, { // ... } impl<A, B, Next, T> SetNode<PathB<Next>, T> for EitherNode<A, B> where B: SetNode<Next, T>, { // ... } </code></pre> Yay, it works! But why? It turns out that Rust’s trait solver is in fact quite powerful and capable of recursion. Setting a breakpoint for each set_node</code> fn makes it clear what’s happening. If we’re calling set_node</code> for a type located at right.right.left</code>, Rust’s trait solver can apparently iterate through each layer, finding a matching impl at each step (all at compile time of course!): SetNode<PathA< SetNode<PathA< SetNode<PathB< SetNode<PathEnd, T> >> >> >> </code></pre> You can play around with a full implementation here</a>. I even included a macro_rules!</code> to abstract away placement of each type in the tree: type MyTree = binary_tree_type![u32, u64, usize]; // vs type MyTree = EitherNode< EitherNode< Node<u32>, Node<u64>, >, Node<usize>, >; </code></pre> I’m not sure if I’ll end up using this, but it was a fun learning experience! p.s. An LLM didn’t write this, I just really like mdashes! https://doc.rust-lang.org/reference/items/traits.html#dyn-compatibility</a> ↩</a> </li> https://rustc-dev-guide.rust-lang.org/coherence.html</a> ↩</a> </li> I used Sonnet 4 on a free plan using the web interface. Better output may have been achieved through using an agent such as Claude Code, or a more powerful model such as Opus 4.1. ↩</a> </li> https://stackoverflow.com/questions/55553281/is-it-possible-to-automatically-implement-a-trait-for-any-tuple-that-is-made-up</a> ↩</a> </li> It was under a heading “Solution 5: Using HList-style approach with type-level programming”, and contained the same pattern and identifier names as in the crate. But I guess it’s possible struct There<T>(std::marker::PhantomData<T>);</code> is pretty common</a>? The code is MIT licensed, so I don’t think it’s presence there is an issue, but it can be easy to forget where the training data came from when using these models. ↩</a> </li> </ol> </section>