Frame Accurate Seeking in H.264 streams with libavformat, FFmpeg

I had the need for accurate seeking in H.264 streams using jjmpeg (FFmpeg/libavformat) and spent some time over the last few days trying to work it out. JJMediaReader has a seek that works quite well but I knew it wasn't correct as it was offsetting the read timestamp by the stream start - I can't exactly remember why I did this but I think its because it didn't work very well otherwise.

Anyway ... I finally found out why. Depending on the container, when seeking it will go to the next or equal keyframe rather than the previous or equal. Sigh. So that offset I was using just happened to be enough to work most of the time and as it reported the same start-relative time it appeared to work properly.

So I changed the seek code to use a reverse offset for the seek point, and then it will run forward to get to the correct frame. This is based on the framerate rather than a fixed time offset because some of the video I have uses low framerates. This works but can be a bit slow as it might seek back too far.

To reduce some of the redundant work I thought of using the packet byte position to read the exact packet of the closest previous keyframe. Unfortunately it seems the container formats i'm using don't support AVSEEK_FLAG_BYTE so that idea went nowhere. Fuck.

The best solution so far has been to implement a two-value seek function. The first argument is the keyframe timestamp and the second argument is the desired frame timestamp. Thus it seeks to the keyframe first and then advances by single frames until the desired frame is decoded. Short forward seeks bypass the seek step This requires an external index but the application in question needed it anyway as it requires a wall-clock-time to frame-time mapping as well.

Summary

• If you know the timestamp is a keyframe, use av_seek_file().
• Otherwise, seek to a timestamp which is at least a few frames before the desired timestamp and then step forwards.