PDFZ stuff

Amongst other hacking, I poked around with PDFZ a bit today:

Tried to bind the outline interfaces, but either i made a big mistake or something odd is going on: the pointer I allocate in the creation function wont resolve when I go to use it (or, it resolves to NULL). Spent a long time going nowhere on that one.
Ported some of the ReaderZ code to swing and made a simple desktop PDF viewer. It renders on the fly as you pan around, and works quite well. I use another thread to load and render pages as in ReaderZ. I'm a bit dented that I couldn't get the outline stuff to work.
Played with an alternative page rendering method: rendering directly to the array of a BufferedImage using GetPrimitiveArrayCritical(). Can't tell the difference on the desktop but it might be useful on the kobo, and could use less memory anyway.
Noticed that mupdf is undergoing rapid development at the moment; i'm not trying to track it at all.

Update: I got in touch with the mupdf devs, and I found out the next release is targeted by the end of the month; this will be a good opportunity to sync up with the api whilst the project is still fresh in my mind.

Blah, ffmpeg changes and pain.

Sigh. I was looking at moving jjmpeg to ffmpeg 0.10, and along the way remove the use of deprecated APIs.

But there's a lot of pain involved here:

There's quite a lot of deprecated stuff - Well what can you do eh?
Custom streams are completely deprecated with no public API to replace them. That means AVIOStream will have to be thrown away and at best you're left with using pipes or sockets.
Some arguments are now arrays. This is a lot more of a head-fuck than you'd imagine and will require messy and inefficient code to marshal values around.
Some constructors now take in-out parameters as the object pointer. This requires a new constructor mechanism and frobbing around.
Some arguments are also in-out parameters. Hello CORBA style holder arguments, or some other mechanism (the Holder type seems the easiest though) and a lot of fuffing about with jni callbacks to access them. Also, this cannot be inferred from the prototype alone, so I also need to expand the generator for that.

The last 3 not being in libav* was one of the reasons I tackled jjmpeg in the first place, it was a nice clean api using fairly consistent conventions. This made it quite simple to bind.

Tuning ...

Had a poke at some performance tuning of jjmpeg.

I took 2 videos:

PAL: A PAL DVD, half hour show.
1080p: A half hour show recorded directly with a DVB-T receiver. 1440x1080p, ~30fps, 10MB/s.

I then used JJMediaReader to scan the files and decode the video frames to their native format. I then took this frame and converted it to an RGB format using one of the tests below.

ByteBuffer: Code uses libswscale to write to an avcodec allocated frame in BGR24 format. The frame is not accessed from Java: this is the baseline performance of using a ByteBuffer, and it could be the end point if then passing the data to JOGL or JOCL.
ByteBuffer to Array: Perform the above, then use nio to copy the content to a Java byte array.
IntBuffer: Code uses libswscale to write to an avallocated frame in ABGR format. Similar to the first test, but a baseline for ABGR conversion.
IntBuffer to Array: Perform the above, then use nio to copy the content to a Java int array.
int array: Use JNI function GetPrimitiveArrayCritical, form a dummy image that points to it, and write to it directly using libswscale to ABGR format. This gives the Java end an integer array to work with directly.

In all cases the GC load was zero for reading all frames (i.e. no per-frame objects were allocated). I'm using JDK 1.7. The machine is an intel i7x980. I'm using a fairly old build of ffmpeg (version 52 of libavcodec/libavformat).

The timing results (in seconds):

Test \ Video          PAL    1440x1080p

ByteBuffer            81.5   237
ByteBuffer to byte[]  86.0   279

IntBuffer             81.3   242
IntBuffer to int[]    86     297
int[]                 81.9   242

Discussion

So ... using GetPrimitiveArrayCritical is the same speed as using a Direct ByteBuffer - but the data is faster to then access from Java as it can just be indexed.

Using RGB and ByteBuffer's is a bit quicker than using RGBA. Apart from the differences down to libswscale there seems some overhead using an IntBuffer (derived from a ByteBuffer) to write to an Int array.

Using RGB is marginally quicker than using RGBA - although that's mostly down to libswscale, and for my build nothing is accelerated. When I move to ffmpeg 0.10 I will re-check the default formats i'm using are the quick(?) ones.

When using a direct buffer and then copying the whole array to a corresponding java array, the overhead is fairly small until the video size increases to HD resolutions. At 23% for 1440x1080xABGR, it is approaching a significant amount: but this application does nothing with the data. Any processing performed will reduce this quickly. At PAL resolution it's only about 5%.

Conclusions

For modern desktop hardware, it probably doesn't really matter: the machine is fast enough that a redundant copy isn't much overhead, even at HD resolution.

Possibly of more interest is how the rest of the pipeline copes. Obviously with JOGL or JOCL the work is already done when using ByteBuffers, or ideally you'd process the YUV data yourself. I'm not sure about Java2D though, from a previous post there's a suggestion integer BufferedImage is the fastest.

However there are possibly cases where it would be beneficial and for Java image processing it is probably easier to use anyway: so I will add this new interface to jjmpeg after confirming it actually works.

I also found a bug in AVPlane where I wasn't setting the JNI-allocated ByteBuffer to native byte order. This made a big difference to the IntBuffer to int[] version (well 44% over no array copy in PAL), but wouldn't have been hit with my existing code.

Sleep n Whinge

ugh, what a crappy day. I hit the grog a bit hard last night (sister dropped by for a couple of hours on her way to the airport), and subsequently had very little sleep; and the neighbours decided today was a good day to re-start the work on the extensions next door. Had a nap about 5, at least until some dodgey scam out of India rang up about 7:30. Blah.

But I played a bit with some code during the day. I poked around with my slideshow creator, working on some more transition wipes - worked out a 'clock' transition which seemed to take much longer than it should have (for lack of inspiration I'm looking at the SMIL stuff for ideas). I was going to write a very simple front-end gui for it, but just didn't have the motivation for that today.

Then I got totally side-tracked with some other stuff: I noticed javafx builds are finally available for gnu/linux, looking at the swingx demo (there's a couple of things that look interesting), the image filters it uses. Mr Huxtable also has an interesting article about BufferedImage stuff (which i'm sure i've read before but must have forgotten about): and that got me thinking about changing the way jjmpeg's helpers work with images as it uses 3BYTE_BGR types and direct DataBuffer access.. And that got me thinking about JNIEnv.GetPrimitiveArrayCritical (to avoid 2 copies), and well by this time I was too hung-over and tired to do anything useful.

I also noticed the neighbours were building a really big verandah which will block most of the direct light into my bathroom, and they over-cut a bit of a tree that hangs over the boundary. And I got a letter from my insurance company whining about an over-charge they shouldn't have been making in the first place. All all that together with the severe lack of sleep, put me in a terrible mood and made me feel really rather miserable. And now it's 3am and they'll be at it at 7am again next to my bedroom window so tomorrow probably wont be much better ...

Update: Oh fun, 7:25am, shit radio station was bad enough, now it's with the jack-hammer.

VideoZ

I had a go at writing a simple 'media mixer' today. So far it's only video, but i'm already thinking about how to do the sound (hence some work on JOAL yesterday, I'm planning on using OpenAL-Soft to do the mixing, which gives me '3d sound' for free as well). Sound is a bit more difficult than video ...

As output it generates an encoded video file; using jjmpeg of course.

With a small amount of code i've got a slideshow generator, together with affine transforms, opacity, and video or still pictures. I'm just using Java2D for all the rendering: so the compositor is fairly slow, but it's workable.

But, the biggest part of any real application such as this is the user interface for setting up the animation parameters ...

Paged layout, busy dot.

After poking around at jjmpeg a bit this morning, I played a bit more with ReaderZ. First I added an animated 'busy' icon for when the reader is busy, and moved the epub html loader to another thread so it animates. It's ugly, but it works. I simplified the use of the event manager as well.

Then I redesigned the BlockLayout code in CSZ so that I could sub-class it to create a paged media layout. It isn't 'conformant' by any stretch, and has a bug with tall images, but at least it forces lines to align to a new page once they've overflowed the viewport.

During this I realised I probably wont be able to get away with a single-pass for the layout. e.g. if you have an auto-sized box, it's size depends on things like the size of floats and the lineboxes inside of it. But you have to lay these all out before you can determine what it is, and then must lay them out again afterwards once you've determined the real size you're working with (also required for things like text-align). I might have to lay out individual words instead so then the second layout can be fast as well as letting the layout be handled separately from the text object.

jjmpeg transcoding

Well I had a go at transcoding using jjmpeg. I added the binding required to get it to work and added a new JJMediaWriter class to handle some of the details.

It doesn't work very well - many formats just crash. But at least avi with a few formats works. I presume i have some problems with the buffer sizes or some-such.

Update: A misunderstanding of the JNI api means I was getting a ByteBuffer pointing to 0, rather than a null ByteBuffer. I've fixed that up and now the transcode demo works a bit better. I'm still not flushing the decoders on close, so it isn't complete yet.

Sources:

e-reader, epub

Had a mini hack-fest today, and whipped up an e-pub backend for ReaderZ based on CSZ.

I added very basic img tag support as well, as is obvious.

Apart from the code to parse the content.opf file from the .epub archive which was fairly small, I spent the most time trying to work out a URL handler for a made-up 'epub:' protocol. I copied the way the jar: protocol handler distinguises between the base archive and the filename using "!/" - this is so that the normal url resolution mechanism work. But I also wanted to resolve by the manifest ID and I use the url fragment for that (although in hindsight I probably don't need it). But anyway in the end it wasn't much code, and having it there made everything 'just work', which was nice.

I also had to deal with all the crap XML brings along: i.e. dtd resolution.

The actual viewer is a bit unwieldy as it works as a set of html pages. So you need to pan around to read each 'page' (i.e. chapter, or whole book), and changing pages flips between the items in the spine (i.e. chapters or whole book). To do better than that I really need a paginating layout engine: which is something for later.

I have no svg support not surprisingly, so title pages which are pure svg come up a re-assuring blank.

Still a bit slow opening new chapters, but what can you do eh?

It's all been checked in to ReaderZ and CSZ.

About Me

Tags