jjmpeg android hardware decoding

I somewhat embarassingly just discovered that FFmpeg already has some support for using libstagefright for hardware video decoding on Android.

I always thought this was the place it should go, since mucking around with OMX or other junk is just such a pain, and it belongs there.

Bit of a headfuck getting it to compile, particularly since this adds a C++ dependency, and then after all that ... it just crashes.

I/ffmpeg  (10484): Assertion i < avci->buffer_count failed at
   /home/notzed/svn/jjmpeg-1.0/jjmpeg-core/jni/ffmpeg-1.0/libavcodec/utils.c:603
F/libc    (10484): Fatal signal 11 (SIGSEGV) at 0xdeadbaad (code=1), thread 10607 (VideoDecoder)
I/DEBUG   ( 1894): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG   ( 1894): Build fingerprint: 'samsung/m0zs/m0:4.1.2/JZO54K/I9300ZSEMB1:user/release-keys'
I/DEBUG   ( 1894): pid: 10484, tid: 10607, name: VideoDecoder  >>> au.notzed.jjmpeg <<<
I/DEBUG   ( 1894): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadbaad
I/DEBUG   ( 1894):     r0 00000027  r1 deadbaad  r2 40126b0c  r3 00000000
I/DEBUG   ( 1894):     r4 00000000  r5 620f2a04  r6 60bfcc00  r7 61d0b8b0
I/DEBUG   ( 1894):     r8 5b127c80  r9 61bd5718  sl 000104d5  fp 61bd5718
I/DEBUG   ( 1894):     ip 5dc4bcbc  sp 620f2a00  lr 400f8c65  pc 400f52fe  cpsr 60000030
I/DEBUG   ( 1894):     d0  0000000200000000  d1  4b0000004b000021
I/DEBUG   ( 1894):     d2  000000094b000021  d3  0000000000000000
I/DEBUG   ( 1894):     d4  3ce5999061da5385  d5  3f34e1653f349a33
I/DEBUG   ( 1894):     d6  3f35287b3f356f75  d7  0106999ec0000000
I/DEBUG   ( 1894):     d8  0000000000000000  d9  0000000000000000
I/DEBUG   ( 1894):     d10 0000000000000000  d11 0000000000000000
I/DEBUG   ( 1894):     d12 0000000000000000  d13 0000000000000000
I/DEBUG   ( 1894):     d14 0000000000000000  d15 0000000000000000
I/DEBUG   ( 1894):     d16 8000000000000000  d17 ffffffffffffffff
I/DEBUG   ( 1894):     d18 416347d4c0000000  d19 3fe0000000000000
I/DEBUG   ( 1894):     d20 3fe0000000000870  d21 0000000000000000
I/DEBUG   ( 1894):     d22 0000000000000000  d23 0000000000000000
I/DEBUG   ( 1894):     d24 0000000000000000  d25 0000000000000000
I/DEBUG   ( 1894):     d26 0000000000000000  d27 0000000000000000
I/DEBUG   ( 1894):     d28 3f3504f3bf3504f3  d29 bf3504f33f3504f3
I/DEBUG   ( 1894):     d30 0000000000000000  d31 3f3504f33f3504f3
I/DEBUG   ( 1894):     scr 20000010

So yeah, i dunno. Perhaps it is a bug in jjplayer, but I tried removing all output handling, and it still just crashes inside codec->decode(), and the api to that point is so simple I don't think I can screw it up.

I've wasted half a day on this and it's losing it's interest very fast ...

Update: So, just when I was about to give up, I found a threading bug in the codec implementation. This at least lets it decode more frames but it's still crashing later on inside the GLES library.

I haven't got the NV21 image format working properly yet, so it may be related to that, maybe.

Update: Well that was pretty much a full day down the drain, I submitted a patch to the ffmpeg-devel mailing list (yay, subscribe to a high volume list of zero interest to me just to submit a 10 line patch).

It turned out that the file I was testing against refuses to play at all in the bundled players, so it's something to do with the hardware/firmware. At best they load the first frame then abort, although they don't segfault (probably the parser/demux stops it going as far as the codec).

On a video recorded from the camera it seems to work ok, although it certainly doesn't appear particularly smooth - I would need to do some timings to see what's going on. I haven't written the NV21 support either. And the build stuff needs cleaning up and parameterising before I can check it in.

If it's going to still be so useless i'm not sure I really care to be honest. And I've had more than enough for today so whatever I decide will have to wait.

duckduckgo

I've decided that i'll be giving duckduckgo a 'go' for a while for my search.

I've been an extremely heavy user of google's search for many years, and it is an indispensible tool for saving time. But i'm getting sick of it trying to be 'smarter than me' - I pretty much have to manually tell it to do a verbatim search every fucking time I search for anything technical (i.e. almost every search) which makes every new search that bit more tedious and frustrating than it already is. And given their spate of recent prunings who knows how long that will even last.

And just in the last couple of days they seem to have fucked around with it even more: i get 4-5 results and then a 'similar searches' section, which I do not find useful whatsoever. Sadly I often find searches I do pointing back to my own posts or code, which indicates to me that perhaps I really am a beautiful and unique snowflake after-all, and subsequently knowing what is "popular" isn't terribly useful to me.

Targetted search based on seeker popularity by default really seems a pretty strange feature unless you're writing a search engine for pop-culture or a shop (insight: oh hang on, that must be exactly what they are doing! silly me). Searches should first be for facts, not for reinforcing currently popular interests or fasionable trends. I already felt extremely uncomfortable with the fact that google search was customising results at all, but this is so much worse. If you went to a researcher and asked for detail on a topic, would you want them to take into account both your appearance and other customer's desires when they fulfilled the request, or would you prefer an unbiased, objective collection of all available information?

I guess those of us working niche fields or with niche interestes are going to have to get used to the fact that we are simply not enough of a product-base for Google to care about (remembering we are the product). They don't really care if they lose us because their customers don't really care if google loses us. Based on "The story so far", I think we can expect to see more of a mainstream/pop/fashion based focus to their products, which is very sad turn of events. With the closing down of various networked services, we're also going to start to lose trust in their long-term commitment to some of them. Surely blogger will not live outside of Google+ for much longer (tbh, not sure i care, so long as google+ is linkable and crawlable by any other search engine), and what about google code?

I also finally worked out one major reason why firefox was constantly using a good chunk of cpu time: apart from adverts and other annoyingly pointless crap, the main culprit was simply the google search results page. Which I find quite baffling because there are no visually moving parts whatsoever. I really wish firefox would simple disable javascript on tabs you're not currently looking at, because all it does is let crappy web page coders make firefox look like a bloated piece of shit, even if it isn't really.

Duckduckgo is certainly more concise, perhaps overly so to the point of missing relavant information, so it may not fullfill my requirements. A search for jjmpeg only finds 1 post from this blog for instance, and I often prefer to see multiple in-site references (perhaps it is an option: at this point i've used it only a few times). But i'll see how it goes, my search needs vary depending on what i'm doing. I just wish it paged the results using numbers mind you, I'm using the html version since the infinitely scrolling "Web 2.1.x(tm)" page completely sucks: as they all do.

Bummed out, or am i?

This week I've been experimenting with the performance of some NEON code. It is from an algorithm which was developed in OpenCL for desktop GPUs and then downscaled to fit on a beagleboard (only for development purposes). The overall algorithm is identical but the way some of the steps are implemented is different (and for some significant components much less computationally and bandwidth intensive).

The OpenCL code took many months to develop - although that included dead-ends, multiple steps if refinement, and other distractions including completely unrelated work. Even with that, I'd put the effort at around 4-10x that for the NEON code.

The NEON code took a few weeks. It obviously helped immensely that the algorithm was primarily known in advance although the downscaling alterations were not. On the other hand, my total experience with NEON is far less than OpenCL and certainly C or Java in terms of hours.

One reason the NEON code was much easier to write is that because as it is so cheap to invoke, one can just concentrate on the bottlenecks, and leave the housekeeping to C. e.g. I can write a routine that processes as little as 16x16 pixels in assembly, and leave the addressing crap to C. There is also no marshalling or other api binding to worry about: the C is plain C, and the assembly is plain assembly, and even though JOCL is far far better than using the C api directly it's still quite a bit of work. As much fun as OpenCL is, it's even more fun hacking NEON because you can concentrate on the fun bits even more.

Although the OpenCL model is also based on simple kernels which should equally be simple and isolated - it isn't really quite like that in practice. All but the simplest of kernels end up turn into 64-way parallel subroutines utilising LDS, barriers, and so on. Without that you end up leaving skads of performance on the floor, so it really is necessary. Not to mention all the marshalling and boilerplate in the host-code to communicate with it. And because of the marshalling and invocation latencies pretty much everything is forced onto the GPU.

So what's the point i'm getting at?

Well after all that, the projected performance on the previously-latest-version of a popular handset is only about 5x slower than a HD7970 on a pretty beefy desktop!

Yes that 5x speedup is important enough that it is worth it and opens it up to more applications, but on a personal level i'm just totally bummed it isn't much more. It's a highly parallel and bandwidth intensive workload which should be well-suited to a GPU. Obviously opencl has the advantage that it isn't tied to a single bit of hardware. It's a pity SSE sucks so much otherwise it would be interesting to see how a desktop cpu fared on it's own.

I plan to "back port" the algorithms so it can be improved on the GPU, but I have a fairly educated feeling that another 2000% performance isn't very likely. I will also need to use some AMD proprietary extensions, so the portability will suffer too.

I'm sure I can improve it, but I just take it as a big personal slap in the face for all the effort that's gone into it so far!

Of course, the alternative view is that ARM+NEON is the bees knees - with less effort i'm getting relatively great performance. But we all knew that so it isn't such a revelation ...

The main bottleneck on ARM cpus at the moment is the memory, and if you can utilise the cache effectively it really flies. I would really like to see how a beagleboard like machine with big/little A15/A7 quad core and much faster memory would fare, all these cheap android dongles are far too constrained by their form-factor.

Update: Well I might need to eat my words here. Today I started to look at GPU optimisations based on what i'd learnt from the ARM experience and trying to reduce the bottlenecks of the GPU code.

The key word of the day: batching.

One reason I wasn't previously batching the processing is because it didn't really fit the data-flow of an earlier application. But I have now achieved something like a 50x boost in one key algorithm by a combination of batching the work more aggressively and some other significant algorithmic changes.

This is more like it. No longer bummed out ...

Another pointless bruhaha at an IT conference

I just have 2 words to say: Google Glass.

Better get used to it, and more.

#$#@$ Android

One of those "I can feel the hair going grey and falling out" mornings. Although at least it wasn't one of those "I made every mistake I could have, every time I did anything" one. My throat is hoarse from all the screaming of definitely un-pc obscenities.

I'm trying to get some jjmpeg stuff working on android - but I can't use the jjmpeg android branch due to licensing issues. I need to use a fully shared library version.

Compiling the code was easy enough but trying to use shared libraries on Android turned out to be a complete head-fuck.

First, for some fucked reason although shared libraries are placed into a per-architecture location, the bloody library path isn't setup to automatically use it. So you have to hard-code all the architecture into the dlopen() calls.

Actually second, it's even worse than that, you need to use absolute paths for dlopen() otherwise it doesn't work. I just hard-coded it, a stack overflow answer stated that finding the path from Java was 'trivial', but then neglected to include which api trivially provided it. I didn't really want to change jjmpeg anyway.

And thirdly, it's flaccid linker doesn't support versioning of the shared libraries. To fix that I hacked up ffmpeg/library.mak to only create non-versioned sonames, and then hacked a build script to make it compile properly as with that change it didn't seem to want to build the libraries properly. I did this by specifying each lib*/*.so manually as the build target.

And the fun's only just started, i've still got a pile of C and NEON code I've got to get to work ...

Update: Oh fucking joy, no complex numbers now.

Update 2: Well I managed to replace complex.h with a few lines of #defines - fortunately they're complex numbers are in the compiler, just not in libm, and apart from the basic arithmetic I only needed creal/cimag and conjf.

And somehow, after re-arranging 1500 lines of C and 1500 lines of NEON assembly language, creating JNI wrappers, and hooking it into ffts and jjmpeg/head ... it worked almost first time. I'd set myself all week to get that far so i'm pretty chuffed, even if the headache from the days earlier tribulations hasn't yet receded.

I really have the beagleboard and a GNU operating system to thank for that - without being able to develop on a proper system first it would've been a nightmare.

Time for a couple of glasses of nice Barossa red methinks.

clAmdFft

Just a quick note, I created a small JNI wrapper using JOCL to access the clAmdFft library from AMD, to see how it compared to the Apple FFT I ported to Java.

Bummer, the Apple FFT is still faster for my problem size - image processing. Kernel time on a 1024x1024 sp fft is ~0.5ms vs ~0.38ms for the Apple one (HD7970). And the apple one only requires 3 passes. Smaller and/or batches seems to scale linearly too.

All that (minor) frustration with the clMiXeDAbbrCamlCase API for nought.

Update: So I actually tried writing a 64-element parallel FFT to see how I could do ... well, not good. Apart from wasting a lot of time wondering why "no matter what I did it took 15ms" because I hadn't allocated a properly sized buffer (out of range access I guess), when I finally got it going it was about 10x slower than the Apple one. Blah. Actually the apple one was about the same speed (if not faster) than a simple memcpy (using float2).

I wasn't being very smart about it, I just broke a 64-element FFT into one-calculation steps and used shared memory to communicate the results. A big overhead was the index twiddling for which I just used lookup-tables, but even the shared memory communication was expensive.

I may have gotten something wrong anyway, the profiler said the apple fft code had many many fewer global read/write operations, and I couldn't work out why. Although I was trying to do it all in LDS in a single kernel.

Since I didn't verify the results - I was just seeing how the memory access patterns would work - I have no real idea whether it worked or not anyway. But I may have another play another day, I wrote some code which outputs the expressions required between two arbitrary "layers" of the calculation so I can use larger kernels than the radix-2 ones I was testing with.

Nvidia and OpenCL

I don't particularly follow much what Nvidia are up to anymore - for the last 18 months that I was subscribed to the 'gpgpu' mailing list of theirs it never mentioned OpenCL once, so I de-subscribed a few weeks ago because it contained nothing of interest.

I've been poking around anantech/tomshardware too much recently (work-stress induced apathy mostly) and read a few articles about Nvidia's latest developer conference.

Well, no mention of OpenCL anywhere, and now they're pushing their CUDA stuff on mobile as well. The lack of OpenCL 1.2 support, and the poor showing in compute performance of their recent hardware makes it pretty obvious that they're not interested in it, but this really nails the coffin shut.

In comments people always claim that the poor OpenCL performance is just unoptimised drivers. However this doesn't hold much water - because the architecture of their driver design is such that both CUDA and OpenCL are simply thin layers above the same infrastructure. It would be like saying ``Oracle only focuses on Java, and that's why Scala isn't as fast'' - but any improvements they make to the JVM benefit Scala too.

The only alternative is too crazy to imagine, that they're deliberately knobbling the OpenCL performance significantly on purpose. Anyone investing heavily in their super-computer infrastructure would be well to take such nonsense into consideration.

Risky move?

With fairly broad support for the HSA foundation (it seems?), and a computing model which is more advanced than CUDA, can they go it alone? Well apart from their crappy controller/game thing, it doesn't seem like anyone is too interested in Tegra4, so let us all just hope that it is a no on that one.

I'm a little bummed that the priorities at work has kept me away from OpenCL for a while, I had a single day back at it after some leave but then the focus changed again. Although I might be doing some more NEON assembly soon so it's not all bad.

The farmer quest

I haven't had much time to work on anything this week, but I did get the "farmer quest" from Zabin's game implemented (or at least, the script for it). I'm a bit blank on the imagination at the moment, so by just getting old functionality working the problem is just one of filling out the implementation and script api.

This was pretty nasty in Dusk script, but is relatively straightforward in javascript. I experimented with just using the java api's a bit here too, rather than just adding script-friendly functions; it's a bit harder to use, but is more powerful.

The 'quest' is operated by walking onto a location infront of the farmer, this is just attached to the action for the location. Rather than 'bounce' the player back a square I just let them stay there - actions only trigger when you enter a location, not for standing on one. Prevents some ugly flicker in the client.

if (trigger.isPlayer()) {
    var state = trigger.getInteger("farmerquest");
    var list = trigger.getAllItems("bugcorpse");

    if (state == 0) {
        // first meeting
        trigger.chat("Farmer says: It doesn't look like my crop is going to support me this year.");
        trigger.chat("Farmer says: Those $%$# bugs ate half my crop.");
        trigger.chat("Farmer says: I have a job for you, if you could kill those bugs and bring me back their corpse, i'll reward you for each one.");
        trigger.chat("Farmer says: If you bring me 20 corpse's at once and I'll give you something special.");
        trigger.setInteger("farmerquest", 1);

Since the state can be tracked explicitly, it's easier to test at what position the script is, without resorting to returning from the script. Actually as a side-effect of the way scripts are invoked, it's actually impossible to return anyway - every script must be fully structured just like the original PASCAL. i.e. there is no 'return' or 'exit' keyword allowed, since the scripts are run as top-level code.

    } else if (state < 3 && list.size() >= 20) {
        // Check for 20 at a go
        for (var i=0;i<20;i++) {
            trigger.removeItem(list.get(i));
        }
        trigger.chat("Farmer says: That's 20 corpses, here take my Kaizer Blade for helping me.");
        trigger.setInteger("farmerquest", 3);
        trigger.addItem(game.createItem("kaizer blade"));

This bit can also just access the Java List interfaces directly to do bulk operations, as well as the wider 'game' object which has general utility functions. I do however have to make sure I design these public interfaces properly as for example the scripts are executing on another thread (eventually in a sandbox), so getting the mix right will be an ongoing experiment.

    } else if (state <3 && !list.isEmpty()) {
        trigger.removeItem(list.get(0));
        trigger.chat("Farmer says: Great, you killed another Bug, here's 30 gold for your reward.");
        trigger.addGold(30);
        trigger.setInteger("farmerquest", 2);

Simple reward is trivial to implement, it also tracks if you've done it at least once.

    } else if (state == 1) {
        trigger.chat("Farmer says: Have you collected any corpses yet?");
    } else if (state == 2) {
        trigger.chat("Farmer says: What are you doing back again, haven't you got work to do?");
    } else {
        trigger.chat("Farmer says: Thank you for your work, I think my crops will be safe for another year.");
    }
}

And finally a bit more detail for the default case since we track the state explicitly. And this version doesn't let the player keep killing the newbie enemies for easy cash once they've got the grand prize.

Update: I decided to keep up with this approach and i'm working on porting Zabin's game to the new engine bit by bit. I've imported the map just as another world which you can enter via a door and started on the tutorial. This will let me work on it in little pieces and also expose any implementation issues in manageable chunks.

e.g. When I added all the mobs from the original I hit a bug with the entity index and decided to change it completely. Fortunately as one of the first things i "fixed" was to abstract the map and map-related operations it means I have complete flexibility on how the internals are implemented.

Previously it was implemented as a 2d index storing a pointer to the first entity at a given tile location. These were then linked using a next field in the object and managed using some single-linked-list logic. For big maps with sparse entities it takes a lot of memory just for the array - 2MB just for the original dusk map. I changed it to use a hash table keyed on an x+y pair and stored using a LinkedList. Lookups will be marginally slower but it's made up for by the reduced footprint and ease of use, and because I'd abstracted the map previously I can always change it.

Update: Another thing I noticed that doorways were a pita to write. For each one you need at least an entrance and leave action, as well as an entrance and leaving location. And each action requires a one-line script. So I added another function in the map file which allows jumps without requiring a script to be run.

   x.y.goto=alias-name

It still requires 4 settings, but it doesn't need the two single-line scripts. This also means that map location aliases must now be globally unique.