Accepted for GSOC 2010, XBMC acceleration

Well the students and projects have been announced for the GSOC 2010, and the beagleboard project ended up with 6 slots, which is quite good considering it is their first year (and I was expecting only 4).

And one of the projects I put my name down as a mentor made the cut - the XBMC acceleration project which seems to have been a favourite amongst all potential mentors. Since we knew what projects were in the top running I have already had some contact with Tobias in preparation, and he seems both keen and also good at communicating with the XBMC project which will of course be vital. Well I'm not sure what I will have to do to help as it is not a project I've worked on but I guess I'll learn as I go too.

I have also been trying to build XBMC on a beagleboard (silly I know - I need to set up a cross-dev environment for it) - wow, what a beast of an app. I spent about 3 days just setting up a new angstrom and the build dependencies (I just did a bit when I walked past and noticed it'd finished the last bit/failed again). Very slow - also doing it over NFS so that probably isn't helping. And I don't really need to, as I'm using my old PS3 hard drive as the system drive - it's much faster/more reliable than flash cards are.

Just as well I don't have 'a life' - i've got plenty of time to play with this stuff without any distractions.

PS Sorry to those who didn't make the cut this year - there were a lot of applications and plenty of competition.

Java n i/o

I needed to write an application to process very large files, and it had to run on M$ stuff. Well for me that means Java since my workstation is GNU.

I would normally use C, and would definitely use GNU/Linux or some other system with a modern filesystem on a task like this - NTFS is so god-awful slow. But that wasn't my choice.

Just to make sure i/o would be as minimal a bottleneck as possible I wrote a very simple async i/o loop - I don't even know if makes a lot of difference because Linux does quite aggressive read-ahead anyway, but it certainly doesn't hurt it. Java 7 is supposed to have an async i/o interface that maps more directly to the OS one, but that isn't available so I just rolled my own.

I used the 'nio' interfaces - just for the buffers really to avoid excess copying. Oddly searching for 'async i/o' brings them up but they only have a non-blocking select type thing which has no use in this application.

It's pretty simple anyway. First a bit of setup in the processing thread.

        final ByteBuffer sentinal = ByteBuffer.allocate(0);
        final ArrayBlockingQueue queue = new ArrayBlockingQueue(numBuffers+1);
        final ArrayBlockingQueue done = new ArrayBlockingQueue(numBuffers+1);

        // must be one less than queue size - room for sentinal
        for (int i = 0; i < numBuffers; i++) {
            done.add(ByteBuffer.allocate(bufSize));
        }

The main idea is using the blocking queues as a read-ahead mechanism. They seem to have a fair bit of overhead but it can be easily tuned by using different sized blocks.

The sentinal is used to mark end of file.

Then a thread is setup with a simple loop:

                    while (true) {
                        ByteBuffer bin = done.take();
                        bin.rewind();
                        int len = fc.read(bin);

                        if (len > 0) {
                            bin.limit(bin.position());
                            bin.position(0);
                            queue.add(bin);
                        } else if (len == -1) {
                            queue.add(sentinal);
                            break;
                        } else {
                            done.add(bin);
                        }
                    }

It took me a little while to realise how to use the ByteBuffer properly - to reset it after reading so the processing loop can read the same data, but it's pretty straightforward.

The processing loop is then simply:

            while ((bb = queue.take()) != sentinal) {
                ... do stuff ...
                bb.clear();
                done.add(bb);
            }

The processing loop can get a bit 'hairy' if you want to avoid copying data around too much, but this is the same in any language.

For comparison I whipped up a simple C version that did the same processing but with just the Linux read-ahead. The overall processing time (taking out java startup) is identical because - surprise surprise - the system is completely i/o bound. The Linux read-ahead is obviously quite aggressive too - which might not be so good if you're doing more random access.

The C version used 1/10th of the memory and about 1/4 of the CPU time ... but it had a better inner-loop, and the Java has a nice GUI as well. The C one only took about 20 minutes to write vs a few hours for Java - but I'm more familiar with C, and particularly writing this sort of stream processor to run efficiently in C.

Anyway it was an interesting diversion from my current work, and a good learning opportunity.

Another week down

Blah. Been a long week - got a bit caught up with hacking away so had some pretty late nights, and then early mornings from phone calls or visitors. Hmm.

Some friends are in town from OS next week so that should be nice, hopefully I catch up with them and their new twins. I'd been trying to call for the last few weeks - after not having seen them for about a year - and finally got through at some silly time on Saturday morning. We also tried the PS3 video phone application (alas, i upgraded my firmware in the end ...) which wasn't too bad - a bit like those old 'hello from space' videos, but a bit clearer and sometimes the framerate was very good. Audio delay was noticeable though. First time i'e used a video phone, and it's a bit weird talking to someone through a window in another lounge room (maybe all the stranger stranger for it being 8am in the morning after being up all night). Didn't know where to look.

We've just had some great weather and it's a pity it looks like they'll arrive on ANZAC day - which is typically the obvious start to winter, and it looks like this this year will be a typical year. After hibernating for a Chicago winter i'm sure the week of 25+ we just had would've been nice.

On the international traveller front it sounds like my sister has had enough of the ignorance and absurdity of the country, and sick of not being able to get decent work and is heading back from America fairly soon. She's only been away for a few years but she might get a bit of a shock to see how far down that path we have also gone. Even if she doesn't decide to stay long, I'm hoping she cleans up my 'other house' that's sat empty for a couple of years collecting dust anyway!

Speaking of ANZAC day - that's a pretty good example of how the nation has become a bit feral. How a total rout and a day meant to remember the utter stupidity of war got turned into a day of nationhood and the glorification of `fighting for ones country' i'll never know. I always thought it was about remembering the poor diggers who were mis-used as pawns in political games or by incompetent chains of command, and the 'least we forget' was about the stupidity of sending them in the first place, not the 'honour of their sacrifice' - typical right wing bullshit. Now it all seems to be about draping yourself in the flag (a flagrant violation of the honour of any nation's flag - and they didn't even fight under that flag anyway - it was all as British subjects) in a sort of quasi-national day and thinking of it as the day 'we grew up' - I can't really say if it was, but we surely have 'grown down' since if this is the thought of the common man these days.

Hello world

Well I integrated a timer interrupt into WoofƆs and got that to work eventually - the MMU makes things like this very fun as I have to map everything I need to use. A few silly bugs in the MMU code didn't help either.

After a 7 hour stint last night I finally booted up an executed a 'user process', including sending it the initial welcome message. It doesn't do much other than call a debug printf function to say hello, but given that it is running in it's own address space successfully that seems a reasonable achievement.

Interrupts are going to be a bit of a pain. Because I want user code to deal with interrupts the kernel has to clear the hardware interrupt status bits itself - which means it needs to know about all the devices that might produce interrupts and how to turn them off. I can't see any way around this right now. I also need some way of letting the user-code know what triggered the interrupt, which is another issue - even with 96 interrupts there is some double-up on the interrupt source. I guess there's always the mailbox idea, or a syscall to query it. There might be some hardware where the interrupt needs processing immediately too, but i'll come to that when I find it.

Hmm, next thing might be to think about devices and get a serial device working.

Timers

Finally got timers `worked out' - i'd played with it before but couldn't get it to work - but it turned out that the code was basically fine, but I had a bug in the irq `enable' routine ... der. They are really rather simple devices that work much the same in almost all microprocessors ever made ..., so it was a bit embarrassing it took this long.

It's using the 32 768Hz clock - which means you have to use some extra hardware logic to get a true 1Khz divisor, but the manual explains that pretty clearly as referenced in the source.

Source is in irq-timer.c.

I'm at the point in WoofƆs that I need at least one timer to get much further so it seemed a good time to poke it into puppybits. I managed to squeeze in a little time to play with WoofƆs over the last few nights and got the code that i'd written a few days ago to load and run - after a stack of mucking about as usual. It doesn't do much - with the trivial round-robin scheduler and no interrupts once the memserver task goes to sleep the idle task starts and is never pre-empted :). I had some linking issues, but the main problem was that for some silly reason I was calling some assembly from C as a function *pointer* not a function and it took me far too long to spot it.

But there's an excuse this time. It's been a really long week - 49 hours of work, quite a few very late nights, an on-site meeting I had to wake up at 7am for, a couple of midnight meetings for GSOC - one of which I missed because I fell asleep, and then I woke up at 2am and didn't crash again till after 8am this morning. Not sure 'tired' covers it. The sunrise wasn't bad though.

Guess which application?

"I have fixed the score on that particular proposal to match the score allotted to it. It is a known issue that occurs randomly sometimes because this particular action is not run in a transaction ."

No comment necessary.

ARGH!

Damnit. Damnit. I need a beer. Oh hang on, I already have one.

Turns out that the algorithm I was converting ... was broken on the C end. No matter how much I tried testing the individual parts, when I turned on the full processing loop it just didn't match. After re-implementing the entire algorithm in both java and c (removing all the layers of abstraction i'd added) they all just checked out - but both produced crap - the original just wasn't doing the same thing at all. Well that was pretty much 2 days wasted, ending in a nasty headache at that.

Well probably more time wasted, as the algorithm I thought actually did something wasn't doing anything useful after-all. Well at least I can re-focus on that, and not waste my time.

Along the way I had a bit of a fight with the swing layout classes. Oddly enough the main memory I have from very first GUI programming I did in uni was how much we all had to fight with the toolkit layout system (well, after just how dreadfully slow it was - something beyond the complexity of gtk but running on a single sparc with 30 other students). Layout always seems to be an issue ... although TBH I think Gtk+ did a reasonable job for the most part - but then again 6 years of programming in it probably helped me forget it's weirdness. I kind of have it working but now when I use a different theme the content renders over the pane border.

Mid-afternoon being stumped I did some work in the yard - clearing some crap out of the way so I can level things off, and started said levelling, so at least my day wasn't a total loss. Although I think I have a bit more earth than I thought I did, so I might have to find somewhere to put it ... I have a rectangle of space I want to put a raised garden bed in, so I might have to look at putting a wall around that now too to hold some of the excess dirt (i'd hoped to leave that for later). I had a dead apricot tree there which i'd almost hacked out (without an axe), and today I managed to break the rest of the roots with some brute force so maybe i've just made myself a heap more work to do! While laying a string-line to get the levels I pinched my finger with a retaining wall block - barely hurt at the time but now I have a huge blood blister on it.

Hmm, the beer's finally starting to work. For a bit there I thought I might need something a bit stronger ...

Melange stinks

So the GSOC project is run on this half-arsed pile of shit called `melange' ('the spice of creation' / 'soc' / geddit?). I've been so utterly and literally dumbfounded by just how poor this application is that I simply haven't been able to find the words to describe it, but I think i'm finally willing to give it a go. But I wont do it on the mailing lists because someone will get offended by the naked truth, and there's already been enough of a shitstorm any time it's mentioned.

... half-arsed

pile of shit

called `melange'

It's very very slow, very ugly, lacks almost all of the basic features you'd expect in any web application developed since about1998, uses javascript where it doesn't need it - interfering with basic browser operation, and is generally an exercise in frustration trying to use it to do basic tasks. And on top of all that the project maintainers seem to be yahoos who like to do things like drop in new versions in a live system in the middle of critical periods of the process, and don't see 6 hours of downtime during said critical limited-time phase as any sort of problem.

Yet when anyone criticises it, the tired old canards of 'send us patches then' 'give us feature requests', and so on get dragged out - at best. Or they get all defensive and agro about it. I believe it's written in Python, so that vastly limits the pool of competent programmers even able to send patches. I believe it runs on googles app server which further limits the programmer pool - both for physical reasons and political ones ('fog computing' et al). And since the application is so poor at demonstrating its basic worth or indeed raison d'être - just why would anyone bother? The latter point is of particular note when it comes to feature requests - if they can't even manage the basics well (after two years?), I have zero confidence they could ever get anything more complex achieved.

This is not some toy project someone is doing in their spare time - this is a production system being used by thousands of people relying on it to ease what is a pretty difficult and tedious process to start with, and to decide how to dole out millions of dollars of real money. It shouldn't be getting in the way at every turn. We shouldn't all be beta testing what is barely a beta-level prototype at this point - this is the second year of it's use, and if that's all I came up with after all that time i'd be seriously consider whether I was capable of ever getting it to work. Even after just one day of failures (like yesterday) i'm questioning my own path.

Some of the mentors were discussing the provision of basic features like history and version control a-la-every-other-document-management-system-since-1970 and Chris Dibona had the inanity to suggest the application needs to `get simpler', not add features! What sort of nonsense is that? The app is difficult to use because it is too simple, remove any one `feature' and it would be utterly worthless - it really does the absolute bare minimum, and does that poorly.

I think what most frustrates me is this attitude more than the application itself - one, that being 'open sauce' you should expect to be a permanent beta-tester of alpha-grade software, and by implication that 'open sauce software' is always barely usable crap (which is clearly quite FUDish). Two, that you're all so privileged that Google is allowing you to bask in their glorious name in any way that you should be thankful for anything they do provide. Third, that is basically works so what are you complaining about. And finally that it's better than what they had before even if it's only because it's not proprietary.

It might not be a Google product, but their name is all over this whole thing and all it does is tarnish that name. As it does of their application server. I just hope I don't have to continue to deal with it beyond the acceptance stage (if I end up being a mentor), but I fear it is a critical component of the entire programme.