Crappy Stories, Shitty Journalism, etc.

I read the tech news fairly regularly: mostly via boycottnovell.org since it's conveniently catalogued there (but not only for that reason). I occasionally drive-by comment. But today's headlines do seem a bit crappier than usual so I guess it's time for a bit of a rant. I think I finally got the hacker worked out of me for the time being too (and I need to get the house ready for a party), so that's always good for a rantfest.I wont link to the articles, check the source if you care, which you probably don't.

Is Windows 8 Metro failing even at Microsoft?

SJVN is normally worth a read, it's all a bit fluffy but at least he's (usually) fairly on track. But today SJVN's lost the plot a bit, surely an ex-developer using an Apple PC must be the end for Microsoft?!

This one almost got me to sign up to zdnet, but thankfully I decided against that - the commenting readers there are pretty rabid and like them i'm only likely to comment when pissed off.

Although clearly the 'tablet shell on a desktop pc' idea is insanity, and so is a 'desktop shell on a phone', but the main thrust of the article seems to be dissing a photograph of some ex-microsoft lad sitting at a desk adorned with an Apple PC?

Firstly, who cares what some guy i've never heard of has behind him in a picture of him seated at a desk. Even if he was still at microsoft (although I'm lead believe their culture is kind of fucked up so wouldn't allow it), it's no big deal to track the competition: actually it's quite smart. And apart from that we all know that plenty of Free Software, `open source', and Linux developers use Apple PC's.

And people in big companies leave all the time, and for software that is more likely to happen at the end of a release. This is something Roy on boycottnovell.org seems to get worked up a bit too much over as well.

FSF Wants To Police JavaScript Use

Well any article purportedly about the Free Software Foundation which mentioned `open source' in the first paragraph isn't even worth reading. So I didn't.

Can't have had much thought put into it.

iOS More Crashtastic Than Android (they've even got some stupid 'copyright' notice when trying to copy and paste the title, i decided to delete it)

This seems to be a PR release from 'Crittercism' (whomever-the-fuck they are) with almost no journalistic input.

From a magazine calling itself 'Linux Insider', you'd think they'd point out one of the main points of Android's stability: it's kernel is Linux, which has had far more resources put into it than any kernel in history. No single corporate identity could ever compete with that.

Not to mention the application layer technology which is the real issue when talking about end-user application software:

Android's application layer is based on a mature platform proven to be stable, robust, and above all: crash-resistant. Java.
iOS' is based on C. That should be `'nuff said', but not only that, some fucked up weird-arsed dialect of C, unfamiliar to most of the programming world. And we all know how robust and crash resistant C is. I really love C, and can write good C (IMNSHO), but crash-proof and robust it is clearly not.

I've told this anecdote before, but when I first started coding in Java again (after a year-long-ish stint pre-2000), I was astonished that my code didn't ever crash. Sure i could get piles of exceptions and things wouldn't function; but the application would keep plodding along tickety-boo. I had just had 3 years of c-hash, and I'd always been lead to believe that dot-net was basically just the same as Java.

And that crashed all the time. It's not as easy to crash as C which makes something of an art out of it, but it still crashes, and even what should be non-fatal errors can bring the application down with no indication of the cause.

But of course it isn't like Java at all: at the heart of it, dot-net is just a different and more dynamic linkage system for object files distributed in an intermediate format. So it still crashes like C, particularly when the platform was so immature and badly written (WPF, how I don't miss you). That's before you add the shitty memory management and crappy compiler to boot (oh man, and visual studio: i'm still astonished anybody could do more than barely-tolerate that piece of shit: some people actually seem to like it).

Is GNU/Linux just not cool anymore?

This article isn't really too bad, for what it is (and it isn't much). For for the `Free Software Magazine', they could have at least grabbed the google trends on 'free software' and 'open source'.

Particularly considering the only non-commercial searches they performed were for 'linux mint', and 'gnu/linux', and two others weren't even related to software development in any way-shape-or-form and seem to have been added just to find an up- and a down-pointing curve to add visual aesthetics to the page.

Ok, so maybe the article wasn't so good after-all.

If they had have tried `free software', at least they'd have found a fairly flat trend. Unlike `open source' which only sees steady decline (despite it being term more often showing up in discussions about security matters based on it's other, more-descriptive meaning).

Australian politics and the reporting thereof.

Plumbing the depths of irrelevance and insignificance.

At best, the reporters seem to think they're writing the society pages for Canberra socialites, or following TV celebrities (i.e. famous for no reason). At worst, it's just the local small-town gossip column.

And all the pollies only seem to be interested in making those pages as well.

Real shit happens every week that affects us all and the so-called reporters only want to talk about leadership squabbling (which seems to be made up for the most part) and Tony's dick stickers - when they're not talking about American politics that is. It's like they're writing/talking shit for their 'in-crowd' mates to chat about at their next cocktail party with the 'stars' they fawn over.

It doesn't help that Tones is a complete and utter nut-case, and Julia - just like Kev before her - only seems to want to lead from behind, making decisions and statements based on polls or some fukwit's column in the Murdoch press, or some local talkback radio station that most of the country can't even listen to ...

Apart from the 'insider' reporters and idiots like Barners, who really gives a fuck about this gossip crap?

And Crabb with her, `Cooking in the cabinet' (or whatever it's called) - a pretty ordinary looking 'celebrity host' cooking show: with active cabinet members presumably. Surely a low-point of both Australian politics and political reporting ...

(I usually just don't even bother watching the news, but I caught some earlier this evening about the never ending Rudd leadership ambitions crap and it ticked me off).

PDFZ searching

So after finding and fixing the bug in my outline binding - a very stupid paste-o - I added a table of content navigator to PDFReader, and then I had a look at search.

Which ... I managed to get working, at least as a start:

As can be seen, it's a little flaky - mupdf is adding spaces here and there in the recovered text, and i'm not sure i'm processing the EOL marker properly (and possibly I have a bug in the search trie code too). But as I said - it's a start.

I decided to use a Trie for the search (Aho-Corasick algorithm) - because I know it's an efficient algorithm, and because I know there was a good implementation in evolution. So I grabbed an old copy from the GPL sources and modified it to work on the mupdf fz_text_span code. Thanks Jeff ;-) Basically it's a state machine that can match multiple (possibly overlapping) words whilst only ever advancing the search stream one character at a time.

I tried to copy the emacs mode of searching to some extent:

/ or ctrl-s starts a search
The search updates immediately for the current page as characters are typed.
The next search on the current page is highlighted when ctrl-s is pressed again.
If there are no more results on the current page, the search starts scanning the document if ctrl-s is pressed again.
ESC closes the command prompt.

I put some code in there to abort the search if the page is changed while it's still searching (because I hooked the search into the page loader/renderer), but on the documents i've tried it on it's been so fast I haven't been able to test it ...

This is one of the first times I used JNI to create complex Java objects from C - the array of results for a given page. It turns out it's fairly clean and simple to do.

I guess the next thing is to see if i can integrate the search functionality into ReaderZ. Time to write one of those horrible on-screen keyboards I guess ...

But for now ... the weather's way too nice to be inside, so I think it's off to the garden, beer in hand ...

PDFZ stuff

Amongst other hacking, I poked around with PDFZ a bit today:

Tried to bind the outline interfaces, but either i made a big mistake or something odd is going on: the pointer I allocate in the creation function wont resolve when I go to use it (or, it resolves to NULL). Spent a long time going nowhere on that one.
Ported some of the ReaderZ code to swing and made a simple desktop PDF viewer. It renders on the fly as you pan around, and works quite well. I use another thread to load and render pages as in ReaderZ. I'm a bit dented that I couldn't get the outline stuff to work.
Played with an alternative page rendering method: rendering directly to the array of a BufferedImage using GetPrimitiveArrayCritical(). Can't tell the difference on the desktop but it might be useful on the kobo, and could use less memory anyway.
Noticed that mupdf is undergoing rapid development at the moment; i'm not trying to track it at all.

Update: I got in touch with the mupdf devs, and I found out the next release is targeted by the end of the month; this will be a good opportunity to sync up with the api whilst the project is still fresh in my mind.

Blah, ffmpeg changes and pain.

Sigh. I was looking at moving jjmpeg to ffmpeg 0.10, and along the way remove the use of deprecated APIs.

But there's a lot of pain involved here:

There's quite a lot of deprecated stuff - Well what can you do eh?
Custom streams are completely deprecated with no public API to replace them. That means AVIOStream will have to be thrown away and at best you're left with using pipes or sockets.
Some arguments are now arrays. This is a lot more of a head-fuck than you'd imagine and will require messy and inefficient code to marshal values around.
Some constructors now take in-out parameters as the object pointer. This requires a new constructor mechanism and frobbing around.
Some arguments are also in-out parameters. Hello CORBA style holder arguments, or some other mechanism (the Holder type seems the easiest though) and a lot of fuffing about with jni callbacks to access them. Also, this cannot be inferred from the prototype alone, so I also need to expand the generator for that.

The last 3 not being in libav* was one of the reasons I tackled jjmpeg in the first place, it was a nice clean api using fairly consistent conventions. This made it quite simple to bind.

Tuning ...

Had a poke at some performance tuning of jjmpeg.

I took 2 videos:

PAL: A PAL DVD, half hour show.
1080p: A half hour show recorded directly with a DVB-T receiver. 1440x1080p, ~30fps, 10MB/s.

I then used JJMediaReader to scan the files and decode the video frames to their native format. I then took this frame and converted it to an RGB format using one of the tests below.

ByteBuffer: Code uses libswscale to write to an avcodec allocated frame in BGR24 format. The frame is not accessed from Java: this is the baseline performance of using a ByteBuffer, and it could be the end point if then passing the data to JOGL or JOCL.
ByteBuffer to Array: Perform the above, then use nio to copy the content to a Java byte array.
IntBuffer: Code uses libswscale to write to an avallocated frame in ABGR format. Similar to the first test, but a baseline for ABGR conversion.
IntBuffer to Array: Perform the above, then use nio to copy the content to a Java int array.
int array: Use JNI function GetPrimitiveArrayCritical, form a dummy image that points to it, and write to it directly using libswscale to ABGR format. This gives the Java end an integer array to work with directly.

In all cases the GC load was zero for reading all frames (i.e. no per-frame objects were allocated). I'm using JDK 1.7. The machine is an intel i7x980. I'm using a fairly old build of ffmpeg (version 52 of libavcodec/libavformat).

The timing results (in seconds):

Test \ Video          PAL    1440x1080p

ByteBuffer            81.5   237
ByteBuffer to byte[]  86.0   279

IntBuffer             81.3   242
IntBuffer to int[]    86     297
int[]                 81.9   242

Discussion

So ... using GetPrimitiveArrayCritical is the same speed as using a Direct ByteBuffer - but the data is faster to then access from Java as it can just be indexed.

Using RGB and ByteBuffer's is a bit quicker than using RGBA. Apart from the differences down to libswscale there seems some overhead using an IntBuffer (derived from a ByteBuffer) to write to an Int array.

Using RGB is marginally quicker than using RGBA - although that's mostly down to libswscale, and for my build nothing is accelerated. When I move to ffmpeg 0.10 I will re-check the default formats i'm using are the quick(?) ones.

When using a direct buffer and then copying the whole array to a corresponding java array, the overhead is fairly small until the video size increases to HD resolutions. At 23% for 1440x1080xABGR, it is approaching a significant amount: but this application does nothing with the data. Any processing performed will reduce this quickly. At PAL resolution it's only about 5%.

Conclusions

For modern desktop hardware, it probably doesn't really matter: the machine is fast enough that a redundant copy isn't much overhead, even at HD resolution.

Possibly of more interest is how the rest of the pipeline copes. Obviously with JOGL or JOCL the work is already done when using ByteBuffers, or ideally you'd process the YUV data yourself. I'm not sure about Java2D though, from a previous post there's a suggestion integer BufferedImage is the fastest.

However there are possibly cases where it would be beneficial and for Java image processing it is probably easier to use anyway: so I will add this new interface to jjmpeg after confirming it actually works.

I also found a bug in AVPlane where I wasn't setting the JNI-allocated ByteBuffer to native byte order. This made a big difference to the IntBuffer to int[] version (well 44% over no array copy in PAL), but wouldn't have been hit with my existing code.

Sleep n Whinge

ugh, what a crappy day. I hit the grog a bit hard last night (sister dropped by for a couple of hours on her way to the airport), and subsequently had very little sleep; and the neighbours decided today was a good day to re-start the work on the extensions next door. Had a nap about 5, at least until some dodgey scam out of India rang up about 7:30. Blah.

But I played a bit with some code during the day. I poked around with my slideshow creator, working on some more transition wipes - worked out a 'clock' transition which seemed to take much longer than it should have (for lack of inspiration I'm looking at the SMIL stuff for ideas). I was going to write a very simple front-end gui for it, but just didn't have the motivation for that today.

Then I got totally side-tracked with some other stuff: I noticed javafx builds are finally available for gnu/linux, looking at the swingx demo (there's a couple of things that look interesting), the image filters it uses. Mr Huxtable also has an interesting article about BufferedImage stuff (which i'm sure i've read before but must have forgotten about): and that got me thinking about changing the way jjmpeg's helpers work with images as it uses 3BYTE_BGR types and direct DataBuffer access.. And that got me thinking about JNIEnv.GetPrimitiveArrayCritical (to avoid 2 copies), and well by this time I was too hung-over and tired to do anything useful.

I also noticed the neighbours were building a really big verandah which will block most of the direct light into my bathroom, and they over-cut a bit of a tree that hangs over the boundary. And I got a letter from my insurance company whining about an over-charge they shouldn't have been making in the first place. All all that together with the severe lack of sleep, put me in a terrible mood and made me feel really rather miserable. And now it's 3am and they'll be at it at 7am again next to my bedroom window so tomorrow probably wont be much better ...

Update: Oh fun, 7:25am, shit radio station was bad enough, now it's with the jack-hammer.

About Me

Tags