Kobo stuff

Well i had to do a factory reset of the Kobos - I just couldn't be fucked with seeing a blank page every time I opened a text file and dealing with their embarrassingly slow font tweaking window. I left it at the version it came with and just bypassed the "must have an account and be logged in to even use the machine" bullshit it comes with.

Very nice when simply "logging out" has it reboot to what appears to be a factory reset (it isn't, it just looks that way).

I left a rant on the mobileread forums about how shitfully fucked and slow the firmware is, but i'm just one voice amongst many there.

Anyway, so I guess I did do some kobo hacking soon rather than later after-all, and today I spent too much time today playing with some new widget code.

Basically i'm not all that happy with the old gadget toolkit stuff - although it surely would do the job. And although some guy who took a copy of some of the ReaderZ code via the koper project worked out how to make Swing talk to an alternative device (looks easier than I thought) ... after JavaFX i'm not a big fan of Swing either.

So, foolishly I started hacking on another toolkit which is basically a "cut down JavaFX". Which is obviously a bit pointless in itself because it should eventually be available in the ARM backend ... although that depends on if they support it for the soft float abi, and unless it comes with native eink support it might not be so easy to add it in.

Actually I just noticed that openjfx is GPLv2 + classpath exception, which I missed the first time I looked - should have been obvious I guess. Maybe I should just look at that (when it's ready?) ... although I guess by the time I get anything working maybe kobo will have released a working-enough firmware like 2.0.0 was.

Kobo glo

So i was a bit bored so I went and got a kobo glo today - although there's nothing wrong with my kobo touch and I use it all the time, I though the light might be handy. And besides it isn't very expensive. Actually I got two, as i'm going to give one to my luddite sister who is in to reading books but otherwise wouldn't bother with such a machine.

I'm still going through the 'something happened to your network' errors whilst configuring it using the kobo setup software inside wine, but at least it appears to be working ... It did the upgrade now it's just syncing some books or something. Oh I can just skip that.

So I finally got to try the backlight after all that - the one in the shop (officeworks - bloody useless for customer service but it's close to me, although they only had black, or black) wasn't configured so just showed the setup screen. Looks like it's supposed to I guess, I will have to wait until I read tonight. If anything there is a slight bright line across the very top and slight dark line across the bottom but the whole text area is nice and even. The higher resolution vs the kobo touch is noticeable too, and now it's about the limit of my tired eyes.

The hardware looks nice as always - but it's a pity the software still has issues. Although the lack of the re-assuring home button is ... well, not as re-assuring as it is having one on the kobo touch. And the soft quilted back isn't as soft in it's quilting as the touch is. As to the software, the i/o code needs to be multithreaded or something so it's more responsive to clicks, and other annoying things. Like setting the font waits for the whole document to be reformatted before refreshing. And it still takes FOREVER to close a text file down (sometimes ... garbage collection is a massive massive win).

Speaking of fonts, Times New Roman has vanished! Only left with Georgia as a remotely decent serif font - which is just not good enough. I tried some of the 'fattened' ones on the mobileread forums but ended up with DejaVu Sans Serif for now - the descenders are a bit squat and the serif's aren't pointy enough but otherwise it seems ok. Unfortunately the font fine-tuning which I used on the touch doesn't work on third-part fonts but i'm happy enough with the weight as it is.

Anyway, this frees up my kobo touch for just hacking on, although I don't have any immediate plans for doing so. Whilst I have some holidays left I need to get some shit organised around the yard, well if i don't get too lazy ...

Update: Well I 'upgraded' to the latest firmware, and it's pretty fucked. Dropped linux filesystem support - well that's not the end of the world. But text files always open blank until you change the font settings. And they are excruciatingly slow to close or to suspend - it takes 25 seconds to suspend when reading one text file, which is about how long it takes to 'close' a text file as well. The touch with it's older firmware is pretty fast.

The hardware is mostly excellent, why is it so let down by shitty software?

So maybe it wont be that long before i'm hacking on it after-all ...

Update: So after having it for 6 weeks, here's a little update.

I did a factory reset to restore the original firmware it came with and bypassed the "must be logged in to use" mis-feature, and just load books using usb mass-storage or the sd-card. This makes it a pretty usable reading device, although without connecting to kobo I don't get a dictionary - but I can live without that even though it would be nice to have.

I've warmed to DejaVu Serif (extended version of Bitstream Vera Serif) and actually prefer now it to the Times New Roman on the touch (I never quite liked Times New Roman before, a little too thin). It just has a nice weight and horizontal density and good readability on the e-ink screen. The higher resolution of the glo is definitely noticeable compared to the touch too.

The backlight is really useful, allows me to read anywhere without any extra bits or reaching to turn off lights. I like the way the on/off button for the light works - unobtrusive and easy to access. As mentioned in some reviews I read, it would be nice if it was a touch dimmer at it's lowest setting for reading in a completely dark situation, but it's still comfortable enough as it is.

The battery life is definitely better than the kobo touch was which seemed to drain fairly fast in suspend mode, although i've been reading more epub's which seem to use orders of magnitude less cpu resources than text files.

The touch responsiveness is "adequate". It still seems to miss touches sometimes or get ones I didn't intend at others if my fingers are near the edge of the screen, but it's a minor annoyance. I'm not a 'skim reader', so any delay from a wrong page isn't the end of the world.

My sister is also really happy with the one I bought her, for the convenience and backlight, and also having access to old editions of classics which are hard to find in the cultural backwater where she lives.

The only thing I can say I really miss is the soft-quilted back cover! The touch is definitely nicer to hold with the raised diamond pattern versus the flat one. It's a feature I would never have thought to be quite so important but it was one of the first things my sister mentioned when I showed it to her and I would have to agree it is noticeable.

Although I wish they would improve the software, it works fine as a novel reader and I would get another one. Although I can't see needing one until the battery dies it's inevitable non-removable death. The support for standard formats and the ability to use it without any "desktop software" is a real winner for me. I just wish it came like that and didn't need poking at first.

A/V sync II

Had another go at a/v sync this morning. So far it's looking quite good although it needs a good clean-up from so many aborted experiments.

I managed to remove most of the synchronous code that was being used to try and coordinate everything and instead I'm using a centrally coordinated sequence number.

Every packet and frame created has a sequence number associated with it which is maintained through the data flow graph. Every time a seek operation is performed the sequence number is incremented. This feeds through to each stage and lets them make cheap decisions on what to do:

• Decoders just flush if the sequence number changes.
• Decoders discard any packets which have the wrong number.
• Renderers discard any frames which have the wrong number.
• Renderers reset any output they need to (i.e. audio flush, re-sync).

I also use similar code to JJMediaReader so that after a seek it discards any frames which come through before desired seek position.

I still need some out-of-band callbacks for seek and pause because all of the above means the renderers may never see anything, but they don't need to do much.

The android player is somewhat broken so I need to fix that before committing anything.

Update: Got it working well enough and checked everything in. The new sync code causes some extra 'recovery time' after seeking, but once it's settled down it ends up with better sync. The recovery time is only noticeable on slow hardware.

A/V sync

Had a bit of a limp stab at a/v sync today.

I started with something simple although it ended up a bit too complicated - trying to synchronise multiple decoding and display threads is a little messy. I was trying to hide as much as possible inside MediaReader and the Audio/VideoDecoder classes, but it got ugly.

But as far as the timing, the simple approach nearly worked as far as sync goes, there's just a couple of issues remaining:

• After seeking on the container, the audio stream starts well before the video stream, so one ends up with a visual pause before starting, or even the video decoding never catching up (wrt packet buffering) and some nasty frame jumping.

  Hopefully this can be solved as I do in JJMediaReader: after a seek I discard packets and/or frames until the seek position is reached.

• Getting the amount audio data "buffered" is a bit of a pain. Trying to use the reported position is fine until you seek since it throws the position out.

The simple approach was to have a central MediaClock object which tracks the audio renderer position, and then does various timing calculations for the video sync. It manages pause as well as interacts with seek. Maybe it isn't so simple ...

Eventually I should work it out.

Preserving arbitrary Aspect Ratio in JavaFX

I had an attempt at displaying the proper aspect ratio in JavaFX, and after a couple of false-starts came up with a pretty simple solution. The ImageView does it's own aspect ratio preservation but for a WritableImage the pixels are always square - as far as I can tell.

So one must adjust it outside. First I just used:

        vout.setScaleX(aspect);

To set the ratio - this displayed the video properly but didn't take the adjust size into account during layout and fitting to the display area. Not really a show-stopper as the user can just adjust the window until it fits, but I was sure I could do better than that.

I tried various things such as placing the vout (an ImageView) into a Group and so on - but this didn't work (of course) as I was setting the dimensions of the ImageView relative to the window size.

Actually it turned out to be extremely simple: since I'm scaling the ImageView when it is being displayed, I just have to scale the inverse when I'm binding it's dimensions:

        vout.fitWidthProperty().bind(root.widthProperty().divide(aspect));

The height is simply bound to the root.height.

And also remember to take it into account on the initial scene size too:

        Scene scene = new Scene(root, width * aspect, height);

So apart from calculating the aspect ratio, those were the only lines of code required. Seems to work although i've only tested it on one PAL 16:9 video so far ...

Although the screen capture stuff just stores the unscaled frame still ... which is probably what I want tbh.

Well not a bad haul today for a poor nights sleep and feeling a bit crap overall.

Update: Added some shots to show it it works in JJPlayer.

Window too narrow - automatically scaled to fit horizontally:

Window too wide - automatically scaled to fit vertically (and controls fading out):

A raw non-corrected frame grab showing the image using square pixels:

More video player work

Mucked around with a few things in JJPlayer last night and this morning. Nothing on TV last night and I just wanted a couple of hours today.

• Tried a much more aggressive fallback mode for frame skipping - it starts to drop decoding of frames as well.

  I'm not sure how useful it is if the cpu is just too slow - it eventually turns the video into a irregular slideshow, but can let the sound play fairly smoothly if there's enough time left. So maybe it has some use.

• I noticed the "get audio location" isn't very reliable in Android (at least), so I changed the way it syncs to audio. I think get audio location is just reporting the head of the last buffer submitted to the sound device or something similar, so it jumps around compared to the actual time. So now it only re-sync's if it gets well out of whack. It seems to remove some annoying frame rate jitter, but as i'm playing back 50Hz material on 60Hz it's still pretty jittery.

• I only allocate 3xtextures in the GLES backend if I don't need them to buffer. Less memory usage.

• Moved some more functionality into the MediaReader class where it has the proper non-asynchronous state from which to make accurate decisions.

• Added a full-screen mode in JavaFX (F11).

• Bound the image size to the window size so it sizes to fit, maintaining aspect ration. Still assumes square pixels.

• Added keyboard controls in JavaFX. Space to pause/unpause, left/right to skip back/forward 30s, page up/down for start/end of file, escape exits full-screen mode or quits, and the afore-mentioned F11 for full-screen mode.

• Changed to SVG icons in JavaFX, and even if they're a bit ugly they're consistent. Converting Inkscape SVG to code is a bit of a pain so it's a bunch of hard-coded stuff plus the curved triangle thing.

• Did some styling to make the buttons flat/borderless, they pre-light.

• Fixed some other seek related stuff in JavaFX frontend. It's better but still not perfect. Pause/resume still messes things up sometimes.

The new buttons and styling.

There were a few annoying things along the way. The keyboard input was a bit of a pain to get working, I had to override Slider.requestFocus() so it wouldn't grab keyboard focus when used via the mouse (despite not being in the focus traversal), and I added a 'glass-pane' over the top of the window to grab all keyboard events. I had to remove all buttosn from the focus traversal group and only put the 'glass-pane' in it. I made the glass-pane mouse-transparent so the buttons still work.

Another strange thing was full-screen mode. Although JavaFX captures ESC to turn it off, the ESC key event still ends up coming through to my keyboard handler. i.e. I have to track the fullscreen state separately and quit only if it's pressed twice.

Next to fix is those a/v sync issues ...

Update Well instead I added a frame capture function. Hit print-screen and it captures the currently displayed frame (raw RGB) and opens a pannable/zoomable image viewer. From here the image can be saved to a file.

Although I haven't implemented it, one could imagine adding options to automatically annotate it in various ways - timestamp, filename, and so on.

Update 2: I subsequently discovered "accelerators", so i've changed the code to use those instead of the 'glass pane' approach - a ton of pointless anonymous inner Runnables, but it feels less like a hack.

        scene.getAccelerators().put(new KeyCombination(...), ...);

Update 3: I tried it on my laptop which is a bit dated and runs 32-bit fedora with a shitty Intel onboard GPU (i.e. worthless). It runs, but it's pretty inefficient - 2-4x higher load than mplayer on same source. Partly due to Java2D pipeline I guess but it's probably all the excess frame copying and memory usage. I tried to do some profiling on the workstation but didn't have much luck. Just seems to be spending most of it's time in Gtk_MainLoop, although the profiler doesn't seem to know how to sort properly either.

JJPlayer controls

This morning I added sound and started work on some controls for the JavaFX version of JJPlayer.

It's mostly pretty buggy but it kind of works ... some of the time.

I got JavaSound working easily enough at least - although i'm just converting to stereo 16-bit for now as I did on Android.

But there are some big problems with the way i'm handling the a/v sync when pause or seeking, so things get messed up some of the time. Too lazy to fix it right now ...

I kept playing a bit with the UI and added a fade in/out of the controls as well as hiding the mouse pointer, and some stylesheet stuff.

Clearly the styling and the ASCII-icon buttons both leave something to be desired at this point.

I thought i'd have trouble getting the hiding to reverse if the user started moving the mouse whilst it was hiding, but it turned out to be pretty easy. Just change the rate on the fading out animation to reverse it, and it still runs to completion at the other end instead. So it fades in and out smoothly depending on user action, and without any eye-jars.

JJPlayer for JavaFX

I finally had enough of Android and started poking at a JavaFX version of JJPlayer this morning.

I'm hoping that working on both of them i'll be able to fine tune the design a bit and help with debugging - e.g. i already fixed the end-of-file bugs and "discovered" the av_frame_get_best_effort_timestamp() function (guh, what a name, bad memories of GNOME dev flooding back).

So far I just have unscaled video going (no sound). There are no controls but I should think JavaFX will make that pretty easy to add (and an opportunity to add some bling). But it can play multiple videos in sequence and the window sizes to fit without eye-jars. Once I get sound going i'll look at filling out the GUI for it - I prefer the OpenAL API but I might try with JavaSound this time.

Performance seems ok, although my first measurements had it on par with mplayer with the GL backend but that seemed to be a specific video (actually it's a touch lower cpu usage on that video). Uses 3-4x memory, but that's cheap isn't it? Unfortunately as one can only write RGB data to WritableImage's, it has to do a YUV-RGB step separately and then perform a redundant copy but there's not much option there.

No screenshot yet as it's pretty basic ...

The code (GPL3) will be in the jjmpeg-1.0 branch of jjmpeg by the time you read this or shortly thereafter inside jjmpeg-javafx.