jjstuff n stuff

Well i've mostly been busy with work for the last couple of weeks. I more or less ended up taking a couple of weeks off around Easter and ANZAC Day so I had to do some more hours. There's no deadlines at the moment and I was waiting for some resources to become available so there wasn't much else to do anyway, and between all the hacking I did I managed to get a little of the waning sun before winter really hit a week ago.

I did a little bit on jjmpeg over the weekend. Partly as I was using it to prototype some ideas I was evaluating for work; I will probably end up doing it in C but its an easy platform to experiment in.

I did a fair bit of cleanup around JJMediaWriter, although it still needs work. Video works well but audio is probably broken.

The VideoPlay demo is a little nicer and I moved to the java.time stuff for date formatting. Fixed a bug with error handling.

Another bug I fixed was with the initialisation of native methods for functions which don't derive from AVObject. AVObject contains the code which loads the native library so if you don't use it first you don't get any native linkage. I hacked a simple mechanism using a NOP function that seems to work properly. You can't just load it in another class.

JNI Callbacks

I spent a bit too much time working on filling out AVIOContext to make it more 'complete'. AVIOContext is used to read/write files and network connections and allows for user-supplied callbacks to implement custom i/o. jjmpeg supports both ffmpeg and custom i/o but there was an updated open2() api for the former which takes an interrupt-check callback when opening a context. This posed a problem because for the custom i/o case I had a free pointer opaque I use to track a hard reference to the Java side callbacks but for the ffmpeg i/o case it is used by ffmpeg.

jjmpeg relies on the object pointers being the actual objects so I can't easily put this through a proxy object that contains both pointers. Actually there is only one place it is used but even then it would be very messy to make it work sanely.

The solution I came up with was to have the Java object retain a reference via another NativeZ managed object which retains any number of JNI side references which gets freed JNI side when it gets cleaned up. I then just poke an instance of this onto the AVIOContext so it lasts at least as long as the AVIOContext itself. I put this manager object into NativeZ because it might be useful elsewhere.

While it should work, it does seem like a bit messier than it might be. Just now I thought perhaps I could just store the Java listener in the Java object, but the C callback still needs a reference to this to retrieve the listener and it can't hold a hard reference otherwise the object will leak. But I just looked up the JNI docs and found a WeakGlobalRef API so that should probably work. I will try it and if so I will use that instead and roll back the code I added to NativeZ as well.

Documentation & The Rest

I also started a longer term and possibly pointless exercise of cleaning up the accessor methods and documentation. The accessor methods just access the struct fields directly but most fields are also accessible via the pretty messy libavutil/opt.h interface which is mirrored in part in AVOptions. Originally I went through the structures and just inserted the ones that looked like they might be useful but there's a lot of cruft in there and AVOption covers all the important onces. So I went throuhg AVCodecContext and retained the useful looking ones and cleared out the rest. I also cross-referenced the options table and documented the corresponding names if they exist.

After that I got the javadocs to build. I had to fix a bunch of badly formatted comments before I read the javadoc manual and a bunch of broken comments that get imported by the constant extractors. It all looks pretty bare though. It will take a lot of work to make them something worth using. Will see, will see.

I even did a little OpenCL today. Just a tiny bit of code to run a random fern inferencer. It's been so long since i've written any it took a few iterations to get things going. Didn't need to make any changes to zcl though, so that's a plus. But even with all the niceties of zcl it's still a bit, well, messy. The CLTask is a good kernel of an idea (pun intended) but there might be ways to build on that to be something a little more convenient. How deep I look into this will depend on how much OpenCL I end up writing but that probably wont be much.

Tutorial

Oh another thing I did over the weekend was some Java tutoring for a mate. He's doing JavaFX, so yeah it wasn't much effort even with the hangover I had acquired from the night^* before.

I was somewhat surprised at how bad the assignment was though. For a first assignment the scope wasn't so bad but right from the start things were a bit strange.

Must be Eclipse, must be JavaFX11. Why JavaFX11? Who knows, it's not really that different as a toolkit compared to JavaFX8 apart from being a much bigger pain to install not to mention the complication with the module stuff. Just what you want for an introductory course. Eclipse is pretty much total pants too if you ask me.

There is a page for an eclipse module or plugin or whatever it is but the 'for the lazy' page is so short it seems to be missing all the details. And the screenshots are so low resolution you can't read any of the text. The only remotely likely looking available option didn't match the description and looked wrong - and it didn't work when you installed it regardless. We did get it running but I had to put in JVM arguments and frob around a bit. Admittedly I need to do that in netbeans too but at least it supports modules and you don't need to use explicit --add-module JVM args.

(Being an apple machine i wasn't familair with it's weird filesystem heirarchy and the funky mess of a thing 'Finder' has turned into so that didn't help either).

Back to the assignment itself. Rather than a list the requirements, it stepped you through the parts that needed to be done. This is after first insisting that it wasn't a guideline of the steps to follow - even though you couldn't do the assignment without following them. In the addition to this odd idea, the formatting was completely miserable, like an email where carriage returns have been inserted randomly between sentences. What text there was was also often contradictory, and when it wasn't it offered questionable solutions to common problems.

For example one paragraph would talk about placing code in a constructor then the next sentence would say the same thing needed to go in some other method. In one case this was talking about the root Application where you pretty much never use the constructor and you put this stuff in start() so it was just wrong. In another case it talked about creating a 'XYStage' and then the code suggested for the constructor is 'stage = new Stage()'. It even mentioned deriving from Stage although not consistently! For the rest of the assignment it kept talking about this stage variable which shouldn't exist. I really don't like using the word wrong for software since so many ways will work - poor, snot, junk, totally shithouse, sure all of those, but there really was a lot of utter wrong here!

Even strange stylistic choices that indicate the author doesn't write or read any code of any language. Who creates any rectilinear object like a Window using (sizex,sizey,x,y) syntax rather than (x,y,width,height)? And lets start out teaching bad ideas like using undecorated windows so you can implement your own buggy and non-accessible window manipulation features yourself. Garish colour suggestions, check! No namespaces. Variables which change names or types from paragraph to paragraph, go you covered there too.

Bit bored tonight. I almost did some more work-related stuff but decided I needed a break from it. Not much else to do though which is why I posted this. Well it looks like i'm now playing with jjmpeg for a little while at least.

Minimalistic Perfect Hashing

As a small diversion I played a little bit with minimal hash functions over the weekend.

The goal here is to convert a static list of words into a token value which could be used to implement a string switch statement or other such uses.

The GNU gperf tool is a production-ready general-purpose solution to this problem but I wanted to see if I could reduce the code size and try a more naive approach.

The approach is to just perform an exhaustive search over a limited set of possible functions. The function I am using is a multiply and shift. The input is an integer formed from the next 1, 2, or 4 bytes. I map these to a power-of-two table and just find a combination which produces no collisions over the table size. Then there are a couple of tables used to map these to an index and to a string which is used to verify the string is the same and not just the hashcode.

Using the 32 keywords for the C language this is one solution that takes only a single multiply-shift stage:

#include <string.h>
int hash_keyword(const char *word)
{
    static const char *words =
        "char\0" "long\0" "register\0" "if\0" "signed\0" "unsigned\0"
        "float\0" "switch\0" "int\0" "for\0" "while\0" "volatile\0"
        "static\0" "auto\0" "do\0" "union\0" "enum\0" "typedef\0"
        "struct\0" "sizeof\0" "const\0" "extern\0" "case\0" "default\0"
        "return\0" "break\0" "continue\0" "short\0" "else\0" "void\0"
        "double\0" "goto\0";
    static const unsigned char index[] =
        { 166, 0, 166, 166, 166, 4, 8, 16, 166, 18, 166, 166, 166, 166,
166, 166, 166, 24, 32, 37, 43, 166, 46, 166, 49, 166, 166, 54, 62, 166, 68, 72, 74, 79, 166,
166, 83, 90, 96, 102, 107, 166, 166, 113, 166, 117, 166, 124, 130, 166, 166, 166, 135, 166, 143,
166, 166, 166, 166, 148, 152, 156, 162, 166, };
    static const char value[] =
        { -1, 3, -1, -1, -1, 17, 18, 15, -1, 21, -1, -1, -1, -1, -1, -1,
-1, 28, 12, 25, 16, -1, 13, -1, 31, -1, -1, 30, 23, -1, 0, 7, 27, 10, -1, -1, 26, 24,
22, 4, 11, -1, -1, 2, -1, 6, -1, 19, 1, -1, -1, -1, 5, -1, 20, -1, -1, -1, -1, 9, 29,
8, 14, -1, };
    const unsigned char *p = (void *) word;
    unsigned int h = 0;
    {
        unsigned int v = 0;
        for (int i = 0; i < 4 && *p; i++)
            v = (v << 8) | (*p++);
        h ^= (v * 54991) >> 12;
    }
    h &= 63;
    return (strcmp(word, words + index[h]) == 0)
        ? value[h] : -1;
}

This is not a minimal perfect hash function because the hash size is 64 for 32 input words, but it does calculate the result using a single hash table calculation and probe.

Using smaller sample inputs and more steps provides more solutions but it requires more code at runtime. This one compiles into .text=87, .rodata=352 bytes.

By comparison equivalent functionality using gperf creates 595 bytes of .text and constants and 720 bytes of .data. The primary reason it is so much larger is because it stores the word and index in a structure which will take at least 16 bytes extra per entry (8 bytes for the string pointer, and 8 bytes alignment including the index value). This more than offsets the smaller table it uses.

The runtime should be comparable although being 1/3 the size should help the cpu caches.

It's slow to search and might not produce a solution but with some tuning it has worked for a limited number of cases i've tried. There are some other options it could try:

• Use the word length. This finds more solutions;
• The samples needn't be linearly spaced;
• Other operations apart from multiply-shift;
• Other combining operators than EOR.

I think it's already as minimal as possible in terms of .text and .data size, at least on amd64.

As a further comparison I compared a smaller problem to using a linear search. This case was just for 6 strings. Apart from the hash generator I wrote a cascaded if-else-if and a compacted linear search where all strings are stored in 1 string with embedded nuls used to separate them.

            .text   .(ro)data
phash	       79      64
if-else	      206      41
linear	       74      42
  

So the only one that's smaller uses atypical coding anyway.

Blogz Live

The blog is now using blogz as the driver.

The code is mostly the same so there's no big differences but I did change the tag-history urls to use the path for the postid rather than a query string. So those have changed but as they should be excluded from indexing and it doesn't make sense to use them as a link it shouldn't break anything.

I didn't do a lot of testing - there isn't a lot to test - but if i find things i'll fix them.

I did have a small few bugs I had to fix so i'll have to do another 0.3 series release I guess.

I'm doing stuff on 0.4 now, so it might not be far away. That'll move to using a database as the index which again wont make for very large visual changes although I will be able to add titles to the newer/older post linkes. Once that's done I can look at a bunch of other stuff from comments to vote buttons.

And another thing ...

What about memory mappable tagged format?

Whilst this couldn't be done for a struct mapping it would be useful for arrays.

Not hard to add, but needs agreement on both ends. Build-time decision? Runtime decision? Stream meta-tags?

The puzzle never ends.

Super Cereal!

I got way too caught up with writing a new serialiser over the last couple of days. Actually I finished off another one I had so I ended up with two.

There are two cases i'm interested in. One is tight coupling where simplicity and performance outweights extensibility; basically for IPC. The other is where extensibility and size are the main considerations; for object serialisation / data storage.

So I have an XDR-like implementation for the former. The layout of items is the same as XDR (sans mistakes) but it uses native ordering of elements, so i dubbed it XDRN for xdr-native.

For the latter i have -yes- yet another tagged format. Each field is tagged and each object is also a tagged container. The header is at least 2 bytes - a control byte and a tag byte. I can't be bothered typing out all the details - here is whatI have in the source code at the moment.

      This is a streamable self-describing byte-oriented binary format.
      It is a general purpose format and supports a super-set of the
      ez_blob descriptor.  It supports primitive and struct types and
      sequences thereof and there is room for extension.

      Each item beings with a descriptor byte, then followed by a tag id,
      a possible count, and the payload.

      xxxxttcc control byte

      xxxx type code

      0 uint8       unsigned int, value zero-extended
      1 uint16
      2 uint32
      3 uint64
      - reserved
      5 float16
      6 float32
      7 float64
      - reserved
      f struct

      note that for int/float types, (code&3) == log2 of element size in bytes

      tt log2 of tag size in bytes

      0 1 byte
      1 2 byte
      2 4 byte
      3 reserved, no tag?

      cc log2 of count size in bytes, used to indicate sequence length or non-sequence.

      0 1 byte
      1 2 byte
      2 4 byte
      3 none, single item follows

      ff is struct-end code

      A header is a control byte followed by an optional 1/2/4 byte-length tag,
      followed by an optional 1/2/4 byte-length count.

      A structure payload is a list of tagged fields until a struct-end
      code.  A structure sequence is a list of count struct-encoded blocks.

      Integers can be stored in the smallest number of bytes, i.e. with
      all leading $00 bytes removed.

So basically each field has a type, a tag, and a count. Scalar values are with a special count code so don't require a count value. It also differentiates between scalars and single-item sequences. Sequences all have a count and no end sentinal.

It's versatile enough to hold most likely structures but isn't universal. String encoding is application layer. No 128+bit primitives (but there is room to add them). No map type, but there is room to add it (it could just application layer). Probably the only significant one is a 32-bit limit on sequence (array) lengths (for some level of significant!). There are only 96+1 valid codes defined now so there is room in a single control byte for some but not all of these but it's not likely to be as tidy.

One example: tt+cc only defines 12 codes, one could swap tt,cc when tt=11 and thus use all codes and support 1/2/4/8 byte counts with 1/2/4 byte tags.

    ttcc

    00cc   tag size 1, count size 1/2/4/8
    01cc   tag size 2, count size 1/2/4/8
    10cc   tag size 4, count size 1/2/4/8
    11tt   count size 0, tag size 1/2/4
    1111   spare (primtive) / sentinal (struct)
  

Ok, maybe that would work, and it's not really any more complex in the code. It could use a lookup table but shifts would probably be faster. And this still leaves room for 8 more data types.

I went through a few similar iterations to get to this point. It has a couple of noteworthy features.

write streamable

It doesn't need to calculate information it doesn't know in advance. For example the size of an encoded object. This was a mess in my initial attempts and sometimes required multiple recursive passes.

self describing / read streamable

To be robust to data format changes it needs to be able to skip over data it doesn't understand. The tag defines the field so can be used to identify known ones. The data type and length fields combine together to define the number of bytes to skip for unknown fields. An unitendified sequence of structs must be skipped one at a time, but they provide enough information to do so.

compact

Well, relatively compact for the features it provides.

Tags and integers only use the significant bytes. The minimum overhead for scalar values is 2 bytes per field for control+8 bit tag, which will cover almost everything. The minimum overhead for sequences if 3 bytes (control, tag, count), and for structures is also 3 bytes (control, tag, sentinal).

Fields all have default values and such values are simply not encoded into the byte stream.

I dunno I feel it's a bit over-engineered, but I couldn't see a way to simplify it as I really need that tag. It takes about 2x the amount of code to implement vs the xdrn implementation although a lot of that is mapping to the ez_blob descriptor tables. As it is a self-describing format it may be useful to have a map or stream based api too, and an implementation of either would be straightforward.

Internally both use a common robust i/o mechanism which is simple and reliable. This helps protect against common coding errors like buffer under/overruns. I may expose this as an api in itself.

I'm pretty useless at writing tests (can't be good at everything!) but I have tried to write a more comprehensive set of tests here. Particularly if i'm dumping information into a database I don't want it breaking.

I could've used an existing design, but well, where's the fun in that?

More developments

libeze and blogz are now on code.zedzone.au.

I still haven't moved zedzone to use it but I probably will this week.

I've already prototyped all the code required to move blogz to using lmdb as an index. Conceptually it's very simple, just 3 tables and a few indexes. But there's a lot of boiler-platish code to map it to tables and enforce referential integrity across multiple secondary indices.

Although I haven't implemented it yet the tables support the required meta-data for versioning, renaming, tagging and so on. It will also support comments, wiki-like, and book-chapter like articles. So it will be a stepping stone toward that.

And votes.

I'm thinking about another blob serialiser that uses a tagged format. This is to support simpler schema upgrades should they be required. On the other hand it may never be necessary and it start to fall into over-engineering so I need to see how it falls out once i've gotten there.

A big bloody rant

Ok it's time for another rant. I had a list of things to rant about and I lost it ... so I created another one. I may remember on the way.

Children or pearl clutching nannikins may want to look away.

In short; i'm getting pretty sick of junk on the internet, junk in software development, junk in politics. There's the other crap like anti-vaxxers and climate change denialists - but well that's more of a collective fuckup that we'll all have to reckon with in the end.

Alright where to begin. Firefox. Fuck firefox. Every time it upgrades it seems to break somefucking thing. Now the preferences have been lobotomised for no obvious reason: rather than a click you have to scroll forever to find what you want. And the default scroll using a wheel-mouse is so frustratingly slow it's a total headfuck. Why can't they just include a proper javascript and image blocker in the base fucking application rather than a buggy plugin (oh no, extension!) that breaks every upgrade. And then when you run it it wants to run you through a fucking tutorial about all the great shit it does. Yeah i've been using a browser for 20 years, they're just not that special you know. The pocket. I don't even know what it is but I know I don't want it. Fancy url rendering: look it's just a bit of text, I want to see what it is, you don't need to hide the http or the www or the com. It doesn't make it any more usable or more readable. Multi process; which just means heavy swapping. Javascript is just such a stupid language the last thing anyone should be doing is encouraging more of it.

cmake, got what a load of junk. After years of failures I finally am able to actually compile some projects with cmake out of the box. But it's still usually fucked and the only solution seems to be to go edit the usually non-documented application-specific configuration files.

C++ is such a shit language. Templates are joke. You have<to::tell::the::compilerExactlyWhat>youMean.EveryTime(YouUseIt(X)). And it's still uses gobs of memory and runs like a total pig and gives overly verbose and low-information error dumps. Everyone uses templates because it inlining absolutely everything makes the fastest and most efficient code. Actually, no, it really doesn't. Over-inlining starves registers (particularly on the terminally register-starved x86 platform), blows out the L1 cache, and wastes memory and disk space. Sure you can use it judiciously but but the time you've done that you may as well have just use something else.

Python. Spaces are important. Oh no, actually the number of white-space characters are important. I haven't got time to wait for the molasses of a cursor to step-one-space-at-a-fuck-ing-time-to-get-to-where-the-action-is. I saw a comment on reddit yesterday about how python is so cool because of the indenting. It really isn't. With C or any other sane language you can type whatever you want and the compiler can work it out. The bit that python is missing is you can then automatically format it in various more readable ways that show exactly how the compiler will treat it. With python you're stuck with what you see. Add a block? Now you how to indent everything else and make sure you do it right, now just add a couple of braces and hit re-fromat. Ugh.

This isn't even the real problem with python. I recently had to try to compile pytorch, a 'lightweight' (ROFLMAO what the fuck?) artificial neural network engine and the abomonation that was once caffe. The webpage says use anaconda. Ok 4-fucking-gigabytes later I get something that doesn't work; this version isn't compiled with the features I need (probably because of fucking software patents and ffmpeg). So I spent a couple of days trying to get it to build from source (actually I tried that first but had to go back to it). Fighting with cmake and whatnot I finally get it compiled. But then the python doesn't run. Just a meaningless backtrace (which is pretty much my experience with anything python). Ok so some package isn't there. Ok so ... one uses pip or whatever to install it like you might with apt or yum. Except it isn't really like apt or yum at all. It isn't even like slackpkg. All it does is download a tar and drop in on the filesystem. Dependencies? What are they? Versions? Hah, surely you jest! It's just a bunch of files on the system.

So you run the script again. Again it breaks. Again you install another package. Then lmdb wont even bloody install because setup.py is broken. The internets claim this is a pip bug but then you fuck up and end up with 3 copies of pip installed (/usr, /usr/local, ~/.local) ... and well you fuck it all off and start again. Eventually it turns out lmdb is a compile option hidden somewhere inside the cmake scripts of the fucking custom OpenCV build you had to compile when you started on this journey a couple of days ago.

Javascript. I mean I can't even manage to encompass the size of the fuckup of this conceptually let alone put it into words.

Web page feedback. You go to AMD or wherever to look up some hardware or software details and before the front page has even loaded the page goes white and an overlay asks you to provide feedback for your 'web' experience. Here's some fucking feedback:

                     /"\
                    |\./|
                    |   |
                    |   |
                    |>~<|
                    |   |
                 /'\|   |/'\..
             /~\|   |   |   | \
            |   =[@]=   |   |  \
            |   |   |   |   |   \
            | ~   ~   ~   ~ |`   )
            |                   /
             \                 /
              \               /
               \    _____    /
                |--//''`\--|
                | (( +==)) |
                |--\_|_//--|
   

Jesus, even that was hard to find: nobody knows what the fuck ASCII is anymore. It's all windows-cpfuck-you-too or some other junk.

Highly related is a web-site that keeps asking you to subscribe. Or keeps giving you 'helpful hints' on how to use it. You know what, I can read, if it's not obvious it doesn't fucking matter. I don't need to be stepped in turn through every single feature that is so unique it's on every other bloody website. I'm sure you're very proud of getting it to work, well good for you. Now you just fucked it up by bugging the fuck out of me.

Or bloody achievements. I just don't bother with any forums anymore but I am on the FSF member forum although it's pretty dead and I just don't fit in (story of my life). I mean I get a happy-face-stamp for writing my first post. Ok thanks I guess? What am i, in grade fucking one? There's enough trouble with the world infanticising everything, do we really need it on every bloody forum we visit? I know i know, it drivers user engagement and all the metrics. Well fuck the metrics.

Speaking of metrics, why the fuck can't the yanks get with the programme? Even when they do use them they can't spell it properly. It's a bloody metre, a meter is something with a dial. Fuck the freedom units.

Modern user interfaces. aka the hamburger menu. aka the stack of shit menu. Yeah so apparently a small row or column of easly read text is too much for the kool kidz these days and it's all hip 2D iconography and animations and shit. It's a shit style that will age poorly. Every time a site or application updates menu's go missing or get moved somewhere else. Jesus fucking stick to something you don't have to actually change this this just for the sake of it. You know that people will just get used to whatever you have; the cognitive overload of having to relean it every time massively increases the real costs of any 'improvement' you might make.

Glaring white pages. Well this goes back to the original internet explorer, M$ windows, M$ word (yes I said itt! There's good solid reasoning behing it!). I regularly spend 12+ hours a day on a computer, if I was being blasted with that white shit I would be blinder than I am already. I don't get headaches for a reason.

Constant updates, rolling releases, fluid apis. It's just too much. Sometimes things are just done and don't need updating anymore. But if you do that you dissapear off the google search results. You're no longer part of the zeitgeist! Heavens! Can't have that! No you need a fancy logo and an annual rebranding `event' and need never maintain anything again, just throw out a new release when you've got enough new buttons to make it look like it matters! Gotta rush to the end of that sprint!

If you can't keep up, well you've just been out-evolved by the equivalent of a telephone sanitiser.

There was more i'm sure but that'll do for now.

Oh hang on, one more rather important one in my notes. Back to firefox. I normally run firefox which is finely tuned to be actually usable. I don't use many plugins - disable javascript, disable fonts and colours and that's about it. But I customise the fuck out of everything else: no fucking animations of smooth-this-or-fucking-that, middle-click to open a url, etc etc. I even have a userContent.css that hides a whole of shit on sites I commonly use (like the banners on stackoverflow and so on). But I tried a naked profile in order to test out the stylesheets i've worked on for code and docs. Boy what a fucking joke. All the reminders about shit I don't want. They even fucked up ctrl-tab so it goes forward but wont return where you came from (isntead it opens a menu with very different keyboard navigation) - i mean what's 25 years of convention for anyway?

But the most alarming one was when I went to download my own fucking sourcecode. Apparently a tar.gz file is now a dangerous download.

Yeah fuck off mozilla you bunch of stupid cunts.

Now I just feel like shit, this has been really depressing.

Sad face.

blogz 0.3

I did a bit of work on blogz today and pushed out a 0.3 release. This is not really tested and i'm still not using in on zedzone but I will be testing it and perhaps do another 0.x release if I have bugs. But the reason it's out now is that this is the snapshot base that I will load into git and then work on that from now on.

If it all works out the next step will be working on using a database for indices - at the very least it will be some lmdb thing, but it may go the whole hog and start to look at using crez for the backend so I can look into some more interesting ideas, although this is a lot more work. This will also allow me to look into a comment system eventually.

See the blogz homepage for the downloads as usual.

L-O-L like anyone gives a flying fuck.