Development Ongoing
I've spent all of Easter so far just hacking away at a bunch of stuff. Today looks nice so I think I will try to get out of the house, but who knows. My sleep apnoea has been pretty terrible lately and when I foget to use my MAD (mandicular advancement device) i've been wiped out all day, which has been most of this week. Sigh.
I've primarily been working toward a new modular and portable release of jjmpeg and zcl, but i've been distracted along the way and also had some difficulty just making basic decisions.
java.make
I spent way too long on trying to perfect the build system and things like where intermediate files are stored. There are still trade-offs that might mean another change, sigh.
But it does work pretty well. The main makefile isn't particularly complicated even if some of the macro quoting makes it messy to read. The driver makefiles are trivial - like a simpler version of automake.
It works by the master makefile including config.make, defining some variables, and then including java.make.
This for example is the complete Makefile for nativez.
dist_VERSION=1.0 dist_NAME=nativez dist_EXTRA=README \ build.xml \ nbproject/build-impl.xml \ nbproject/genfiles.properties \ nbproject/project.properties \ nbproject/project.xml include config.make java_MODULES=notzed.nativez include java.make
It's a trivial case so it has a trivial makefile. I added a make dist target to build up an archive of the source so that meta-data is really most of the file.
The makefile also builds the native libraries and that is covered
by another makefile
fragment src/<module>/jni/jni.make which is
automatically included if detected.
This is likewise quite trivial for such a trivial example.
notzed.nativez_JNI_LIBRARIES = nativez nativez_SOURCES = nativez-jni.c nativez-$(TARGET:-amd64=).c nativez_CFLAGS = -Os -Wmissing-prototypes nativez_HEADERS = nativez.h
This includes the main .c file and either nativez-windows.c or nativez-linux.c depending on the current target.
So make does this:
- Compiles all .java under src/notzed.nativez/classes to .class files.
- Generates all native binding headers (from javac -h).
- Automagically creates .d dependency files for all .o files.
- Compiles all .c into .o objects.
- Copies nativez.h file into a staging area for the .jmod.
- Links the .o files into libnativez.so or nativez.dll depending on the target.
- Copies all the .class files into a modular .jar file.
- Copies all the .class files, the .so file, the .h files and license files into a .jmod file.
And only does each step if necessary. For example step 2 doesn't
just take the output of javac which would force every .c file to
be recompiled any time any .java file changes regardless of what
it included. So I use install -C to only copy
changed headers from a staging area. I also use some tricks to
avoid building all the .c dependencies (which requires first
compiling all the java) when you do make clean or make dist.
I'm still debating whether i blow away some of the javac output directories to ensure an absolutely correct build, or let that be a separate step and just rely on using the per-module compile flag which is much faster if less accurate.
I'm also still debating whether it builds the .jar and .jmod files by default or as a seprate target. For these trivial projects it doesn't really take any time but if it did it could become a pain as the code can be executed without either.
jjmpeg is a somewhat more complex case which includes
generated java sources, external dependencies, and (now) generated
c sources and it isn't much more complex to define.
Initially I had everything in the same project but decided to split it out. The former did allow me to hone down the details on the build system though and i'm pretty happy with it now; it's very accurate, simple, flexible, and easy to use.
nativez
I mostly just moved over to the new build system (trivial) and added a few new functions.
However due to some of the latest in jjmpeg I will probably revisit some of the underlying ideas here.
zcl
I converted this into a modular project which mostly meant moving some stuff around. I also changed it to use NativeZ as it's GC/binding mechanism and to use nativez utility functions where appropriate. Being modular lets me split of auxilliary demo code more cleanly so i've done that too.
zcl is all pretty much implemented in a single large (5KLOC) file. I actually spent a couple of hours splitting it apart into multiple files because I thought it was getting a bit unwieldy (and modern emacs struggles with the size of the file unfortunately). But then I changed my mind and threw it all away.
zcl uses a generated pointer table and typedef casts to manage the OpenCL function linkage and while it works i'm thinking of revisiting it again.
I haven't used OpenCL for a while and don't need it at work at the moment and I can never really think of anything useful to do with it. So it hasn't been tested much.
jjmpeg
I already had a modular jjmpeg so this wasn't a big change, until I decided to keep going. First I moved to the new build system. I already had moved it to NativeZ but there were some lingering functions that I hadn't cleaned up.
Then I wrote some demos. A simple example to read video frames. One that saves frames to image files via Java2D functions. Another that displays video frames in a resizable window at about the right speed (i.e. a mute video player). And then a music player, which lets you adjust output parameters.
The music player needed some more bindings and converted enums so I added that. And it needed a way to get to the samples in a format I could play.
First a bit of background. In earlier versions of jjmpeg I just exposed the data pointer array of an AVFrame as a ByteBuffer and provided entry points into libswscale to allow one to convert frame data formats. This is very messy and error prone and really quite pants to use. So copying the ideas from the PixelReader from JavaFX I've created a similar AVPixelReader which is flexible enough to implement MOST of the PixelReader api and hide all the details of libswscale. The current implementation works by creating an AVPixelReader targeting the size you want and then you can write it various Java targets in whatever pixel format you want. I've got some nice helpers that wrap this to write to some Java2D BufferedImage (via the DataBuffer) or JavaFX WritableImage (via a PixelReader).
So I decided to do the same thing with libswresample and samples. But I don't need to support other pixel readers/writers and only just dump out arrays of bytes so I made a much simpler interface.
Although libswresample can stream I basically enforced that a whole buffer needs to be converted at a go otherwise the logic of tracking the frame gets a bit messy.
At this point I was cleaning things up and thinking about a release but got side-tracked again ...
jjmpeg - bindings
jjmpeg has a couple of different ways that things are semi-automated. All simple parameter accessors are created using C macros. Java JNI handles are loaded via a table which describes the type, name, and signature. Library functions from libavcodec.so et al are defined by a table and loaded by a simple platform-specific function. The tables are all const'd, the variables are mostly static, so everything should go in read-only memory and be fairly efficient to utilise.
The tables are handrolled. As are the variables which hold the function pointers - which means i'm copying them directly from header files and turning them into variables. This is error prone and provides no compile time safety confirming that the function matches the prototype.
And with 64-bit pointers the tables themselves are quite large. Each entry has pointers to strings, pointers to pointers, and some flags.
So I thought i'd revisit that.
With a simple code generator I create a struct with all function variables in it, extracted from the headers at compile time. I have a compact descriptor - essentially a stream of bytes - to encode enough information in order to fill out said structure.
It's a mess of shit perl but it works.
The resolution of java objects for jni use is also likewise table driven, so I did the same for that. It makes the assumption that jclass, jmethodID, and jfieldID are all pointer sized but that seems a pretty safe bet.
In short I shaved off about 20% of the size of libjjmpeg.so doing this.
This is an example of the driver file which defines things of interest.
header avutil libavutil/opt.h {
av_free
av_dict_free
av_set_options_string
av_opt_serialize
av_opt_copy
av_opt_next
av_opt_find
av_opt_set
av_opt_get
...
}
java AVOption au/notzed/jjmpeg/AVOptions$AVOption {
<init>, (JLjava/lang/String;Ljava/lang/String;IILjava/lang/String;)V
p, J
}
I'm not particularly enamoured of the Java definitions but it simplifies the generator. Hmm, looking now I can use javap to get this info so maybe I can simplify this somewhat whilst performing additional verification checks at the same time.
The generator creates a couple of static anonymous structs and a descrption string, and as a bonus includes the required header files. There are also some #defines so I didn't have to change any of the code that uses them.
/* This file was autogenerated on Monday 22 April 09:35:47 ACST 2019: */
/* src/notzed.jjmpeg/jni/extract-proto.pl src/notzed.jjmpeg/jni/extract-proto.pl -f /opt/ffmpeg/4.0 src/notzed.jjmpeg/jni/jj-avoptions.def */
#include <libavutil/opt.h>
static struct functable {
/* libavutil */
void (*av_free)(void *ptr);
void (*av_dict_free)(AVDictionary **m);
int (*av_set_options_string)(void *ctx, const char *opts, const char *key_val_sep, const char *pairs_sep);
int (*av_opt_serialize)(void *obj, int opt_flags, int flags, char **buffer, const char key_val_sep, const char pairs_sep);
int (*av_opt_copy)(void *dest, const void *src);
const AVOption *(*av_opt_next)(const void *obj, const AVOption *prev);
const AVOption *(*av_opt_find)(void *obj, const char *name, const char *unit, int opt_flags, int search_flags);
int (*av_opt_set)(void *obj, const char *name, const char *val, int search_flags);
int (*av_opt_get)(void *obj, const char *name, int search_flags, uint8_t **out_val);
} fn;
static const char *fn_names =
"#avutil\0"
"av_free\0"
"av_dict_free\0"
"av_set_options_string\0"
"av_opt_serialize\0"
"av_opt_copy\0"
"av_opt_next\0"
"av_opt_find\0"
"av_opt_set\0"
"av_opt_get\0"
;
static struct {
// au/notzed/jjmpeg/AVOptions$AVOption
jclass AVOption_classid;
jmethodID AVOption_new_jlliil;
jfieldID AVOption_p;
} java;
#define AVOption_classid java.AVOption_classid
#define AVOption_new_jlliil java.AVOption_new_jlliil
#define AVOption_p java.AVOption_p
static const char *java_names =
"#au/notzed/jjmpeg/AVOptions$AVOption\0"
".<init>\0(JLjava/lang/String;Ljava/lang/String;IILjava/lang/String;)V\0"
",p\0J\0"
;
Rather than have a single global table I have one per file (roughly on per class). This leads to a small amount of duplication but keeps everything simple and isolated and also aids incremental compilation. By having each table static to the compilation unit also allows the optimiser to remove any address calculation - it just goes to the linker via the .bss section.
I'm still thinking of other ways to reduce the binding size and/or overhead. For example the accessors are pretty small ... but there's many of them. The problem is if I use tables I need to keep them in sync between Java (e.g. field ids), or if I use strings then I need table lookups. ffmpeg seems to be moving toward the 'libavutil/opt.h' mechanism for accessing everything anyway (probably because of python influence?) so perhaps I should just use that or at least evaluate it.
But I will first have to move these changes to nativez and perhaps zcl.
Other
I fixed a few bugs along the way. I wasn't creating a few objects correctly so they weren't being tracked by the recovery process - mainly a holdout from earlier implementations which tracked objects differently. I wasn't freeing an AVFormatContext properly - these can be created in two different ways and need to be freed differently. Sigh, even the free functions take different argumements (some of the avcodec api is such a mess). Anyway I use the opaque pointer to track this so I could do it properly. There are probably other leaks but it's rather much harder to test than a standalone C program like playerz.
I also looked very briefly at writing a Vulkan binding as well. Hmm, i'm not sure on this one, it's a pretty big API although as I learn more it will probably shrink in apparent size. But regardless it's a bit more than a weekend project that OpenCL was.
It will probably need a code generator to make it managable, at least for certain objects like the construction parameters. I'm mostly concerned with the overheads so will be looking into different mechanisms.
And these might trigger a reworking of everything else along the way.
Still, i'm not sure yet on it, I don't really have a need for it or know it well enough to do a good job either.