About Me

Michael Zucchi

 B.E. (Comp. Sys. Eng.)

  also known as Zed
  to his mates & enemies!

notzed at gmail >
fosstodon.org/@notzed >

Tags

android (44)
beagle (63)
biographical (104)
blogz (9)
business (1)
code (77)
compilerz (1)
cooking (31)
dez (7)
dusk (31)
esp32 (4)
extensionz (1)
ffts (3)
forth (3)
free software (4)
games (32)
gloat (2)
globalisation (1)
gnu (4)
graphics (16)
gsoc (4)
hacking (459)
haiku (2)
horticulture (10)
house (23)
hsa (6)
humour (7)
imagez (28)
java (231)
java ee (3)
javafx (49)
jjmpeg (81)
junk (3)
kobo (15)
libeze (7)
linux (5)
mediaz (27)
ml (15)
nativez (10)
opencl (120)
os (17)
panamaz (5)
parallella (97)
pdfz (8)
philosophy (26)
picfx (2)
players (1)
playerz (2)
politics (7)
ps3 (12)
puppybits (17)
rants (137)
readerz (8)
rez (1)
socles (36)
termz (3)
videoz (6)
vulkan (3)
wanki (3)
workshop (3)
zcl (4)
zedzone (26)
Saturday, 27 April 2019, 07:17

blogz 0.3

I did a bit of work on blogz today and pushed out a 0.3 release. This is not really tested and i'm still not using in on zedzone but I will be testing it and perhaps do another 0.x release if I have bugs. But the reason it's out now is that this is the snapshot base that I will load into git and then work on that from now on.

If it all works out the next step will be working on using a database for indices - at the very least it will be some lmdb thing, but it may go the whole hog and start to look at using crez for the backend so I can look into some more interesting ideas, although this is a lot more work. This will also allow me to look into a comment system eventually.

See the blogz homepage for the downloads as usual.

L-O-L like anyone gives a flying fuck.

Tagged blogz.
Friday, 26 April 2019, 02:20

Moar Sauce!

I've now moved my main projects over to using modular Java 11 as well as using git. I've quite a few projects remaining.

I'm undecided on many of them whether they just go in the bin, get updated and published (and abandoned) or get worked on actively again.

blogz is almost ready.

This is mostly a git setup but I'm not actually using it yet on this site either, although i've synchronised the code between them now. Once I do both it'll be there.

playerz is a mess.

This is in the midst of a large amount of changes and I lost track of where I was at. Its a fairly interesting little project though so I will get back to it.

izlib is unfinished

I think I discussed this years ago but basically it's unpublished and well from unfinished. It's basically an image processing toolkit and an experimental platform for Java Streams and API refinement.

I don't really have any plans but I noticed the other day there's a lot of code just sitting there rotting away.

mediaz is archived

mediaz was a project I had on google code which was sort of a predecessor to izlib as well as a collection of early JavaFX experiments which are more or less tutorials for others. And the basic start of a layered bitmap image editor (imagez).

It was about the only project anyone asked about when google code went down. I have the repository archived so I guess I could at least convert it.

termz is unfinished

This was a little fun project using OpenCL to drive a terminal emulator display. It's pretty much pointless and could be done directly with OpenGL.

cdez is rotting

I actually have a C version of dez 1.2 too. It could be updated and published or something.

rez and crez are complicated

rez is a project that implements a `personal' versioned blob store. It supports free branches and cheap copies (iirc), renames, metadata and all that jazz, and uses dez for compact storage. Berkely DB (JE) is used as an embedded database.

This is a project i've worked on and off for years (15?+) with separate C and Java implementations. The original driver was for a web CMS. The last time I worked on it was a year ago porting it to C (including a C versin of dez), for this very blog - although in the end I stayed with the simple blogz.

It's explicitly not `enterprise' oriented on purpose.

There's probably plenty of alternatives around now and I don't really know what to do with it. I never quite got the API down to the point I was really happy with. The C version should probably use lmdb now.

What I have done actually works though so I should probably drop it out somewhere. The problems aren't all that complicated but I think I solved them in a fairly tidy and compact and reusable way.

libeze - 2.0

I just need to drop this into git as is.

This was also in the midst of a large number of changes but I kept them out for the last release. But I have a good number of bits and pieces I can add to it. playerz is the main driver here, but that doesn't need much.

SynthZ - the popular one

I'm not doing anything with this, but it has been downloaded fairly often. Maybe I will git it.

I did learn some more about SourceDataLine so the soft keyboard is real-time now!

wanki is who the hell knows

This was a wiki engine using texinfo markup and with the abiity to properly organise and export multi-page documents. It's always been pre-alpha and gone through a number of varients, from C to JavaEE.

It was the original driver for what became (c)rez.

Who knows, maybe one day, I still think it's got some merit. Wiki's still have major trouble with multi-page documents ordered like a book.

DuskZ is unfinished

An attempt at working on a simple game. I got caught up in pointless (de)serialisation stuff and other sort of unimportant details. I'm sorry to the lad that I chatted to about it. He is a nice fellow. But it was sort of at a bad time personally and I just dont have the interest to work on it anymore. I hate letting people down.

I don't know if there's really anything salvagable at this point as all I did was break a bunch of stuff.

socles is archived

This was an OpenCL image processing library. Nobody cared and eventually neither did I. There might be something salvagable for an eventual izlib backend, maybe.

low level arm code, puppybits, zedos

puppybitz is somewhat stale but maybe there's some stuff useful there.

zedos stopped when I hit the USB driver. Fuck intel for making that junk.

paralella is DEAD TO ME

The parallella stuff i'm not longer working on and it will be staying where it is (at least it's published).

I still have some boards but the whole thing soured me on kickstarter and I will never put money into anything of the sort again. It seems it's mostly used for these sort of projects as a 'lets get some zero-cost bridging finance so our real investors have more confidence' which is pretty much fucked-up capitalism at it's finest.

Wherein capital should be the one taking risks to invest in capital to make more of it. This instead is, well, get the plebs to give us free money and take all the risk for no return!

Android apps are going nowhere

Again the source is already out, it's unlikely I will do more but whether I do will be on a case-by-case basis.

I'm sure there is more - that's just what I found from my archive of google code and a quick look at what I have sitting on THIS computer. I've got drives and backups elsehwere, who knows.

Probably if anything I should start checkpointing more often and dumping shit on code.zedzone.au. Until I had that I didn't really have anywhere to put the random otherwise not really publishable-in-themselves experiments which abound.

It will be a while before this list is fully processed.

Tagged code, java, zedzone.
Friday, 26 April 2019, 02:10

dez 2.0 for DEZ1 is here!

I just released dez 2.0 for the DEZ1 binary delta format.

Consider this a 1.0 release! The format is now frozen.

It was just going to be 1.3 but I noticed that 1.2 came out 4 years ago so I thought 2.0 was in order. I did prepare 1.3 in the repository but then I just tagged it again with 2.0.

There aren't really many tunables left so I dropped the flags field and made the OP_SINGLE_SPLIT value tunable. It doesn't make much difference anyway.

This is now a modular Java 11 project using my common java.make. I'm also using it to work on adding junit(4) testing to the java.make system. I may split out java.make into jni.make, java.make, junit.make, and so on. tuning it so haven't rolled it back into the base yet.

As usual the Delta-Z home page has download links and other snot. code.zedzone.au houses a browsable copy of the repository; which now uses git. q

Update: I decided to update the javadocs, as they are. The interesting page is the DEZFormat class which defines and describes the format.

These initial links may not be long-term stable as I will possibly look at a unified javadoc build across all my java projects. And I really need to do something about the blinding default stylesheet. My eyes! My EYES!

Update: I couldn't put up with it so I fixed the stylesheet. This time I used some Workbench 2.0 colours but without the 3D borders. Came out quite nice. I filled out the api documents somewhat as well.

Tagged dez, java.
Wednesday, 24 April 2019, 12:39

Source Code Browser

I've just installed a source code browser on code.zedzone.au and re-added Code back to the main menu of most pages.

I'm using a slightly hacked up version of gitweb. I needed to add a bit more structure and style stuff to get the look I wanted and disable some junk I didn't need. Some more Commodore colourations for a bit of fun.

This is still work in progress and i'm still trying to decide how i'll manage the repositories but hey it works. I'll be slowly migrating all my projects over to it over time. Any java projects will become modular projects first. I'm not bothering to keep any history because I don't need it.

Tagged jjmpeg, nativez, opencl, zedzone.
Wednesday, 24 April 2019, 04:10

libeze 2.0, 1.1

I had a few fixes I did ages ago I thought I should push out. Mainly some stupid list errors.

I also redid the tree api such that a version bump was required. The changes are sore of either/or and add an extra argument to several functions but it was for some consitency with the list api (no allocations, static init) and I found some intersting uses for changing the comparison function at search time while working on playerz.

The software is on the libeze page as usual. I also added a CHANGES file with the release notes/changelog.

Weeks ago I also did a lot of other experiments and played with ideas but they never fully formed. Support for more sophsiticated serialisation, elf stuff, and lots of experiemnts with memory and pool allocators. Just can't make a decision on those and I forgot where they were at so who knows.

I haven't done it for libeze yet but i've started moving my projects to using git. I don't really like it (the commands are often obtuse and the documentation isn't user oriented) but CVS support is starting to wane quite markedly in modern tools. Sigh.

Tagged hacking, libeze.
Tuesday, 23 April 2019, 03:14

Big Easter Software Bundle!

After basically spending the entirety of Easter (a 4 day holiday here in Australia) plus half of today I've packaged up and released a big ol bunch of stuff.

NativeZ 1.0 is an updated version of CObject which is portable and modular. I've moved my other projects to use it rather than their own copies which had evolved and diverged over time.

jjmpeg 4.0 is updated to use NativeZ as described on the previous post. There are also a bunch of bug fixes and small improvements. It now includes a module for some demos.

zcl 1.0 has been converted to a modular project and mostly to using NativeZ apis. The tools package has moved to a demo module.

They are all now capable of cross platform builds and they all generate platform specific .jmod files as one of the outputs. To do this they all use a new common java.make which is a modular and updated version of an earlier java.make as covered in the last post (and I made further improvements). Apart from modules it supports parallel builds, generated java sources, native code libraries, and cross platform builds using a very simple configuration mechanism.

They're also a bit smaller due to compression used for some initialisation tables.

Another project I have gone quite far on is an incremental java compiler. By scanning the .class files and .java timestamps i'm able to generate complete and accurate lists of both files that need to be rebuilt and files that need to be deleted across a multi-module source compilation. I have all the parts and I think they should work but I need to tie them together into a usable system, and then extensive testing.

Sigh, but I need a holiday. I might take most of the week off - like pretty much everyone else is anyway.

Tagged java, jjmpeg, nativez, zcl.
Monday, 22 April 2019, 00:39

Development Ongoing

I've spent all of Easter so far just hacking away at a bunch of stuff. Today looks nice so I think I will try to get out of the house, but who knows. My sleep apnoea has been pretty terrible lately and when I foget to use my MAD (mandicular advancement device) i've been wiped out all day, which has been most of this week. Sigh.

I've primarily been working toward a new modular and portable release of jjmpeg and zcl, but i've been distracted along the way and also had some difficulty just making basic decisions.

java.make

I spent way too long on trying to perfect the build system and things like where intermediate files are stored. There are still trade-offs that might mean another change, sigh.

But it does work pretty well. The main makefile isn't particularly complicated even if some of the macro quoting makes it messy to read. The driver makefiles are trivial - like a simpler version of automake.

It works by the master makefile including config.make, defining some variables, and then including java.make.

This for example is the complete Makefile for nativez.

dist_VERSION=1.0
dist_NAME=nativez
dist_EXTRA=README                               \
 build.xml                                      \
 nbproject/build-impl.xml                       \
 nbproject/genfiles.properties                  \
 nbproject/project.properties                   \
 nbproject/project.xml

include config.make

java_MODULES=notzed.nativez

include java.make

It's a trivial case so it has a trivial makefile. I added a make dist target to build up an archive of the source so that meta-data is really most of the file.

The makefile also builds the native libraries and that is covered by another makefile fragment src/<module>/jni/jni.make which is automatically included if detected.

This is likewise quite trivial for such a trivial example.

notzed.nativez_JNI_LIBRARIES = nativez

nativez_SOURCES = nativez-jni.c nativez-$(TARGET:-amd64=).c
nativez_CFLAGS = -Os -Wmissing-prototypes
nativez_HEADERS = nativez.h

This includes the main .c file and either nativez-windows.c or nativez-linux.c depending on the current target.

So make does this:

  1. Compiles all .java under src/notzed.nativez/classes to .class files.
  2. Generates all native binding headers (from javac -h).
  3. Automagically creates .d dependency files for all .o files.
  4. Compiles all .c into .o objects.
  5. Copies nativez.h file into a staging area for the .jmod.
  6. Links the .o files into libnativez.so or nativez.dll depending on the target.
  7. Copies all the .class files into a modular .jar file.
  8. Copies all the .class files, the .so file, the .h files and license files into a .jmod file.

And only does each step if necessary. For example step 2 doesn't just take the output of javac which would force every .c file to be recompiled any time any .java file changes regardless of what it included. So I use install -C to only copy changed headers from a staging area. I also use some tricks to avoid building all the .c dependencies (which requires first compiling all the java) when you do make clean or make dist.

I'm still debating whether i blow away some of the javac output directories to ensure an absolutely correct build, or let that be a separate step and just rely on using the per-module compile flag which is much faster if less accurate.

I'm also still debating whether it builds the .jar and .jmod files by default or as a seprate target. For these trivial projects it doesn't really take any time but if it did it could become a pain as the code can be executed without either.

jjmpeg is a somewhat more complex case which includes generated java sources, external dependencies, and (now) generated c sources and it isn't much more complex to define.

Initially I had everything in the same project but decided to split it out. The former did allow me to hone down the details on the build system though and i'm pretty happy with it now; it's very accurate, simple, flexible, and easy to use.

nativez

I mostly just moved over to the new build system (trivial) and added a few new functions.

However due to some of the latest in jjmpeg I will probably revisit some of the underlying ideas here.

zcl

I converted this into a modular project which mostly meant moving some stuff around. I also changed it to use NativeZ as it's GC/binding mechanism and to use nativez utility functions where appropriate. Being modular lets me split of auxilliary demo code more cleanly so i've done that too.

zcl is all pretty much implemented in a single large (5KLOC) file. I actually spent a couple of hours splitting it apart into multiple files because I thought it was getting a bit unwieldy (and modern emacs struggles with the size of the file unfortunately). But then I changed my mind and threw it all away.

zcl uses a generated pointer table and typedef casts to manage the OpenCL function linkage and while it works i'm thinking of revisiting it again.

I haven't used OpenCL for a while and don't need it at work at the moment and I can never really think of anything useful to do with it. So it hasn't been tested much.

jjmpeg

I already had a modular jjmpeg so this wasn't a big change, until I decided to keep going. First I moved to the new build system. I already had moved it to NativeZ but there were some lingering functions that I hadn't cleaned up.

Then I wrote some demos. A simple example to read video frames. One that saves frames to image files via Java2D functions. Another that displays video frames in a resizable window at about the right speed (i.e. a mute video player). And then a music player, which lets you adjust output parameters.

The music player needed some more bindings and converted enums so I added that. And it needed a way to get to the samples in a format I could play.

First a bit of background. In earlier versions of jjmpeg I just exposed the data pointer array of an AVFrame as a ByteBuffer and provided entry points into libswscale to allow one to convert frame data formats. This is very messy and error prone and really quite pants to use. So copying the ideas from the PixelReader from JavaFX I've created a similar AVPixelReader which is flexible enough to implement MOST of the PixelReader api and hide all the details of libswscale. The current implementation works by creating an AVPixelReader targeting the size you want and then you can write it various Java targets in whatever pixel format you want. I've got some nice helpers that wrap this to write to some Java2D BufferedImage (via the DataBuffer) or JavaFX WritableImage (via a PixelReader).

So I decided to do the same thing with libswresample and samples. But I don't need to support other pixel readers/writers and only just dump out arrays of bytes so I made a much simpler interface.

Although libswresample can stream I basically enforced that a whole buffer needs to be converted at a go otherwise the logic of tracking the frame gets a bit messy.

At this point I was cleaning things up and thinking about a release but got side-tracked again ...

jjmpeg - bindings

jjmpeg has a couple of different ways that things are semi-automated. All simple parameter accessors are created using C macros. Java JNI handles are loaded via a table which describes the type, name, and signature. Library functions from libavcodec.so et al are defined by a table and loaded by a simple platform-specific function. The tables are all const'd, the variables are mostly static, so everything should go in read-only memory and be fairly efficient to utilise.

The tables are handrolled. As are the variables which hold the function pointers - which means i'm copying them directly from header files and turning them into variables. This is error prone and provides no compile time safety confirming that the function matches the prototype.

And with 64-bit pointers the tables themselves are quite large. Each entry has pointers to strings, pointers to pointers, and some flags.

So I thought i'd revisit that.

With a simple code generator I create a struct with all function variables in it, extracted from the headers at compile time. I have a compact descriptor - essentially a stream of bytes - to encode enough information in order to fill out said structure.

It's a mess of shit perl but it works.

The resolution of java objects for jni use is also likewise table driven, so I did the same for that. It makes the assumption that jclass, jmethodID, and jfieldID are all pointer sized but that seems a pretty safe bet.

In short I shaved off about 20% of the size of libjjmpeg.so doing this.

This is an example of the driver file which defines things of interest.

header avutil libavutil/opt.h { 
        av_free
        av_dict_free

        av_set_options_string
        av_opt_serialize
        av_opt_copy
        av_opt_next
        av_opt_find

        av_opt_set
        av_opt_get

        ...
}

java AVOption au/notzed/jjmpeg/AVOptions$AVOption {
        <init>, (JLjava/lang/String;Ljava/lang/String;IILjava/lang/String;)V
        p, J
}

I'm not particularly enamoured of the Java definitions but it simplifies the generator. Hmm, looking now I can use javap to get this info so maybe I can simplify this somewhat whilst performing additional verification checks at the same time.

The generator creates a couple of static anonymous structs and a descrption string, and as a bonus includes the required header files. There are also some #defines so I didn't have to change any of the code that uses them.

/* This file was autogenerated on Monday 22 April  09:35:47 ACST 2019: */
/*  src/notzed.jjmpeg/jni/extract-proto.pl src/notzed.jjmpeg/jni/extract-proto.pl -f /opt/ffmpeg/4.0 src/notzed.jjmpeg/jni/jj-avoptions.def */
#include <libavutil/opt.h>
static struct functable {
        /* libavutil */
        void (*av_free)(void *ptr);
        void (*av_dict_free)(AVDictionary **m);
        int (*av_set_options_string)(void *ctx, const char *opts, const char *key_val_sep, const char *pairs_sep);
        int (*av_opt_serialize)(void *obj, int opt_flags, int flags, char **buffer, const char key_val_sep, const char pairs_sep);
        int (*av_opt_copy)(void *dest, const void *src);
        const AVOption *(*av_opt_next)(const void *obj, const AVOption *prev);
        const AVOption *(*av_opt_find)(void *obj, const char *name, const char *unit, int opt_flags, int search_flags);
        int (*av_opt_set)(void *obj, const char *name, const char *val, int search_flags);
        int (*av_opt_get)(void *obj, const char *name, int search_flags, uint8_t **out_val);
} fn;
static const char *fn_names =
        "#avutil\0"
        "av_free\0"
        "av_dict_free\0"
        "av_set_options_string\0"
        "av_opt_serialize\0"
        "av_opt_copy\0"
        "av_opt_next\0"
        "av_opt_find\0"
        "av_opt_set\0"
        "av_opt_get\0"
        ;
static struct {
        // au/notzed/jjmpeg/AVOptions$AVOption
        jclass AVOption_classid;
        jmethodID AVOption_new_jlliil;
        jfieldID AVOption_p;
} java;
#define AVOption_classid java.AVOption_classid
#define AVOption_new_jlliil java.AVOption_new_jlliil
#define AVOption_p java.AVOption_p
static const char *java_names =
        "#au/notzed/jjmpeg/AVOptions$AVOption\0"
        ".<init>\0(JLjava/lang/String;Ljava/lang/String;IILjava/lang/String;)V\0"
        ",p\0J\0"
        ;

Rather than have a single global table I have one per file (roughly on per class). This leads to a small amount of duplication but keeps everything simple and isolated and also aids incremental compilation. By having each table static to the compilation unit also allows the optimiser to remove any address calculation - it just goes to the linker via the .bss section.

I'm still thinking of other ways to reduce the binding size and/or overhead. For example the accessors are pretty small ... but there's many of them. The problem is if I use tables I need to keep them in sync between Java (e.g. field ids), or if I use strings then I need table lookups. ffmpeg seems to be moving toward the 'libavutil/opt.h' mechanism for accessing everything anyway (probably because of python influence?) so perhaps I should just use that or at least evaluate it.

But I will first have to move these changes to nativez and perhaps zcl.

Other

I fixed a few bugs along the way. I wasn't creating a few objects correctly so they weren't being tracked by the recovery process - mainly a holdout from earlier implementations which tracked objects differently. I wasn't freeing an AVFormatContext properly - these can be created in two different ways and need to be freed differently. Sigh, even the free functions take different argumements (some of the avcodec api is such a mess). Anyway I use the opaque pointer to track this so I could do it properly. There are probably other leaks but it's rather much harder to test than a standalone C program like playerz.

I also looked very briefly at writing a Vulkan binding as well. Hmm, i'm not sure on this one, it's a pretty big API although as I learn more it will probably shrink in apparent size. But regardless it's a bit more than a weekend project that OpenCL was.

It will probably need a code generator to make it managable, at least for certain objects like the construction parameters. I'm mostly concerned with the overheads so will be looking into different mechanisms.

And these might trigger a reworking of everything else along the way.

Still, i'm not sure yet on it, I don't really have a need for it or know it well enough to do a good job either.

Tagged hacking, java, jjmpeg, nativez, opencl.
Monday, 08 April 2019, 23:23

Modular Java, Make. FTW!

I've been working on packaging up the modular versions of various bits of software I have. One big change is that due to the way javac (and to a lesser extent netbeans) works I had to rearrange the source tree, and it also completely changes the way javac is invoked. So my previous java.make is no longer useful.

As i've now added some inter-module dependencies (e.g. nativez) there's a another question: do I keep them as separate packages, or do I just combine them into a single modular project?

For work I went through the same question previously and ended up just putting everything in the one project.

So ... for similar reasons i'll probably do the same. One strong reason not to is that the projects aren't as closely related - jjmpeg, zcl, and so on.

So what to do ...

Recursive Makefiles

I thought one solution is to just have a build system which isolates the modules so that you can only build the ones you're interested in. i.e. if you don't have external dependencies for one you're not interested in, just build the rest.

So ... based on the structure I have already I spent an inordinate amount of time trying to get the separate makefiles to work. In short: I fucking couldn't. I got 95% of it working, and the dependencies were relatively simple (yet very fragile!) but I just couldn't get make to rebuild some targets on a single run. A second run would always work. Just not good enough.

The problem is that as the toplevel makefile doesn't know when the sub-makefiles need to run it has to try to run them every time using phony targets. But it has to run them at the correct time. Generators are run first, then the module is compiled (which generates any native .h files), then jni is built if it exists, resources are copied, and jmod or jars created as appropriate. I could almost get it to work but some of the dependencies weren't recognised as updated; make would say 'this target needs rebuilding' then the next line 'this target is updated'. I think it has something to do with the way it handles phony targets and because I had multiple references to a status file from different levels. It would recognise one update but not another. I could get accurate builds, but not always get the jar/jmod from 'make jar'. I also had to do some other weird tricks like dummy actions on rules which where updated by dependencies via phony rules. I mean it worked but it was very obtuse.

I actually got it very close and it even supported parallel make quite well, but fuck what a pain to get working.

One Makefile To Rule Them All

So I did some searching and came across a chapter of a book that suggested using a single makefile, but using includes to make the parts more manageable.

So I took the per-module makefiles and stripped out some common stuff, renamed some variables and redid the toplevel makefile. After a little mucking about with the generated depenedencies a little bit I had something working within an hour. And parallel make works even better.

As an example this is what the toplevel makefile currently looks like. It automatically finds various things like java source, generators, and native sub-systems and runs them all at the correct time via the dependency graph.

nclude config.make

java_MODULES=notzed.jjmpeg notzed.nativez notzed.jjmpeg.fx notzed.jjmpeg.awt

notzed.jjmpeg_JDEPMOD=notzed.nativez
notzed.jjmpeg.awt_JDEPMOD=notzed.jjmpeg
notzed.jjmpeg.fx_JDEPMOD=notzed.jjmpeg

notzed.jjmpeg.fx_JAVACFLAGS=--module-path /usr/local/javafx-sdk-11/lib

include java.make
  

Provided targets include '{module}', 'jar', '{module}-jar'.

So now I have that working I can think about separating the projects into separate ones rather than modules in way that lets them all be built together. Probably in that case I would add an {module}_XDEPMOD variable which references the location and module(s) and just builds them before doing anything else. On the other hand this isn't the way you do it with autoconf (and other build systems, although some handle it) for external dependencies so maybe it's not worth it.

Faster ... Faster Builds

Whilst doing this I noticed early on that javac has a '-m' flag for building modules that performs dependency checking. Either I didn't notice this when I first wrote the modular build system or I didn't think it important: while it only rebuilds what is necessary it doesn't remove stale files (deleted inner/anon classes or colocated classes).

Slightly stale builds are not the end of the world so it's probably not a big ask to require a make clean occasionally anyway as the improved build time is significant.

But there is a better way to do it!

Using the jdk.compiler one can wrap access to javac and do some interesting things. To cut a long story short I wrote a javac server that monitors which source files are recompiled and automatically deletes any stale files. This works if all compiles go through the server but can result in stale files if files are compiled otherwise.

So I wrote some more code that scans the compiled class files once and then does a reverse lookup based on the SourceFile, this_class, and module-source-path to discover stale files and prime the database. This is done at server startup. Actually it should be able to be run at any time to post-fix the class tree, so it could be incorporated into the build in other ways.

I wrote a simple .class parser (almost trivial infact) to be able to retrieve the class metadata and some other simple code to do the source lookup.

I wrapped this up in a simple REST-like-service and a script 'sjavac' that compiles and runs the server on demand and then passes requests to it. The server just shuts itself down if it's idle for too long. The compile times for jjmpeg+nativez isn't very long anyway but using the compile server alone speeds up complete builds significantly, and incremental builds are now sub-second.

Another issue is concurrency from parallel builds. javac will recompile any out of date classes it encounters even if it isn't in the module specified by -m - which can lead to two separate compilers working on the same files. This conflict can break a build run. But I think this is due to incorrect makefile dependency definitions as if they are done properly this shouldn't happen.

sjavac serialises compiles at the module level anyway because the compiler uses multiple threads already (afaict, it's a bit too fast to find out on this project) and I haven't set the code up to handle concurrent compile requests anyway, but I want it to be a drop-in replacement for javac for a project build.

Now that i've moved to a single toplevel makefile it's actually a very simple setup. The java.make referenced above is only 100 lines long. sjavac is 100 lines of bash and 600 lines of stand-alone java. And together these provide faster, more accurate, and repetable builds than a special tool like ant or maven. With significantly less setup.

Update 10/4/19: Oh for fucks sake!

It turns out that javac -m doesn't recompile dependent classes if they are out of date, just the first-level dependencies or any missing classes they require. This doesn't make a lot of sense because there are many changes where this behaviour would lead to an inconsistent build so it makes this option fairly useless.

It's a trivial change to the makefile to fix it but it means that it only builds incrementally at the module level.

I guess this gives me the next problem to solve. I think I have enough scaffolding to be able to extract the dependency tree from the .class files. Well I'll see on that.

Tagged java, jjmpeg, nativez.
Newer Posts | Older Posts
Copyright (C) 2019 Michael Zucchi, All Rights Reserved. Powered by gcc & me!