Fast incremental Java builds with openjdk 11 and GNU make

This post wont match the article but I think i've solved all the main problems needed to make it work. The only thing missing is ancestor scanning - which isn't trivial but should be straightforward.

Conceptually it's quite simple and it doesn't take much code but bloody hell it took a lot of mucking about with make and the javac TaskListener.

I took the approach I outlined yesterday, I did try to get more out of the AST but couldn't find the info I needed. The module system is making navigating source-code a pain in Netbeans (it wont find modules if the source can't). Some of the 'easy' steps turned out to be a complete headfuck. Anyway some points of interest.

Dependency Tracking

Even though a java file can create any number of classes one doesn't need to track any of the non top-level .class files that might be created for dependency purposes. Any time a .java file is compiled all of it's generated classes are created at the same time. So if any java file uses for example a nested class it only need to track the source file.

I didn't realise this at first and it got messy fast.

Modified Class List

I'm using a javac plugin (-Xplugin) to track the compilation via a TaskListener. This notifies the plugin of various compilation stages including generating class files. The painful bit here is that you don't get information on the actual class files generated, only on the source file and the name of the class being generated. And you can't get the actual name of the class file for anonymous inner classes (it's in the implementation but hidden from public view). In short it's a bit messy getting a simple and complete list of every class file generated from every java file compiled.

But for various other reasons this isn't terribly important so I just track the toplevel class files; but it was a tedious discovery process on a very poorly documented api.

When the compiler plugin get the COMPILATION finished event it uses the information it gathered (and more it discovers) to generate per-class dependency files similar to `gcc -MD'.

Dependency Generation & Consistency

To find all the (immediate) dependencies the .class file is processed. The ClassInfo records provide a starting point but all field and method signatures (descriptors) must be parsed as well.

When an inner class is encountered it's container class is used to determine if the inner class is still extant in the source code - if not it can be deleted.

And still this isn't quite enough - if you have a package private additional class embedded inside the .java file there is no cross-reference between the two apart from the SourceFile attribute and implied package path. So to determine if this is stale one needs to check the Modified Class List instead.

The upshot is that you can't just parse the modified class list and any inner classes that reference them. I scan a whole package at a time and then look for anomilies.

One-shot compile

Because invoking the compiler is slow - but also because it will discover and compile classes as necessary - it's highly beneficial to run it once only. Unfortunately this is not how make works and so it needs to be manipulated somewhat. After a few false starts I found a simple way that works:

A phony per-module target depends on all the toplevel class filenames from all the classes in the module. It can optionally build all the changed files in a given module, using the plugin to auto-generate dependency includes.
A per-module pattern rule tracks needed .java to .class compilations by simply appending to a per-module file. It must also be a phony target in that it doesn't actually generate the .class file itself.
Another optional phony `all' target can incrementally build the entire project using a single compiler invocation.
First it must depend on all of the classes; the pattern rules will automagically update the tracking files when they are out of date. If any of the tracking files were created by this time it knows a compile is needed and it runs the compiler with the list of changd source files. It then deletes thes files for next time.

The per-module rules are required due to the source-tree naming conventions used by netbeans (src/[module]/classes/[name] to build/modules/[module]/[name]), a common-stem based approach is also possible in which case it wouldn't be required. In practice it isn't particularly onerous as I use metamake facilities to generate these per-module rules automatically.

I spent an inordinate amount of time trying to get this to work but kept hitting puzzling (but documented) behaviour with pattern and implicit rule chaining and various other issues. One big one was using concrete rules (made files) for tracking stages, suddenly everything breaks.

I resorted to just individual java invocations as one would do for gcc, and trying the compiler server idea to mitigate the costs. It worked well enough particularly since it parallelises properly. But after I went to bed I realised i'd fucked up and then spent a few hours working out a better solution.

Example

This is the prototype i've been using to develop the idea.

modules:=notzed.proto notzed.build

SRCS:=$(shell find src -name '*.java')
CLASSES:=$(foreach mod,$(modules),\
 $(patsubst src/$(mod)/classes/%.java,classes/$(mod)/%.class,$(filter src/$(mod)/%,$(SRCS))))

all: $(CLASSES)
        lists='$(foreach mod,$(modules),$(wildcard status/$(mod).list))' ; \
        built='$(patsubst %.list,%.built,$(foreach mod,$(modules),$(wildcard status/$(mod).list)))' ; \
        files='$(addprefix @,$(foreach mod,$(modules),$(wildcard status/$(mod).list)))' ; \
        if [ -n "$$built" ] ; then \
                javac -Xplugin:javadepend --processor-module-path classes --module-source-path 'src/*/classes' -d classes $$files ; \
                touch $$built; \
                rm $$lists ; \
        else \
                echo "All classes up to date" ; \
        fi

define makemod=
classes/$1/%.class: src/$1/classes/%.java
        $$(file >> status/$1.list,$$<)

$1: $2
        if [ -f status/$1.list ] ; then \
                javac --module-source-path 'src/*/classes' -d classes @status/$1.list ; \
                rm status/$1.list ; \
                touch status/$1.built ; \
        fi
endef

$(foreach mod,$(modules),$(eval $(call makemod,$(mod),\
  $(patsubst src/$(mod)/classes/%.java,classes/$(mod)/%.class,$(filter src/$(mod)/%,$(SRCS))))))

-include $(patsubst classes/%,status/%.d,$(CLASSES))

In addition there is a compiler plugin which is about 500 lines of standalone java code. This creates the dependency files (included at the end above) and purges any stale .class files.

I still need to work out a few details with ancestor dependencies and a few other things.

About Me

Tags