ZCL and Project Panama

So I've been really busy working on zcl again, porting it to the foreign-abi branch of project panama.

Well that's after I ruffled some feathers on the mailing list, I guess I just have strong opinions on C, but I thought I'd better put my money where my mouth was before upsetting anyone too much. To be honest the Oracle engineers have been the most polite and overwhelmingly patient of all the projects I've interacted with lately - guix guys just weren't very polite (that's not entirely fair, some were nice), the google guys for clspv were just condescending, and well the ffmpeg devs never did reply to my patch for a kinect indev.

Anyway back to the topic at hand. I wrote a fairly rushed but quite detailed overview of the issues I came across and some of the features I implemented or needed to change. I titled it JNI to Java with project panama. The prose doesn't flow very well but at least I ran it through ispell which is a habit I'm trying to get into.

Even the lowly HD7970 with the Mesa Driver isn't bad. 1/4 rate DP doesn't hurt this one.

Although it crashes after a few minutes from what looks like a big leak in the OpenCL runtime.

I was originally just going to put into a readme in project panamaz but since I got so far with the development I just published it separately and created a new branch foreign-abi in the zcl repository. The article above has some checkout details about checking it out. It's still work in progress and I'm still making some internal changes but I added a fun little Mandelbrot zoomer to it. The javadocs are pretty shit but for what it's worth I've uploaded those too and will try to keep them relatively up to date while I work on it.

I'm also using the project to experiment with the maven and junit, and now a demo-runner snippet for GNU Make.

It was sort of because I was working in the panamaz project but I wrote almost all of this new code in emacs rather than netbeans. Damn I knew it pissed me off but I didn't realise it wasn't writing Java that was the problem, netbeans keeps just fucking getting in the way when you're trying to write code and those interruptions constantly break you out of 'flow'. I mean the completion and the real-time syntax checking is pretty good but I don't know why the editor has to be such an annoying prick all the time.

Verified Maven Central from GNU Make

Well after the previous post I tried looking at the signature verification step. There isn't too much to it.

define maven_func=
bin/lib/$2-$3.jar:
        mkdir -p bin/lib
        wget -O $$@ $(CENTRAL)/$(subst .,/,$1)/$2/$3/$2-$3.jar
bin/lib/$2-$3.jar.asc: bin/lib/$2-$3.jar
        wget -O $$@ $(CENTRAL)/$(subst .,/,$1)/$2/$3/$2-$3.jar.asc
        gpg --batch --verify $$@ $$< || ( rm $$@ ; echo "GPG verification failed, you may need to import the public key." )
setup: bin/lib/$2-$3.jar.asc
endef

If it fails you need to import the keys using gpg manually. This might be a bit annoying but otherwise you may as well not bother.

gpg will pass the check if the key simply exists in your key store and ignores any trust setting, but that's just the way gpg works.

Maven Central from GNU Make

I had a look at jmh yesterday. It's all driven by maven so that's a bit of a pain but using the ant example I trivially converted it to use GNU make.

As expected, it's somewhat fewer lines of code to just use a generic build tool than either of the other tools which are specifically designed for Java projects. I will integrate it into java.make eventually, although as I don't use any projects that require it I haven't done that yet.

So for this prototyping example you just define a couple of variables. The interesting one just lists the group:artifact:version in a relatively simple/readable format.

CENTRAL=https://repo1.maven.org/maven2

CENTRAL_JARS=					\
 org.openjdk.jmh:jmh-core:1.18			\
 org.openjdk.jmh:jmh-generator-annprocess:1.18	\
 net.sf.jopt-simple:jopt-simple:4.6		\
 org.apache.commons:commons-math3:3.2

And well that's about it. It hooks into the makefile system at the right place to download the libraries. I don't do any signature checks but I can't imagine that would be very to add either.

It only requires a simple macro to download the packages.

define maven_func=
bin/lib/$2-$3.jar:
        mkdir -p bin/lib
        wget -O $$@ $(CENTRAL)/$(subst .,/,$1)/$2/$3/$2-$3.jar
setup: bin/lib/$2-$3.jar
endef
.PHONY: setup
$(foreach nver, $(CENTRAL_JARS), $(eval $(call maven_func,$(word 1,$(subst :, ,$(nver))),$(word 2,$(subst :, ,$(nver))),$(word 3,$(subst :, ,$(nver))))))

Although calling it is a little clumsy, but that's a hidden detail.

To hook it in you have to add jmh-core and jmh-generator-annprocess to the class path of the javac invocation.

bin/.classes: setup $(SOURCES)
        javac -d bin/classes \
                 -cp bin/lib/jmh-core-1.18.jar:bin/lib/jmh-generator-annprocess-1.18.jar \
                $(SOURCES)
        touch bin/.classes

jmh is typically run using a standalone jar file but for simplicity it can just be run in-place from the build directory. Creating a standalone jar wouldn't be that much hard to add, just a few invocations of jar to explode and repack the classes.

bench: bin/.classes
        java -cp bin/classes:bin/lib/jmh-core-1.18.jar:bin/lib/jopt-simple-4.6.jar:bin/lib/commons-math3-3.2.jar \
                org.openjdk.jmh.Main

In the final iteration I will add some code to the macro to create this classpath so it is easier to use.

panama/jextract

Regarding a previous post i've discovered that the annotation mechanism i've been using is probably not the approach to use and there is a lower-level interface. I'm still yet to start on supporting that in the code generator but i've nutted out most of the machinery it will require.

I've bugged the panama developers quite a bit about what I see as some shortcomings in the api. Their idea of a 'C api' is very limited, for example they think that Java shouldn't be able to dereference a C-allocated pointer without signifying 'unsafe' code and passing some scary arguments to the compiler and jvm.

Sure, many modern api's want to wrap trivial memory operations in multiple layers of 'accessor' functions but even those typically return a const char * for a string so you can't even access those without a mess. This seems a little odd because what you can do without resorting to 'unsafe' api's has no safety guarantees already, and infact you can trivially implement an arbitrary whole-memory-access function by just invoking memcpy. Anyway there is hope it is changed by release because it's an unecessary restriction that can be bypassed trivially and just adds more developer work for no good reason. In my experience such complexity just leads to more of the type of errors they're trying to safeguard against.

GTK3 CSD theme improvements

I was frustrated at the inability to re-size gtk3 windows with CSD as the hit-box for re-sizing was a 1x1 button, so I made some improvements to the gtk3 part of my workbench theme.

Now the CSD window borders are more consistent with xfce4wm!

Normal window.

The main trick was finding out which @#$@#$ classes to use for the css selectors. Almost every search sends you to out of date information that references '.window-frame', which is wrong. The GtkWindow documentation actually has the correct names.

And using the :backdrop pseudo-class is required for inactive windows.

Inactive window.

I also used padding and border colours so you get the minor 3D effect so they also match the xfce4wm theme.

Get the theme from code.zedzone.au, details are on the project page.

OpenJDK project Panama hacks

I had a look at the OpenJDK project panama last week. It's a way to replace using JNI to bind Java to native libraries by using plain old java which through annotations can be bound to the native libraries at runtime inside the jvm.

Actually I had no idea how it worked just that one uses a tool called jextract to convert header files into a jar file which you can then link against. However I wasn't really happy with the way it works (and it sounds like it's being changed for mostly those reasons), so i ended up 'reverse engineering' the tool and writing my own.

I probably should've looked harder for documentation but I didn't find much that specified the details so I just ran jextract on some files and tried to generate the same output.

I created a project page and a repository and dubbed it 'panamaz' in an unimaginative following of tradition.

export.cc

The first part of the problem is getting an accurate dump of the c types and prototypes you want to use. jextract is built using clang, but I've had a tiny bit of experience poking about that and I really didn't like what I saw so I thought I'd try using a gcc plugin.

I didn't look very far but i found a couple of bits of example code and opened up gcc-internals.info and got to work. And it was a lot of work. I can see why ICEs were once so common in gcc - you get no compile time errors for bad type assumptions of the syntax tree, and when you do the wrong thing sometimes it will work anyway. The documentation isn't completely accurate either, which is pretty annoying considering how complex the system is.

But anyway in the end of I have a nice tool that can dump considerable information about c header files. The output format is a better, more usable, canonically-decoding version of 'json', or what one might term 'Perl, son'.

The following simple example:

struct bob {
    int a;
    float b;
};
  

Produces the descriptor file:

%data = (
'struct:bob' => { name => 'bob', type => 'struct', size => 64, fields => [
        { name => 'a', size => 32, offset => 0, ctype => 'int', type => 'i32',},
        { name => 'b', size => 32, offset => 32, ctype => 'float', type => 'f32',},
]},
# dumped structs:
# bob
);

generate

This is a (mostly!) better-than-i-usually-write bit of perl which takes the 'perl-son', and turns it into a bunch of java files. These are all interfaces which have the correct annotations to be used by the project-panama jdk to perform runtime linking.

It's got a lot of flexible options but the one I like is that you can specify the set of functions and/or types you want and it will recursively drag in all the types they depend on automatically, and only those. At least for most of it, I still haven't done the same for callbacks yet because they're a bit fiddly.

It ignores enums for now because I just ran out of juice but it covers all the other major parts of the language and even some of the lesser parts.

panama jdk

Most of the information is conveyed in signature strings which are passed to the jdk via annotations in interfaces. The simple example above is translated to an interface with the @NativeStruct annotation that includes the two field names and their types in order.

import java.foreign.annotations.*;
import java.foreign.memory.*;
@NativeStruct(value="[i32(a)f32(b)](bob)", resolutionContext={})
public interface Bob extends Struct {
        @NativeGetter(value="a")
        public int getA();
        @NativeSetter(value="a")
        public void setA(int value);
        @NativeGetter(value="b")
        public float getB();
        @NativeSetter(value="b")
        public void setB(float value);
}

Accessors can be named anything (jextract uses the uber-friendly notation of name$set(...) and so forth) but the field they operate on is indicated using the @NativeGetter or @NativeSetter attribute and specifying the field name, which is defined in the @NativeStruct annotation. There's also an address-of annotation for a rather verbose equivalent of the & operator.

I didn't come across a document describing this format (and I didn't look beyond the annotation source code) but I worked it out by seeing what jextract did. Most of it is pretty straightforward.

The bitfield one is a bit odd, I'm not sure why one has to specify an internal detail such as the actual word-size used for storage, but I don't use bitfields much. Perhaps it's to supports some alignment/packing __attribute__ in bitfields which is used to map hardware registers, although that seems rather obtuse usage for Java api bindings.

So basically I kept banging at the exporter and the generator until I got something working, then tried to compile it and then fixed those problems. Then tried instantiating the object and fixing the breakage. Then tried another type and so on and kept going until it all worked. And then I rewrote the exporter almost entirely to clean up the code. And then fixed the new bugs. Then created a simple example.

Then created a repository and a project page, then sent an email to the panama-dev list, then wrote a blog post ...

Vulkan + OpenCL?

I've been slowly working on some OpenCL on Vulkan stuff using clspv. It's a bit too hot here, my heart isn't really in it, but I'm slowly making progress.

The last thing I checked in was an example that uses more complicated argument passing conventions and uses a small compiler to turn the descriptor map file from clspv into some tables used for initialisation and argument setting. Actually I wrote a patch for clspv to output the data type so the compiler can use it.

Of course along the way I hit fairly obtuse errors from vulkan regarding extensions and options which need to be enabled for the small kernel I wrote, but thanks to the validation layer they were able to be resolved without too much work.

I'm working on abstracting a lot of the details away into a more OpenCL-like API, the first few examples were stand-alone but there's really just so much boilerplate it makes it too confusing for an example. I've discovered a few options like VK_DESCRIPTOR_BINDING_UPDATE_AFTER_BIND_BIT_EXT and it's associated settings that are required for the kernels to behave somewhat like OpenCL where you can modify kernel arguments for subsequent submission. From my preliminary work I think most of the behaviour can be recreated, although the more limiting factor is the compiler and SPIR-V.

It can be found inside the opencl-args directory of zproto-vulkan, but I'll probably re-arrange the directory structure at some point and continue to work on the abstraction. As of this post it's pretty much just when I got it working at all and I didn't try to work on the queuing mechanism.

busymon - tool to force computer breaks

I had need of a tool to remind me (forcefully) to get off the computer and get up and walk around for a while. I was surprised I couldn't find anything simple to do this; I came across rsibreak but it looked like a mess to compile (i mean, cmake, obviously). I looked a bit more but nothing looked simple.

So here's a basic computer monitor that tells you off if you're at the computer too long - as a proxy for sitting.

As usual busymon has a software page and a repository.

In short my bicycle accident has lead to further complications, possibly in part due to sitting down for too long infront of my computer. So now i'm riddled with clots (some poetic license) and on blood thinners and should probably try not to make it worse. Hourly walking breaks are now doctors orders, and well, time just flies sometimes.

amdgpu / x570 / sea islands / IO_PAGE_FAULT

After turning on the amdgpu driver on my new system (radeon.si_support=0 amdgpu.si_support=1) so I could play with vulkan it seemed nice and stable.

But I was only playing around in emacs/netbeans/firefox and wasn't doing much with the graphics system. When I ran a JavaFX thing i've been playing with (the genetic art) and eventually it crashed the graphics driver. So I went back to radeon for the time being until I decided to look at it again.

Anyway today I had another look, and whilst logged on remotely I got the error log:

[  993.135454] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe05000 flags=0x0000]
[  993.135461] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe0a040 flags=0x0000]
[  993.135466] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe338c0 flags=0x0000]
[  993.135471] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe1d100 flags=0x0000]
[  993.135476] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe38900 flags=0x0000]
[  993.135481] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe5dd80 flags=0x0000]
[  993.135486] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe23580 flags=0x0000]
[  993.135491] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe25c00 flags=0x0000]
[  993.135496] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe4ed00 flags=0x0000]
[  993.135501] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe07a80 flags=0x0000]
[  993.135507] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe52a00 flags=0x0000]
[  993.135512] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe55200 flags=0x0000]
[  993.135517] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe5a200 flags=0x0000]
[  993.135521] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe5f2c0 flags=0x0000]
[  993.135526] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe10700 flags=0x0000]
[  993.135531] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe51600 flags=0x0000]
[  993.135535] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe606c0 flags=0x0000]
[  993.135540] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe4c7c0 flags=0x0000]
[  993.135544] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe53e00 flags=0x0000]
[  993.135549] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe58e00 flags=0x0000]
[ 1003.712058] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=144954, emitted seq=144956
[ 1003.712116] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process Xorg pid 1301 thread Xorg:cs0 pid 1303
[ 1003.712118] [drm] GPU recovery disabled.

There wasn't a lot to be found on the internet regarding amdgpu but there were some other similar problems. The suggested fix was to add iommu=soft to the kernel arguments. I'm trying this now and so far it looks good!

I have been tracking the 5.4.x kernel releases and thought that might have something to do with it but fortunately not. When it crashed initially i'd just done a slackpkg upgrade-all and all sorts of things broke horribly. But that was my fault for overzealously removing packages that didn't look important: polkit uses a lot of snot unfortunately so I broke that. Fucking firefox lost all my settings though!