Verified Maven Central from GNU Make

Well after the previous post I tried looking at the signature verification step. There isn't too much to it.

define maven_func=
bin/lib/$2-$3.jar:
        mkdir -p bin/lib
        wget -O $$@ $(CENTRAL)/$(subst .,/,$1)/$2/$3/$2-$3.jar
bin/lib/$2-$3.jar.asc: bin/lib/$2-$3.jar
        wget -O $$@ $(CENTRAL)/$(subst .,/,$1)/$2/$3/$2-$3.jar.asc
        gpg --batch --verify $$@ $$< || ( rm $$@ ; echo "GPG verification failed, you may need to import the public key." )
setup: bin/lib/$2-$3.jar.asc
endef

If it fails you need to import the keys using gpg manually. This might be a bit annoying but otherwise you may as well not bother.

gpg will pass the check if the key simply exists in your key store and ignores any trust setting, but that's just the way gpg works.

Maven Central from GNU Make

I had a look at jmh yesterday. It's all driven by maven so that's a bit of a pain but using the ant example I trivially converted it to use GNU make.

As expected, it's somewhat fewer lines of code to just use a generic build tool than either of the other tools which are specifically designed for Java projects. I will integrate it into java.make eventually, although as I don't use any projects that require it I haven't done that yet.

So for this prototyping example you just define a couple of variables. The interesting one just lists the group:artifact:version in a relatively simple/readable format.

CENTRAL=https://repo1.maven.org/maven2

CENTRAL_JARS=					\
 org.openjdk.jmh:jmh-core:1.18			\
 org.openjdk.jmh:jmh-generator-annprocess:1.18	\
 net.sf.jopt-simple:jopt-simple:4.6		\
 org.apache.commons:commons-math3:3.2

And well that's about it. It hooks into the makefile system at the right place to download the libraries. I don't do any signature checks but I can't imagine that would be very to add either.

It only requires a simple macro to download the packages.

define maven_func=
bin/lib/$2-$3.jar:
        mkdir -p bin/lib
        wget -O $$@ $(CENTRAL)/$(subst .,/,$1)/$2/$3/$2-$3.jar
setup: bin/lib/$2-$3.jar
endef
.PHONY: setup
$(foreach nver, $(CENTRAL_JARS), $(eval $(call maven_func,$(word 1,$(subst :, ,$(nver))),$(word 2,$(subst :, ,$(nver))),$(word 3,$(subst :, ,$(nver))))))

Although calling it is a little clumsy, but that's a hidden detail.

To hook it in you have to add jmh-core and jmh-generator-annprocess to the class path of the javac invocation.

bin/.classes: setup $(SOURCES)
        javac -d bin/classes \
                 -cp bin/lib/jmh-core-1.18.jar:bin/lib/jmh-generator-annprocess-1.18.jar \
                $(SOURCES)
        touch bin/.classes

jmh is typically run using a standalone jar file but for simplicity it can just be run in-place from the build directory. Creating a standalone jar wouldn't be that much hard to add, just a few invocations of jar to explode and repack the classes.

bench: bin/.classes
        java -cp bin/classes:bin/lib/jmh-core-1.18.jar:bin/lib/jopt-simple-4.6.jar:bin/lib/commons-math3-3.2.jar \
                org.openjdk.jmh.Main

In the final iteration I will add some code to the macro to create this classpath so it is easier to use.

panama/jextract

Regarding a previous post i've discovered that the annotation mechanism i've been using is probably not the approach to use and there is a lower-level interface. I'm still yet to start on supporting that in the code generator but i've nutted out most of the machinery it will require.

I've bugged the panama developers quite a bit about what I see as some shortcomings in the api. Their idea of a 'C api' is very limited, for example they think that Java shouldn't be able to dereference a C-allocated pointer without signifying 'unsafe' code and passing some scary arguments to the compiler and jvm.

Sure, many modern api's want to wrap trivial memory operations in multiple layers of 'accessor' functions but even those typically return a const char * for a string so you can't even access those without a mess. This seems a little odd because what you can do without resorting to 'unsafe' api's has no safety guarantees already, and infact you can trivially implement an arbitrary whole-memory-access function by just invoking memcpy. Anyway there is hope it is changed by release because it's an unecessary restriction that can be bypassed trivially and just adds more developer work for no good reason. In my experience such complexity just leads to more of the type of errors they're trying to safeguard against.

GTK3 CSD theme improvements

I was frustrated at the inability to re-size gtk3 windows with CSD as the hit-box for re-sizing was a 1x1 button, so I made some improvements to the gtk3 part of my workbench theme.

Now the CSD window borders are more consistent with xfce4wm!

Normal window.

The main trick was finding out which @#$@#$ classes to use for the css selectors. Almost every search sends you to out of date information that references '.window-frame', which is wrong. The GtkWindow documentation actually has the correct names.

And using the :backdrop pseudo-class is required for inactive windows.

Inactive window.

I also used padding and border colours so you get the minor 3D effect so they also match the xfce4wm theme.

Get the theme from code.zedzone.au, details are on the project page.

OpenJDK project Panama hacks

I had a look at the OpenJDK project panama last week. It's a way to replace using JNI to bind Java to native libraries by using plain old java which through annotations can be bound to the native libraries at runtime inside the jvm.

Actually I had no idea how it worked just that one uses a tool called jextract to convert header files into a jar file which you can then link against. However I wasn't really happy with the way it works (and it sounds like it's being changed for mostly those reasons), so i ended up 'reverse engineering' the tool and writing my own.

I probably should've looked harder for documentation but I didn't find much that specified the details so I just ran jextract on some files and tried to generate the same output.

I created a project page and a repository and dubbed it 'panamaz' in an unimaginative following of tradition.

export.cc

The first part of the problem is getting an accurate dump of the c types and prototypes you want to use. jextract is built using clang, but I've had a tiny bit of experience poking about that and I really didn't like what I saw so I thought I'd try using a gcc plugin.

I didn't look very far but i found a couple of bits of example code and opened up gcc-internals.info and got to work. And it was a lot of work. I can see why ICEs were once so common in gcc - you get no compile time errors for bad type assumptions of the syntax tree, and when you do the wrong thing sometimes it will work anyway. The documentation isn't completely accurate either, which is pretty annoying considering how complex the system is.

But anyway in the end of I have a nice tool that can dump considerable information about c header files. The output format is a better, more usable, canonically-decoding version of 'json', or what one might term 'Perl, son'.

The following simple example:

struct bob {
    int a;
    float b;
};
  

Produces the descriptor file:

%data = (
'struct:bob' => { name => 'bob', type => 'struct', size => 64, fields => [
        { name => 'a', size => 32, offset => 0, ctype => 'int', type => 'i32',},
        { name => 'b', size => 32, offset => 32, ctype => 'float', type => 'f32',},
]},
# dumped structs:
# bob
);

generate

This is a (mostly!) better-than-i-usually-write bit of perl which takes the 'perl-son', and turns it into a bunch of java files. These are all interfaces which have the correct annotations to be used by the project-panama jdk to perform runtime linking.

It's got a lot of flexible options but the one I like is that you can specify the set of functions and/or types you want and it will recursively drag in all the types they depend on automatically, and only those. At least for most of it, I still haven't done the same for callbacks yet because they're a bit fiddly.

It ignores enums for now because I just ran out of juice but it covers all the other major parts of the language and even some of the lesser parts.

panama jdk

Most of the information is conveyed in signature strings which are passed to the jdk via annotations in interfaces. The simple example above is translated to an interface with the @NativeStruct annotation that includes the two field names and their types in order.

import java.foreign.annotations.*;
import java.foreign.memory.*;
@NativeStruct(value="[i32(a)f32(b)](bob)", resolutionContext={})
public interface Bob extends Struct {
        @NativeGetter(value="a")
        public int getA();
        @NativeSetter(value="a")
        public void setA(int value);
        @NativeGetter(value="b")
        public float getB();
        @NativeSetter(value="b")
        public void setB(float value);
}

Accessors can be named anything (jextract uses the uber-friendly notation of name$set(...) and so forth) but the field they operate on is indicated using the @NativeGetter or @NativeSetter attribute and specifying the field name, which is defined in the @NativeStruct annotation. There's also an address-of annotation for a rather verbose equivalent of the & operator.

I didn't come across a document describing this format (and I didn't look beyond the annotation source code) but I worked it out by seeing what jextract did. Most of it is pretty straightforward.

The bitfield one is a bit odd, I'm not sure why one has to specify an internal detail such as the actual word-size used for storage, but I don't use bitfields much. Perhaps it's to supports some alignment/packing __attribute__ in bitfields which is used to map hardware registers, although that seems rather obtuse usage for Java api bindings.

So basically I kept banging at the exporter and the generator until I got something working, then tried to compile it and then fixed those problems. Then tried instantiating the object and fixing the breakage. Then tried another type and so on and kept going until it all worked. And then I rewrote the exporter almost entirely to clean up the code. And then fixed the new bugs. Then created a simple example.

Then created a repository and a project page, then sent an email to the panama-dev list, then wrote a blog post ...

Vulkan + OpenCL?

I've been slowly working on some OpenCL on Vulkan stuff using clspv. It's a bit too hot here, my heart isn't really in it, but I'm slowly making progress.

The last thing I checked in was an example that uses more complicated argument passing conventions and uses a small compiler to turn the descriptor map file from clspv into some tables used for initialisation and argument setting. Actually I wrote a patch for clspv to output the data type so the compiler can use it.

Of course along the way I hit fairly obtuse errors from vulkan regarding extensions and options which need to be enabled for the small kernel I wrote, but thanks to the validation layer they were able to be resolved without too much work.

I'm working on abstracting a lot of the details away into a more OpenCL-like API, the first few examples were stand-alone but there's really just so much boilerplate it makes it too confusing for an example. I've discovered a few options like VK_DESCRIPTOR_BINDING_UPDATE_AFTER_BIND_BIT_EXT and it's associated settings that are required for the kernels to behave somewhat like OpenCL where you can modify kernel arguments for subsequent submission. From my preliminary work I think most of the behaviour can be recreated, although the more limiting factor is the compiler and SPIR-V.

It can be found inside the opencl-args directory of zproto-vulkan, but I'll probably re-arrange the directory structure at some point and continue to work on the abstraction. As of this post it's pretty much just when I got it working at all and I didn't try to work on the queuing mechanism.

busymon - tool to force computer breaks

I had need of a tool to remind me (forcefully) to get off the computer and get up and walk around for a while. I was surprised I couldn't find anything simple to do this; I came across rsibreak but it looked like a mess to compile (i mean, cmake, obviously). I looked a bit more but nothing looked simple.

So here's a basic computer monitor that tells you off if you're at the computer too long - as a proxy for sitting.

As usual busymon has a software page and a repository.

In short my bicycle accident has lead to further complications, possibly in part due to sitting down for too long infront of my computer. So now i'm riddled with clots (some poetic license) and on blood thinners and should probably try not to make it worse. Hourly walking breaks are now doctors orders, and well, time just flies sometimes.

amdgpu / x570 / sea islands / IO_PAGE_FAULT

After turning on the amdgpu driver on my new system (radeon.si_support=0 amdgpu.si_support=1) so I could play with vulkan it seemed nice and stable.

But I was only playing around in emacs/netbeans/firefox and wasn't doing much with the graphics system. When I ran a JavaFX thing i've been playing with (the genetic art) and eventually it crashed the graphics driver. So I went back to radeon for the time being until I decided to look at it again.

Anyway today I had another look, and whilst logged on remotely I got the error log:

[  993.135454] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe05000 flags=0x0000]
[  993.135461] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe0a040 flags=0x0000]
[  993.135466] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe338c0 flags=0x0000]
[  993.135471] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe1d100 flags=0x0000]
[  993.135476] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe38900 flags=0x0000]
[  993.135481] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe5dd80 flags=0x0000]
[  993.135486] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe23580 flags=0x0000]
[  993.135491] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe25c00 flags=0x0000]
[  993.135496] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe4ed00 flags=0x0000]
[  993.135501] amdgpu 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0xffffe07a80 flags=0x0000]
[  993.135507] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe52a00 flags=0x0000]
[  993.135512] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe55200 flags=0x0000]
[  993.135517] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe5a200 flags=0x0000]
[  993.135521] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe5f2c0 flags=0x0000]
[  993.135526] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe10700 flags=0x0000]
[  993.135531] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe51600 flags=0x0000]
[  993.135535] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe606c0 flags=0x0000]
[  993.135540] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe4c7c0 flags=0x0000]
[  993.135544] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe53e00 flags=0x0000]
[  993.135549] AMD-Vi: Event logged [IO_PAGE_FAULT device=09:00.0 domain=0x0003 address=0xffffe58e00 flags=0x0000]
[ 1003.712058] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=144954, emitted seq=144956
[ 1003.712116] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process Xorg pid 1301 thread Xorg:cs0 pid 1303
[ 1003.712118] [drm] GPU recovery disabled.

There wasn't a lot to be found on the internet regarding amdgpu but there were some other similar problems. The suggested fix was to add iommu=soft to the kernel arguments. I'm trying this now and so far it looks good!

I have been tracking the 5.4.x kernel releases and thought that might have something to do with it but fortunately not. When it crashed initially i'd just done a slackpkg upgrade-all and all sorts of things broke horribly. But that was my fault for overzealously removing packages that didn't look important: polkit uses a lot of snot unfortunately so I broke that. Fucking firefox lost all my settings though!

Zed's Bread Baby, ...

So I haven't been spending all my time just on the new computer, I've also been doing some cooking. It's quite a laborious process and the crutches keep getting in the way but most things only require short stints standing while the oven or mixer or some natural process takes it's course. Reminder: I'm on crutches until after xmas due to a broken hip. I can't go anywhere or do much or carry anything and even beer makes my foot swell up so I can't even bloody well get drunk.

Yesterday I made a pretty good approximation of a baguette - perhaps a bit too soft but it made a great chicken sandwich for dinner. And today I made a '(kneaded) no-knead' bread, and a couple of ciabatta.

A dutch-oven bread and a couple of ciabatta.

I sort of fucked up the latter because the starter was too cold in the fridge so the final dough was far too wet, but perhaps it turned out OK. I made some last week and didn't cook it long enough but otherwise it was an interesting bread with an almost rubbery texture quite unlike the standard recipe.

I'm going to write down the recipes I used here in part so I don't forget them. One might notice a lot of similarities in the steps - well that's just how you make bread I guess.

I don't normally bother adding salt but most bread recipes include it, perhaps 0.5 tsp. I notice the yanks put sugar in their recipes - sometimes alarming amounts - so maybe they just need the salt to make it palatable!

The bread improver could just be more flour. I use general purpose flour which is a bit too weak (low protein) to make good bread, and this helps make up for it a little bit. Adding an egg works really well but then you need to adjust the dry/wet ratio to account for it (just add a bit more flour usually).

Another trick which works on any of them is to spray water lightly a few times while it's baking. This will create a shiny hard crust by simulating a steam oven. I usually do it if i remember to as it doesn't take much work and is a nice touch.

I use a standard-sized kenwood chef mixer, and I have a new Bosch oven that has an almost moisture proof seal (nearly steamed my face off a couple of times opening it). I highly recommend a scraper (aka "baker's helper"). Finally the cooking times are a bit arbitrary and you can always go longer if you want more crust.

baguette

1. Put the flour and yeast in the mixing bowl and add the water.
2. Start the mixer with the dough hook on low until they combine then run at speed 1 for about 15 minutes.
3. Set the machine on low and add some olive oil, then turn the machine off. Remove from the dough hook and turn until an oil-coated ball is formed. It makes it easier to work with and stops it drying out.
4. Either leave it in the mixing bowl or transfer to another and cover with a plate. Leave somewhere warm to double in size.
5. Punch down and turn out onto a liberally floured surface and roll out the bubbles. Cut in halves and fold over thirds several times until a rough log is formed. A bakers scraper helps a lot here.
6. I have a funky 80s baguette form that I then place them in.
7. I let them rise in my oven set to 30C, and spray some water on them to stop them drying out. I then remove the tray from the oven and set the baking temperature.
8. Cook at 220C for about 25 minutes or until they look done.
9. Remove onto a wire rack immediately once done.

Up to about 100g of the flour can be replaced by other flours like semolina (more moist) or bulgar wheat (whole-meal like). This is probably a bit wet for a true baguette but it makes great rolls and buns as well.

ciabatta

1. Take 200g of the flour, the yeast, and 200ml of the water and mix in a bowl until a slurry is formed. Cover and leave for 12-24 hours somewhere at a cool room temperature. It should rise and puff up and go stringy.
2. Put the remaining (200g) flour(s) in a mixing bowl. Add the rest of the water (200ml) to the starter to help remove it and pour that into the mixing bowl.
3. Set the mixer running and let it go for about 15 minutes. Put in the oil and try to form a (wet) ball using a scraper.
4. Either leave it in the mixing bowl or transfer it to another and cover with a plate. Leave somewhere warm to double in size.
5. This is likely going to be messy. Pour from the bowl onto a very floured surface and use the scraper and adequate flour to form two soft logs of dough.
6. I then transfer the logs to a deep baking dish with a piece of baking paper folded in the middle to form a barrier.
7. I cover that with another tray and let them rise.
8. Cook at 220C for about 25 minutes or until they look well done.
9. Remove onto a wire rack immediately once done.

This is a simplified version of a stupidly complex multi-stage recipe I found on the internet. I've only made it a couple of times so I might need to make some changes.

Update: So this just isn't working right, yet. I tried a bit last night and it's texture is more like a crumpet than anything else! I'm not sure if I'm just not letting it rise enough before baking it, or the mix is a bit too wet, although I suspect it's a bit of both. Yesterday I ran out of hours in the day and cooked it because the oven was hot but I probably should've left it (much) longer. I will need to experiment more, but will try with a reduced recipe as this makes a lot of bread!

Update: Take 2. I reduced the water and flour a bit and let the formed loaves rise for much longer. The bread itself is soft and 'bready' with a slight sourdough taste. The crust is OK but not thick enough, it probably needs more time in the oven. The dough is still so wet that it 'runs' as it proves to fill the available space, I'm not sure how I could get around that other than less water again. It's nice bread however.

Dutch Oven Bread

1. Put the flour and yeast in the mixing bowl and add the water.
2. Start the mixer with the dough hook on low until they combine then run at speed 1 for about 15 minutes.
3. Set the machine on low and add some olive oil, then turn the machine off. Remove from the dough hook and turn until an oil-coated ball is formed. It makes it easier to work with and stops it drying out.
4. Either leave it in the mixing bowl or transfer to another and cover with a plate. Leave somewhere warm to double in size.
5. Carefully turn out onto a liberally floured surface and use a scraper to fold a few times and form into a ball. Do not punch down.
6. Line a bowl with baking paper and place the ball into the middle. Cover them with a plate and let it rise.
7. When it is nearly done rising put an enameled dutch oven with it's cover on into the oven and pre-heat to 220C.
8. Using the paper, carefully transfer the risen dough into the put and put the lid back on.
9. Bake for about 20 minutes, remove the lid, and bake for about 10 minutes more or until done. If you're keen you can remove the paper at this point but it doesn't make a lot of difference.
10. Remove onto a wire rack immediately once done.

This was based on a '2 hour' 'no-knead' bread recipe but then I realised I was using the mixer anyway so why not just let it do it's thing. The original made more of a bread-cake, and this makes proper bread but with a 'rustic' look and a great crust. I just had a piece and it's really very good.

This one can likewise have some of the flour replaced.