Panama Experiments

After a pretty long break I started playing with the OpenJDK panama branch again - this is the in-development mechanism for Java to be able to call C functions using metadata rather than requiring a C binding library via JNI.

Whilst I get the impression the developers just aren't very experienced with the way C works and keep trying to shoe-horn C libraries into a fixed and awkward model of how they think they should work, overall it's much improved than where it was last time I looked. Of course it's a moving target still so even the updates i've made will break shortly.

Apart from updating to changes in the panama api the main changes have been working on a more flexible api compiler utilising programmable templates, and at least attempting to move away from write-once perl. It's surprising how much more work this took, about 5x the effort and you don't seem to get a lot of benefit out of it initially.

I also have a vulkan wrapper that uses the vulkan registry xml document as the sole source. This allows it to better group various features and automagically generate a nice object-oriented api. One feature it has is auto-generating 'nice' constructors so using the api is pretty much equivalent to the C api, it can also auto-imply some parameters like array lengths. This was an earlier iteration of the code so is definitely write-once code, i'm considering trying to covnert this to use the templating framework from the other generator but it's a huge amount of work.

I'm still working on how ResourceScopes can fit with the various api's generated. In some cases it's an ok fit, in others it's has a very large impedence mismatch which is difficult to resolve in a satisfactory manner. It would be nice if there was something with a bit more granular control available as once you start using scopes you're forced to use it everywhere, or just not use it. They map to problems where you have a bunch of resources you allocate incrementally and then free together, but there are many scenarios in C apis that just don't work like this at all. I've had some discussions on the mailing lists but so far i'm not terribly satisfied with the offered solutions, we'll see I suppose.

The REAMDE has a bit more info.

gcc export plugin

One tool that might be useful for others is the gcc plugin to export all the available type information. It's a pretty messy bit of code because it was mostly written by trial and error and viewing the output of debug_tree(). I admit I didn't read the gcc-internals documentation enough but it's pretty dense and hard to follow, and gcc plugins are few and far between.

Getting the named parameters for function declarations was particularly problematic - in some cases it's simple as you get the parameters, then the function type itself. But an untyped function pointer declared as a field, or as a parameter, comes to the plugin in a different way, sometimes only being able to be resolved after the plugin is finished. I still have some 'dangling' function parameters left over but they don't seem to be important and even on a very complex set of header files like ffmpeg all the fields and named parameters are being resolved properly.

The output of the plugin is a perl hash which completely describes all the structs, unions, function pointers, and enums found in the header files.

Because #defines aren't available to gcc plugins a separate programme is used to generate these from the output of cpp. Because #defines are just C it isn't trivial to convert these to real values so the programme generates a C file which is then compiled and executed to generate the perl output. It uses some gcc features to create a pseudo-function overloading to imply the type of #defines, and has flexible filtering mechanisms to only grab those of interest. It's a bit fragile but can be controlled to avoid some broken cases. Also due to the need to execute the extraction step cross compiliation will be difficult or impossible but that's a hurdle for later.

Have at it

I spent most of January working pretty solidly on this new generator but i've not felt like working on it much for a little while. The ResourceScope stuff is just about to be renamed (the justification seems pretty weak to me but whatever) so it'll all break but i'll patch it up when those changes settle down.

Browse the source code, the git checkout url is https://code.zedzone.au/git/panamaz.

I'm using the foreign-jextract branch of https://github.com/openjdk/panama-foreign.git.