Compilering

So a few months ago Xyhpoid contacted me about the work I did on Dusk some time ago to see if I still had the source. Indeed I did, but I hadn't touched it for 10 years, left it in a half-arsed state, and can't remember much of where it was at or what I wanted to do with it.

Still, at the moment i've got a lot of time on my hands and not much to do with it so I started poking around again. Amongst some minor phaffing about I decided to look at writing a compiler for the dusk script language that compiles directly to the JVM.

It's been a somewhat interesting exercise, from parser for a custom language to generator for class files. A probably goal is to create a new dusk script - it will be something like a relaxed java but with more security by limiting access to classes and functions via white-lists.

Apart from something interesting to play another motiviator is the dropping of nashorn from the OpenJDK. I did a quick survey of extant JVM languages looking for something sutiable but despite a proliferation in the earlier days of Java most have been abandonded, and even those that haven't haven't updated to handle the java modules system or the deprecation for removal of SecurityManager. Another problem is that most give unfettered access to the whole JDK in one way or another which isn't acceptable for my use case.

Anyway some of the exploratory work is going on in the compilerz project. There is also some work going on in duskz but i haven't pushed it stream yet.

JDK 20, foreign-abi

Finally updated the foreign-abi branch of nativez, zcl, and jjmpeg to the OpenJDK 20 API for foreign native access. It's very much untested and probably buggy but it compiles and a few things run so it should be on the right track at least.

Most of the changes were pretty straightforward and some of the API changes simplified a few things.

It was mostly a learning exercise to find out what has changed in the API but I thought I may as well do them all at the same time. I'm sort-of more interested in vulkanz but it's been so long since I worked on it I've forgotten where I was at. So maybe i'm not that much interested in it!

Cancelled FSF membership

So after 15 years of financial support I decided to end my Free Software Foundation membership this year, I think it expired officially yesterday.

There are a few reasons but none are any reflection on the FSF as an organisation.

Primarily it's because i'm not working at the moment as mentioned in previous posts. I'm not struggling financially or anything but need to consider my budget. I more or less got the membership as a birthday present to myself one year and i'm about to turn 50 in a few days so it seems like a good time to consider the future a bit more closely.

A lesser part is that it's an organisation based in the USA and their political action is necessarily focused there - given i'm in Australia it has a fairly weak impact locally.

I also feel (rightly or wrongly) that when I joined the focus of the FSF was primarily on the GNU project - i.e. the software required to achieve their political goals, and now the focus is more on the political side of software freedom in general. I acknowledge this is reflecting a change in the software and political atmosphere over the last few decades; and further something that needed to happen. However I think my time and resources would be better spent elsewhere.

I guess I also think $US 3500+ was a pretty significant amount of money and I think i've more than done my part as a lowly pleb of a private citizen given there are multi-billion dollar corporations who owe their existance to what the GNU project has wrought.

Formatting menus

Update 1/20/2023: So the bar manager decided to try to use it, but in the end gave up. Chrome has a rendering bug in print media that pushed the gin list to the next page immediately after the title. Given the css and html is so simple there isn't anything I could do to fix it. Even without that the process of generating HTML, then printing to PDF, then running it through a bootklet generator before finally sending it to the printer is just too much hassle.

I was at the pub a few weeks ago and saw the bar manager updating the drinks list. The process seemed tedious and error prone, he seemed to be positioning page elements by hand, so that alterations required more work. I don't even know what software he was using - he later mentioned google docs but it would have to have been the slideshow program because none of the others work like that. Looking closer at the menu there were multiple formatting issues - inconsistent margins, indenting, even a few typos ("Rosè" being the most onerous).

Seemed there should have been a better way ...

But after working on the problem quite a bit over the last few weeks i'm not sure there really is. I have some solutions but they require different skills to use.

Latex2ε

My first thought was using latex to generate the menu. I've played with it over the years but never used it much so it took a bit of trial and error but eventually I came up with something that worked pretty well. It could read the price lists from tab-separated-values files, I had some macros to control layout of individual items, I used the options package to allow fine-tuning on individual sections and so on. I even wrote a minimal table editor in Java to simplify its use and worked out how to get it all running in windows.

Latex can be frustrating as fuck, there's some weird environments that don't nest, arithmetic is messy, it's hard to read. But you can also do some cool shit, e.g. I created a brand logo that will properly typeset in any font because it can work with the glyph metrics directly.

I had to disable or work around many of latex's typesetting facilities such as hyphenation and paragraph layout but it produced high quality output and could create an A4 booklet print directly.

But to be honest, it's just not a good enough user-experience. It demands far too much knowledge to modify anything but the list of prices and even though it's simple to adjust things like font sizes the absolutely useless error reporting from latex just doesn't make it practical. And I also found out they don't really have the setup anyway (although i'm not sure if that's just a miscommunication since I seem him using laptops with windows all the time).

HTML5?

So I went back to the drawing board. I tried using HTML+CSS but Firefox didn't seem to support enough fine control on formatting for the task - and always added unacceptable headers and footers at print time.

I discovered that texlive has an online formatting service so started working toward a solution that would utilise that. I installed a local copy of the server software (seems unmaintained and insecure but it works well enough) and worked out how to invoke it.

Then I basically went on an epic side-quest to catch up on about 20 years of web development that i'd thankfully missed during my software engineering career. What a weird collection of technology, bizarre toolkits, and shitty jargon (e.g. pollyfills = fills cracks in walls vs pollyfilla fills cracks in a browser's javascript implementation).

Along the way I learnt about a bunch of technlogy:

• LocalStorage;
• IndexDB (neat ... but not really suitable);
• Modules;
• Custom Components (because of the way templates work they're not anywhere as useful as they could be);
• DOJO (Wwhy? Just ... why?);
• PetiteVue (It's sort of cool but the documentation is miserable).

Although I still skipped some of the more prominent stuff like node.js or jQuery.

In the end I chose to use PetiteVue and modules came up with all the parts I'd need to make an application of it:

• A spreadsheet-like table editor;
• A graphical layout editor using drag and drop;
• Various I/O like saving/loading locally, cut and paste;
• Tabbed panels.

But then I ran into a mess trying to combine these parts into a single scope with petite vue and the lack of documentation became a real problem. I did manage to solve the problems but it became tedious and boring and I simply lost interest. Another issue is it still requires hosting somewhere.

Google Apps Script

So I looked into various was of using google apps. I'd rather just use Free Software but it would be interesting to contrast with something proprietary anyway. This was still frustrating as fuck as there are two apis for everything (e.g. Spreadsheets vs SheetsApp) and trying to navigate the documentation just sends you around in circles. Worse is that the documentation page renders wrong in firefox and with the fonts I use so it's hard to even read.

I spent about a week creating an apps script project that could talk to a collection of files - a spreadsheet with data, a doc with templates for items and a doc with the overall layout - only to find that it's simply impossible to insert a column break in a document programatically. This is a total deal-breaker as there is no other way to control the layout so it would break the formatting - e.g. splitting a pair of rows that should be presented together like the drink name and a description.

A workaround would be to have put both columns into a table and then use the javascript to distribute the values across both tables but I was trying to avoid having to special case template handling in the code. And I was so pissed off with finding out the limitations I basically turned it off and went for a walk and haven't revisited it.

One thing I found is that the google docs api isn't really geared to this type of thing - it's solving the very different problem of a distributed multi-user editing environment. This impedence mismatch made it a pain to work with.

The other pain is the abysmal fucking editor you're forced to use in Apps Script projects. It fucking pops shit up over your text almost every time you press a key, and even if you learn to ignore that incredibly irritating obstruction you have to hit escape all the time anyway as otherwise the cursor keys aren't moving the cursor but sliding down a menu - so you're CONSTANTLY hitting escape to close that shit out of the way. And then when you try to use it the Javascript autocompletion is also a complete joke and the popup documentation is either missing or hard to read. The formatting tools are miserable. It doesn't seem to know what the fuck a tab character is. It doesn't handle primary selection at all.

And anything apart from editing is so slow and takes too many clicks.

And I still I was wasn't sure how to turn it into a standalone app, hosting, and the authentication stuff is weird.

Google Sheets and Apps Script and HTML5

I decided to revisit HTML and see if i could get enough control to generate a good printed output via CSS. And it turned out you could - well mostly could anyway. It fucking also has issues with column breaks but they could be worked around. And you need to edit about:config to stop Firefox adding headers and footers - but hardly anone uses Firefox these days anyway.

I first implemented it as a web-app associated with the spreadsheet. I ran into all sort of issues because the apps are run inside a sandboxed iframe. Made it hard to debug and completely breaks printing.

But I suppose the long story short is I got it to work. In the current version the apps script pops up a dialogue box which inserts a html file. The HTML file does all the processing to insert the data into the page template which is a html template. A button opens a new window with a printable version or lets you save it a a standalone pre-formatted html file.

It's pretty slow - it takes longer to retrieve the data than it did to run latex - but the output is acceptable.

The Result

---

---

Summary

Latex is still king for generating printed output, but the user experience just isn't there for anyone not prepared to spend a lot of time learning how to use it which is an unreasonable expecation for the general public outside of authoring books.

Web development still sucks donkey dicks. It's slow and frustrating. The debugging experience is crummy. The online tools are a pain to use. Everything is too blindingly white - or if they have an alternative theme it's both too black and psychedelic. Javascript is a weird language - you can certainly do some pretty interesting stuff with it but it's so easy to abuse. CSS is very powerful but still let down by implementation issues no doubt due to it's complexity.

I'm mostly astounded at the amount of human effort that has been put into 'Web 2.0' then 'HTML5', and this is what they came up with? But that's what I think about C++ too.

Having said that, it had it's moments of fun like any coding exercise - through exploration, implementation, and finally refining the implementation and details.

Server News

I just upgraded the server from a rather out of date ubuntu to debian. Seems to have gone mostly smoothly so far although I haven't quite finished reconfiguring everythying.

This is more or a less a test post of blogz.

More on project panama

So i've continues to work on code utilising project panama/jdk.foreign to bind directly to C apis. Most of the work has gone into one of the vulkan bindings in panamaz - i started a 3rd one that creates the api directly from registry but utilises templates heavily to be a bit less hairy than the first attempt. It's still grown to be pretty hairy but at least it runs fast and generates a good quality api. I might split it out into it's own project soon as netbeans is getting a bit unwildy opening panamaz with so source files to work with.

Now i'm going through the vulkan tutorial and using tha to refine the api a bit further. I'm working towards using signed distance fields for digital art of some sort. What? Who knows, i'm lacking inspiration.

Also my heart isn't really in it at the moment so it's a pretty slow burn. I've stopped tracking the development version and i'm just using openjdk 18 - the primary reason is that netbeans wont open projects properly when i use my development builds so i've kinda given up. The api has moved on quite a bit from jdk 18 so it will be quite a bit of change when it comes to update unfortunately. I'd been using emacs to get around this but without code completion Java isn't a lot of fun.

notzed.nativez, java.make

I've been working on an update java.make which removes the JNI bits and replaces whem with jdk.foriegn bits via an updated notzed.nativez (jdk-foreign branch). I had made a couple of mistakes: using .PHONY targets as dependencies just doesn't work, and trying to hook everyting to the 'classes remade' sentinal wasn't sufficient granularity. I've added a few more sentinal markers which can be used as dependency targets and this has greatly simplified the dependency setup and allows builds to properly parallelise.

Long term i'll probably update jjmpeg and zcl to use these new mechanisms but for now they are not a priority.

I have another background project going to try to optimsie the build process. I have almost all the code done in both c and java to handle accurate incremental builds, e.g. it can find out what needs to be remade, what needs to be deleted and so on, run a compiler server, and generate makefile dependency rules, but for now i'm using javac with '-m'.

netbeans

Along the way I ran into some performance issues with Netbeans and java code completion. As part of the code completion it runs through a routine which filters out the visible symbols based on Java's scoping rules, but the routine uses an N^2 algorithm, and to make matters worse the inner loop does a string comparision based on output of CharSequence:toString() - which is very expensive, genereates a ton of garbage, and by far the majority of the time expense. Even the somewhat modest 3000 constants from vulkan will slow the routine down to around 500ms on a ryzen cpu. Oddly there was alreayd a solution to this coded up but for some reason it was never hooked up to all uses of the visibility check. I reported it with some potential fixes and then found out jira is no longer the bugtracker for netbeans. Sigh, it sucked but at least it wasn't github.

Netbeans code completion seems more broken than it used to be as well since they switched away from the gpl licensed nb-javac. It wont really complete properly until you've added the import and syntax errors (you know, the ones that exist as you're writing new code because it's decided to repase it mid-phrase) will break all sorts of things. Now i've got a codebase I can edit i might see if i can at least build a local copy without all that tooltip bullshit that keeps getting in the way of what you're doing. Be nice to fix the slow performance of the output window too - there's no reason it should be so miserably slow.

jdk.vector

A few weeks ago I also played a bit with the vector api which allows one to access SIMD instruction sets natively in java. It's a bit better than I thought it was when I first looked at it (it seemed very clumsy), but it's also less abstract than I had hoped. It's more or less a mapping to AVX intrinsics - you need to specify the vector width and although you have abstracted intrinsics you must code to the platform the same way you need to with intrinsics otherwise the performans tanks. Also suffers from the general problem of compilers over-usign registers so it's easy to get spills if you try to unroll anything.

I was mostly poking around with graphics related maths functions like matrices, one routine was a 4x4 matrix inversion using gauss-jordan elimination. I tried to unroll the inner loops entirely using SIMD selection logic to implement the row sorting but it didn't create much speed-up. Oddly I tried a branchy bit of code that generated some very odd hotspot output that executed microbenchmarks quite a bit faster - it seems to branch off to a sort of exception point, re-arrange the registers via memory swaps and then re-enter the main loop through some hotspot function. Kinda hard to follow what was going on with the assembler output. I might post about this later if i revisit it.

Panama Experiments

After a pretty long break I started playing with the OpenJDK panama branch again - this is the in-development mechanism for Java to be able to call C functions using metadata rather than requiring a C binding library via JNI.

Whilst I get the impression the developers just aren't very experienced with the way C works and keep trying to shoe-horn C libraries into a fixed and awkward model of how they think they should work, overall it's much improved than where it was last time I looked. Of course it's a moving target still so even the updates i've made will break shortly.

Apart from updating to changes in the panama api the main changes have been working on a more flexible api compiler utilising programmable templates, and at least attempting to move away from write-once perl. It's surprising how much more work this took, about 5x the effort and you don't seem to get a lot of benefit out of it initially.

I also have a vulkan wrapper that uses the vulkan registry xml document as the sole source. This allows it to better group various features and automagically generate a nice object-oriented api. One feature it has is auto-generating 'nice' constructors so using the api is pretty much equivalent to the C api, it can also auto-imply some parameters like array lengths. This was an earlier iteration of the code so is definitely write-once code, i'm considering trying to covnert this to use the templating framework from the other generator but it's a huge amount of work.

I'm still working on how ResourceScopes can fit with the various api's generated. In some cases it's an ok fit, in others it's has a very large impedence mismatch which is difficult to resolve in a satisfactory manner. It would be nice if there was something with a bit more granular control available as once you start using scopes you're forced to use it everywhere, or just not use it. They map to problems where you have a bunch of resources you allocate incrementally and then free together, but there are many scenarios in C apis that just don't work like this at all. I've had some discussions on the mailing lists but so far i'm not terribly satisfied with the offered solutions, we'll see I suppose.

The REAMDE has a bit more info.

gcc export plugin

One tool that might be useful for others is the gcc plugin to export all the available type information. It's a pretty messy bit of code because it was mostly written by trial and error and viewing the output of debug_tree(). I admit I didn't read the gcc-internals documentation enough but it's pretty dense and hard to follow, and gcc plugins are few and far between.

Getting the named parameters for function declarations was particularly problematic - in some cases it's simple as you get the parameters, then the function type itself. But an untyped function pointer declared as a field, or as a parameter, comes to the plugin in a different way, sometimes only being able to be resolved after the plugin is finished. I still have some 'dangling' function parameters left over but they don't seem to be important and even on a very complex set of header files like ffmpeg all the fields and named parameters are being resolved properly.

The output of the plugin is a perl hash which completely describes all the structs, unions, function pointers, and enums found in the header files.

Because #defines aren't available to gcc plugins a separate programme is used to generate these from the output of cpp. Because #defines are just C it isn't trivial to convert these to real values so the programme generates a C file which is then compiled and executed to generate the perl output. It uses some gcc features to create a pseudo-function overloading to imply the type of #defines, and has flexible filtering mechanisms to only grab those of interest. It's a bit fragile but can be controlled to avoid some broken cases. Also due to the need to execute the extraction step cross compiliation will be difficult or impossible but that's a hurdle for later.

Have at it

I spent most of January working pretty solidly on this new generator but i've not felt like working on it much for a little while. The ResourceScope stuff is just about to be renamed (the justification seems pretty weak to me but whatever) so it'll all break but i'll patch it up when those changes settle down.

Browse the source code, the git checkout url is https://code.zedzone.au/git/panamaz.

I'm using the foreign-jextract branch of https://github.com/openjdk/panama-foreign.git.

A year down

It's been a year since I decided to quit my job, well yesterday was to the day. Also a Friday. At the time I put two weeks notice in but they completely stopped communicating with me after one week so I just gave up.

It's been a weird year. In some ways it seemed to drag on, in other ways it seems to have gone fast - I guess in general i've just lost track of time since i've got nothing in particular to mark the days. I've spent way way too much time (and money) at the pub, and it's getting to the point that I have to try to move on. It's making me feel like a total loser just hanging around mostly drinking by myself - or worse drinking with some random who turns out to be a total fuckwit (last one started telling me Hitler was only trying to protect his country ... sigh) or someone who gets funny on booze. And somehow my circle of friends keeps shrinking, I can probably count on one hand 'friends' i've seen in the last 6 months, and I might not even need another hand for good aquaintances. I suppose the thing which most upsets me is I just don't have any other way to socialise, and although I don't mind a good booze-up, it's just too bloody much doing it almost every day. I try to think of things but even without COVID making a mess of group activities I really just don't like doing organised shit anyway. Maybe it's my lot in life to live alone until I die - it's certainly been a consistent experience so far.

I cut back on the pub a lot this week anyway and spent a lot of time hacking on zcl/foriegn-abi and playing games. Going through God of War, although I don't really how it turned into an open-world RPG - it's just not the same sort of game anymore and i'm a bit lost and well, basically grinding to get enough gear to progress. Sigh. I also got Gran Turismo Sport because it was on sale - bloody 120GB download (what for? shitloads of crap videos i don't give a fuck about?), I dunno it's 'ok' I guess and the Mt Panorama circuit has a tigher (narrower, more realistic) feel to it, but it just seems like i'm playing GT3 again. At least the license tests are easier.

I've continued to work on fitness and general health - been on some long rides (many 60km, one 80km) and even started going for a swim at the beach but it's been cold and windy all week so that's dropped off for now. Down to about 76Kg which is about 'ideal weight' according to probably-questionable sites I can find on the internet (my old scales were bunk but I think I was around 88Kg 2 years ago when i broke my hip). Of course I look pretty skinny but people are just oversized these days. I started having insomnia troubles so I'm trying to fix that using sleep restriction (CBT-I) with ... mixed results so far. I'm mostly struggling to stay up till midnight every night if i'm just at home alone playing games on the couch or using the pc. And so if I nod off for a bit around 10 or 11 it makes it harder to get to sleep at 12. Oh well it's an ongoing effort and maybe it's getting a bit better, it's one of those things that's hard to tell because you can't remember much from lack of sleep.

Fish as anti-deppressant?

I had to go the GP for something else and mentioned the insomnia and a lot of anxiety i've been having lately ... and it just boiled down to psychologist or anti-depressants. Neither of those worked for me in the past and the 6(?) pills I tried either did nothing (with shitty side-effects) or made things worse, sometimes much worse. I took the script because I couldn't remember if i'd had that one before and lo and behond - yeah, and it was another failure. So I decided to try something else - eat fish every day. Dunno if it was just a placebo but it had an immediate effect, literally the next day my mood felt better. Well not the mood exactly but it's like a mild 'warm glow' that settled into my body below my stomach which makes it easier to just feel 'good'. Sort of what one would hope of taking an anti-depressant although none I ever took had such a significant, lasting and well, completely side-effect-free result as simply having a couple of sardines or mackeral once a day.

It's been 5 or 6 weeks since I started and it's remained - and already helped me weather some shitty situations. Improved my skin a bit too, although I was hoping it would fix my cold hands and feet too but it hasn't. It was such a foreign feeling to start with, you know, not feeling miserable all the time.

Semi-Retirement

So I dunno if i'll work again - if i live cheaply and nothing major changes in my life I can probably afford to not work.

But fuck it's boring. And lonely.

So many hours to fill every day and not much to fill it with. If I walk to the pub I'll spend 2 hours walking but there's still so much day to go. A decent ride might be 3 hours or 4 if i'm really keen. Can only read or playstation so much. Gardning. And yeah people get sick of hearing how boring it is when they're too busy to get any time to themselves (maybe that's why my circle of friends keeps shrinking!). I can do some coding and so on but I dunno what's the point?

The loneliness is the real killer at the moment. Sigh. Well for now i'm trying to fix things one at a time - physical health, check, insomia, work in progress, then?

Guess i'll go for a ride to the beach today, although it's set to be warm the water will probably be too cold for a dip but I might get keen. YOLO and all that.