from link to dlsym call

I've been spending an inordinate amount of time the last few days playing with make and doing some work on zcl. Been staying up way too late and then doing it all again the next day. Been a bit of fun poking at little problems though.

The java makefile system i've been working on has got a little fiddly due to some features i'm trying to implement; but so far all problems have been solvable in a fairly straightforward way and it's mostly now taking time because i'm writing some fairly long article(s) about the work. Most of the complication is with generated source and resource files and dealing with the command line to jar which is like but not quite the same as tar.

So I took a break to play with zcl again. Because the sdk version of libOpenCL may not match the system version i've added an option to compile with dynamic linking - i think the linker can also do this automagically but it's something i'd need to learn about first. But just changing it over turned out to be pretty easy an infact most of the work is hidden in a single auto-generated file together with a few small bits of checking to throw exceptions when functions are not available.

I wrote a horrible bit of perl that parses the very strict format of cl.h and creates a list of static function pointers and an init function to set them up, as well as C macros which automagically change the source to go via the function pointers rather than via dynamic linking.

So for example it creates the following bits for a given function:

...
static cl_int (*pclBuildProgram)(cl_program, cl_uint, const cl_device_id *, const char *, void (CL_CALLBACK *)(cl_program, void *), void *); 
...
static void init_clfunctions(void) {
        ...
        pclBuildProgram = cl_symbol("clBuildProgram");
        ...
}
...
#define clBuildProgram (*pclBuildProgram)
...

And I just include this at the head of the file and call init during the JNI OnLoad. Saved most of the work since none of the rest of the code needs to change unless I want to add pointer null checks. This opens the possibility of a "sdk-free" build of zcl since without the need to link it can get everything it needs from the header files which I can bundle.

I also looked into a mechanism for implementing CL extensions - I think i've worked out a decent solution (~interfaces on the platform) but haven't (fully) implemented any yet. I'm having second thoughts on whether the array interfaces for asynchronous calls for buffers are really adding enough to counter the work required to implement them and the bloat of the api size, but for now they will remain.

It's not really so "simple binding for java" anymore at ~10KLOC, but then again most of the parts are simple on their own.

After I remembered how to get going in OpenCL I did a little bit of testing on some of the new interfaces in opencl 2.0 - so far they seem to function; although for some reason my catalyst driver is only reporting the CPU device so it's not the best test - i think i need to reboot the machine because of memory fragmentation. Netbeans has turned into (more of) a pig of late and a few hundred tabs in firefox tends to swallow (all) memory. I guess I made a mistake on the memory size in this machine.

zcl 0.4

Another thing over the weekend I did was update zcl to cover OpenCL 2.0 and drop out a 0.4 release. It's completely untested at this point.

See the homepage

Once I do some testing and perhaps revisit the libCL linking mechanism I guess I'll do a 1.0, but given 0.3 was a year ago, ... yeah - don't hold your breath :)

metaprogramming make

I spent the last few days mucking about with writing some in-depth cookbook articles on using make to build java projects but only just came across this cool series of articles about metaprogramming make today. I haven't read it all yet but i wish i'd known about using $(info ) earlier.

As a quick preview the following is a roughly minimal example of the sort of GNU Makefile i've got working:

JAVA_PROGRAMS = thing

thing_JAVA_DIRS=src
thing_JAVAC_LIBS=lib/library-0.jar     # compile-time
thing_JAR_LIBS=lib/library-0.jar       # run-time & added to manifest
thing_MANIFEST=thing-manifest          # sets entry point or other

DIST_VERSION=-0.0
DIST_NAME=thing
DIST_EXTRA=Makefile java.make COPYING

include java.make

This provides jar, sources, javadoc, package (binary with bundled libs, source + doc jars), and dist targets automatically. All build stages go into 'build' and all final outputs go into 'bin'.

java.make is only about 100 lines of make code, mostly a template for all the targets.

I've also started on a junit.make which does some basic junit stuff but I don't really know much about using that and running it outside of some 'mega-tool' doesn't seem very common.

ahh shit, google code is being scrapped

Joy. I guess it's not making google enough money.

I'll make a copy of all my projects and then delete them. Probably sooner rather than later (probably immediately because i'd rather get it out of the way, i have a script going now). I'm keeping all the commit history for now although i never find it particularly useful. Links in old posts may break.

Some of them may or may not appear later somewhere else, or they may just sit on my hdd until i forget about them and delete them by accident or otherwise.

Wherever they end end up though it wont be in one of the other commercial services because i'm not interested in doing this again.

I need to do the same to this blog, ... but that can wait.

Update: So ... it seems someone did notice. Don't worry i've got a full backup of everything. In part I was pissed off with google and in part I wanted to make it immediately obvious it was going away to see if anyone actually noticed. Who could know from the years of feedback i've never received.

If you are interested in particular projects then add a comment about them anywhere on this blog. All comments are moderated so they wont appear till i let them through but there is no need to post more than once (i'll delete any nonsense or spam unless it makes you look like a bit of a wanker). I will see about enabling anonymous comments for the moment if they are not already on. I can then decide what to do depending on the interest.

I will not be using github or sourceforge nor the like.

In a follow-up post i'll see where i'm going with them as well as pondering the future location and shape of the blog itself. I've already written some stuff to take the blogger atom stuff, strip out the posts, download images, and fix the urls.

(in resp to Peter below) I knew the writing was on the wall when google code removed binary hosting: it's kinda useless for it's purpose without it. This is why all my new projects subsequent to that date are hosted differently.

glNext/Vulkan/SPIR/stuff

Decided it was time i had a look into 'the state of things' wrt these new apis. I've not been keeping up with OpenCL of late either so I guess I should refresh on that (oh that's right, c++ kernels, well fuck that shit).

Vulkan definitely looks interesting. I mean vulkan looks like a huge amount of boilerplate from what little is available so far (the GDC 2015 slides from khronos) but it all makes plenty of sense. Pity it's still work in progress so all I can do is read about snippets. One hopes JavaFX will use it as a backend in the future though, and maybe i'll do a ZVK to compliment ZCL myself (it's good way to learn about an api).

There's a bizarre thread on the khronos forums where one poster wants to scare away beginner programmers from Vulkan least they get confused by such wild concepts as memory allocation and asynchronous synchronisation. That's ... well that's just wrong. All that information hiding in OpenGL is what makes it such a pain to understand beyond trivial cases, particularly since fixed pipeline went away. If people are using C (as they must to use it) then malloc is hardly an insurmountable barrier. And asynchronous sync is the only technique which makes multi-threading code manageable and really should be the main technique for MT code taught in any software engineering course. To be honest I think it's all that learning about critical sections and mutexes which do all the damage and make multi-threading seem so complicated and they should be something unlearnt as quickly as possible (as a general MT sync technique that is, obviously they are needed if only to implement more useful primitives like queues and barriers).

Anyway we know how it will play out for beginners and learning professionals alike: we'll all just use google to find the first example of the boilerplate that compiles and then keep using that for years. If I was khronos I'd make sure it's their up to date examples and documentation that using google finds and not some shitty agglomeration site full of out-dated shit.

SPIR-V

SPIR looks a little "odd" to me at first glance, but i'm not a compiler writer so it probably should. To my mind i would describe it (a bit) like assembly language that has been pre-processed and annotated ready for an optimising compiler and then it's internal object representation flattened into a flat global space referenced by index. Rather than registers or addresses results are referenced by the instruction that produces them. It should make a frontend compiler fairly simple as the compiler doesn't need to deal with registers or stacks, and it makes a backend a bit simpler as some of the work is already done. And apparently it's not a file format, although it sure looks like one and i can't imagine it would make sense to anyone to create a different one to store the same information.

I wonder what this mean for HSA (or infact, any other compiler system using it's own virtual processor)? I've not looked into it so maybe something is already announced but if not i suspect SPIR is in it's future too. BRIG takes quite a different approach in that a specific register-based virtual machine is targeted and it's up to the frontend compiler to perform register optimisation and the backend isn't (supposed to be) much more than a machine code translator (which is still strictly speaking, a compiler). I just can't see hardware vendors keeping both maintained.

Oh my gods, it's full of compilers ...

One realisation I had when playing with the soft-GPU code on the parallella was that modern GPU drivers are mostly just compilers with a bit of memory and queue management bolted on the side. And similarly that modern GPUs are really just moderately wide vector CPUs with a little bit logic for fragment generation and texture lookup. of HSA itself is more about compiler writers than the end-user too.

It's just a bit of a personal bummer because although compilers are a bit interesting they're not enough interesting to me to spend the years required to get up to speed.

dun dit dun

Following on from the last post I did some i/o sutff - went with an s-expression based thing since it's a trivial parser and easy to use - then I rewrote the whole model and some of the ui again. Added zoom for instance. Then I played with it for a while but couldn't manage to create the sort of sounds I was trying to create. I couldn't find modules that did what I wanted to do and trying to construct them from primitives got messy and didn't work either.

I was starting to lose interest.

So then I wrote a whole new synthesiser from scratch followed by enough of a midi sequencer to get a midi of Lazy Jones working - and I think it sounded better than the bundled Java midi synthesiser (more hard/harsh just like the original). It uses a thread to render a frame of samples at a time based on a master clock and uses a simple 'runLater()' style mechanism to synchronise parameter updates on the various components so threading issues are just hidden.

I tried a few other files and none worked either at all or very well but after another day of hacking I've got fully polyphonic voices and adsr envelopes and so on and so forth. Most midi files apart from Lazy Jones still sound kinda shit because I don't have any real instruments or the midi processing units (and well, it's midi), but they do play with the right timing and tone (I think, hard to know since midi files usually sound a bit out to me). For efficiency I created a multi channel oscillator and envelope bank which works pretty well but limits the instruments to simple waveform + ADSR envelope.

Most of it was pretty straightforward considering i just did it off the top of my head but I found I had to do some filtering on the amplitude changes to avoid pops when using a sooth waveform. One problem I have is that square and sawtooth are just so much louder than triangle; some filtering seems to help but I guess that's just the way it is and it's up to the musician to compensate.

Probably wont get much farther than that but it was a fun few days. It can be made an interesting real-time problem and parts are paralellisable so if I was keen i'd probably try some epiphany code but i don't think i'm that keen and it's fairly light cpu wise anyway.

I guess I forgot about the space invaders sfx. If the latency is acceptable i might try just using my synthesiser as a real-time sound mechanism, maybe maybe.

Interactive synthesiser

Rather than 'play some more' i went head-first into writing a better display/model/control thing for the synthesiser stuff.

Doesn't look much different on the surface since the stylesheet is much the same but pretty much everything is backed by controls and skins and observable properties.

I have a 'Workbench' control which is the main controller and it handles the human wiring process and so on. I abstracted the connection points as 'Sockets' in a way which allows me to add other non-JSyn objects such as the dials which just feed out their value to some input port. Not shown but you can also reconfigure input ports to change their name and bounds so that they connect properly to dials.

I added the ability to unplug and move the ends of a given wire which works quite nicely. If you grab a wire near one end and 'pull' it far enough it detaches and becomes a free end, which can either be attached to some other compatible socket or letting go just deletes that wire.

I/O next I guess? Always the pain ...

FWIW The circuit above makes a pretty basic "idling chopper" "fop-fop" sound.

Connecting the dots (with wires)

Yesterday I hacked up a "workbench" gui for JSyn. I just started experimenting with auto-generating the gui based on the UnitGenerator interfaces, it's designed with this in mind so it is fairly simple.

Blocks are added by clicking on the workspace, wires are connected by dragging a line between the discs, blocks can be arbitrarily dragged around, wires or blocks are deleted with delete or backspace keys; typical basic graphical user interface stuff.

One of the more difficult problems was how to wire up the ports ...

The difficulty arises because one uses container classes to form grouped gui components and do the layout, and then you want to be able to bind a linkage between an inner component of the group and components working in a different local coordinate system.

A diagram of the layout hierarchy as it currently exists helps to explain (this is just a prototype so the layout is a bit shit).

     Ellipse    Label
        +----+----+
            HBox
             +
          EndPoint (Pane)
       - ----+---- -
            VBox
       - ----+---- -
         Generator (BorderPane)    Wire (Line)
       - ----+-----------------------+
           Pane

So the Line representing the wire needs to connect to the Ellipse level, but the Line is in the coordinate space of the lower Pane and the Ellipse is in the coordinate space of the HBox which is in the coordinate space of the EndPoint, which is ...

So I created a new set of properties socketX, socketY, which reside on each EndPoint which maintain the location of the centre of the Ellipse relative to the lower Pane. The calculation is straightforward and based on transforming to and from scene-relative coordinates. It gets updated whenever the EndPoint is moved (relative to the Generator, i.e. after layout), or when the Generator is moved (i.e. relative to the Pane). The Wire can then just bind to these locations using javafx properties. I've got each generator maintaining the endpoint updates relative to it's parent but i'll probably have to come up with something better.

There's still a problem to solve though; the user interface allows you to plug a wire in to one socket and then drag it to another socket (i.e. click and drag). Unfortunately javafx currently has no way of picking a widget - i.e. looking it up by location or by the mechanism the event system uses. At first I tried using drag and drop mechanisms but it interfered with other gui operations and I couldn't find a way to show the wire following the mouse pointer.

So unfortunately I have to maintain a separate list of all the EndPoints in a model which I then scan for a match during the drag gesture. But given that this allows me to use layouts and stylesheets and so on, this is acceptable.

Duty Cycle

I created a new square wave oscillator type for JSyn as i was surprised the one it comes with doesn't have a duty cycle (pulse with) parameter. It was one of the more useful features of the square wave oscillator on the SID chip and features heavily in C=64 music. I think it could be created using a sawtooth generator and a gate but this is easier to use and was simple to make.

The circuit above generates a rather grating undulating / phasing sound effect like a 'robot thinking', or with a lower frequency a (metallic) misfiring engine.

I guess i'll add more of the unitgenerators from JSyn and do some experimentation with the code I have. If it's interesting enough i will probably dig deeper into making it a real application; it will need a better system model and probably a way to create a control panel separate from the inner workings of a circuit, as well as oscilloscope and spectrum analyser blocks (JSyn has most of this but it's all Swing).