hot for xmas

Yay i'm on leave at last.

But what am i to do,

with all this time to pass.

It's been really hot this week - from Monday I think it was 39, 40, 41, 42, and the last forecast i saw had 43 44 for Saturday before a bit of a cool-down. I've been going to the beach in the afternoon then dropping into the city for a couple before going home (and yesterday i went back to the beach again but that was a bit keen). Today i stepped on something in the murky water (puffer fish? crab?) which cut deep enough into the pad of my foot to sting, so i came straight home and tended to that. Seems ok so far and i drew more blood hitting my head in the bathroom leaning down to dry my feet so hopefully it will clear up and i'm not stuck hobbling for too long - i've had more than enough of that this year. Apart from not being cold enough there was a line of murk and some shit floating on the water so it's not the peak experience so i might've skipped tomorrow regardless. It's pretty much dull as fuck too but it is decent exercise and i'm getting quite fit and shapely.

The pubs have been a bit feral and i'm not really enjoying the beer much so i might give that part a rest for the foreseeable future too. Drinking alone really is pretty pathetic but it's somehow ended up at a stage where it's either that or stay home alone apart from some occasional visits to friends (well, one friend, who's still close enough and usually home) or from friends (well, mostly one friend, but not for a while). I rarely hear from family unless they're passing through town although that's probably about the right amount. It just sort of happened.

Going to be a touch cooler next week but its looking high 30s for xmas day which at least is a good start for the day. I probably wont go "home" for it but i'm undecided as to it's final fate. I have done it alone before and that was a really nice and relaxing day but I can imagine it being not so nice if the mood swings the wrong way and this year it's not been swinging up much. Friend offered his parents do if i feel like grandchildren running amok.

Garden's still alive but it's taking a lot of water to keep it so. Its been a pretty dry year with rainfall 150mm+ under the average which is only a paltry 550mm to start with but there might be one more thunderstorm left yet. After a short family visit on the first, the second morning of leave I cut down a tree out the front that was over-shading the roses and was rotting at the base - I was expecting the rest of the tree to be a bit dead but it was only right at the stump but i'm glad i got it out since it didn't seem too solid in the ground. But it's been too hot to do much since. Riding isn't actually too bad so long as you're moving and you slow down as soon as you feel over-exertion but i think it's wise to give the hills a miss for the moment unless i feel like an early ride and have the urge to feel sore the next day (yeah i dunno, i'm a bit over that).

I need more winding-down before thinking about hacking so perhaps i should finally crack open some games but eh, maybe. I could use some good sleep-in's too but i keep getting up early instead, at least the heat has slowed down the builders across the road some so i might have a chance if i try. I sort of did this morning but being awake from 3am to 7am beforehand rather negated any possible benefit.

Life rolls by.

Well that sank a lengthy day.

Who would have thought a pair of boardies would be so complicated.

Picking apart seams will be in my dreams.

Clothes are expensive.

I overdid this week a bit and had to get out of the house today. It was also hot for the first time in a couple of weeks so I had to get out into it. I had plans to buy a few things but they partially fell through and all I did was end up pissed off.

I did find some (pretty crappy) sort-of-actual-shorts (as opposed to no-they're-actually-longs, you know, they actually end above the fucking knee) because the typical length shorts of the day are too hot, too uncomfortable, and rub the top of my legs in ways that cause soreness. Now I don't have such a fat-arse at least it's possible to have them fit the leg when they fit the waist. One of the pair I got is basically white - they look like 60s sports day shorts, I just need the white singlet and dunlops and the no 1 haircut to go with an old family portrait, but they're just for for around the house. Either the sizes are a little "generous" or i'm down to a small now and that size definitely wasn't tight.

Actually I haven't really stopped losing weight despite eating pretty normally and doing enough basic exercises to gain muscle. This week I measured a minimum "under 69.0" for the first time in decades (but that was just the extreme of the trough). I somewhat expected it to level off by now.

I went looking for a battery for my bike headlight - the charger for the chinese junk I bought broke but the light is fine and i've been charging the mystery battery with a PSU I made in uni - but i got really pissed off at the dickhead at jaycar who couldn't do more than point at the shelves I'd already looked through so i gave up and walked out. If i just want to get their cheap junk i can get it myself, i only went to the shop to see if they had knew anything I didn't and to support a local business. dx.com it is I guess. Last time I tried the battery bar in the city but the shop assistant was utterly clueless and the 'tech guy' was a condescending fuckwit and it seems more like a "junk bar" than a battery bar anyway.

There's two local shops I will NEVER return to ever again. I made the effort to go to a local shop physically and with a literal pocket full of cash, and they weren't interested in even attempting to close any sort of sale. Fortunately at the expensive wool-clothes shop I went to after the battery bar I had a long chat to the bored shop assistant and she tried to upsell everything 'male' they sold (which isn't much but i hadn't intended to get a jumper when i went in).

I bought some dried fruit from The Market and tried to find a silicone gasket for a (stove-top) coffee maker for one of my brothers - but nobody sells them. At least one guy said it was because they never wear out - which is the whole bloody point! I got one for my machine at Gaganis Bros but they're still in the bad books for their bicycle parking arrangements they didn't have many back then anyway.

At this point i was ready to fall apart or have an accident (i've just been in a foul mood all week and this made it worse) so I said fuck it and had a beer at the Exeter. I quite like the pub itself and the staff are decent but this isn't the first time i've looked around and realised I was surrounded by other middle-aged men so I decided if I have any more I should try somewhere else. I decided at this point to go buy some bike shit after seeing it last time - I grabbed some new shy shorts (30 was the right fit, so yeah I guess I am small now) - the ones i've been using lately are 36 and they were too tight last year but now they feel like nappies with their built-in chamois and the draw-string. Also too long and too hot. Ugh I spent like 170 on another pair of cargoish things at the same time back then which I had been using but they feel like clown pants now, they were even bigger (and both have faded poorly, why are they all black anyway??). And a couple of jersey's which are possibly a bit on the tight side but we'll see - most they had were black and that's too hot so I got a white one and a red one to match my roadie (hey why not). One specific jersey I used most recently still has serviceable cloth but got stretched too much to be passable so at some point I will see if i can take it in somehow. Those expensive bike faded shorts are too big for that but I might also try to take in some other stuff I've never even worn if i can get my sewing machine to cooperate (stuff i was too fat to fit, and now falls off).

So that bike shit (incl, shoes I bought last week, new pedals and cleats), the wool tshirts and jumper, the "POW" PE shorts, jerseys and shorts, a new tyre, well over a grand. It is what it is.

I ended the afternoon with a few beers at the Austral - fuckit, if i get gout so be it (but so far so good?). I had 4 total for the day but I think 2 was probably what I should've stopped as after those two I just wasn't' really feeling anything good and all it left me with was a headache and a churned guts which still persists a couple of hours later.

I contemplated life and it came up very very short. As ever.

I even thought about tattoos. Why? To belong to "something" I guess? I didn't think about them too much.

dumb lockless 'queue' idea

I was playing with linux signals the need arose for a lockless queue, although given the mess of poll vs signal and so on i might just use a pipe for signalling as i've done before. But such a queue is useful for that case as well.

Unless you're dealing with exotic hardware or performance I imagine that a basic bounded lockless queue is fairly easy to implement using a two-index compare and swap but I was thinking about the unbounded case. Not that unbounded is necessarily a good idea in many cases for other reasons.

After a couple of mistakes I think I arrived at a very simple solution which should work. It's actually half a stack, and half a something else I don't even know the name of, but it can be made to appear as a queue cheaply.

enqueue

This is implemented as a push onto the head of a single-linked list. This can be implemented very simply using a compare and swap against the single pointer of the stack head. The next pointer of the node can be freely updated until it is owned by the queue once the compare and swap succeeds.

void enqueue(queue q, msg m) {
  queue old, new;

  do {
    old = q;
    new.head = m;
    m.next = old.head;
  } while (!compare_and_swap(q, old, new));
}

dequeue-all-reverse-order

This is no dequeue operation, instead there is a dequeue-all which removes every element currently on the queue but returns them in reverse order. It just clears the stack head and returns it's old value. This can be implemented trivially using an atomic exchange operation with NULL.

queue dequeue_all(queue q) {
  q = exchange(q, null);
  return q ? q.head : null;
}

dequeue

This has to be performed as a separate step but it is performed local to the reader. It should be efficient enough on typical processors. In the (very) unlikely event that order isn't important then it isn't even necessary.

void dequeue_foreach_rec(msg stack, func cb) {
   if (q) {
     dequeue_foreach_rec(stack.next, cb);
     cb.apply(stack);
   }
}

void dequeue_foreach(queue a, func cb) {
  dequeue_foreach_rec(dequeue_all(q), cb);
}

It supports multiple writers. Whilst it still functions "correctly" for multi-reader queues it probably isn't terribly useful for that case as it's first-come-first-gets-all with no attempt or possibility of balanced delivery.

more termz

I tried a few more variants on the OpenCL rendering code but none are all that fast - the overheads kill it and whilst it does free the CPU up a little bit it isn't much. Probably not worth more time unless I look at OpenGL instead.

I added resizing. It meant I had to add some locking to the snapshot routine but it only needs to lock around the resize operation so it adds almost no overhead to normal operation (merely detecting a resize has occurred). I'm not yet sure what's supposed to happen with saved cursors and alternate screens when a resize occurs, probably just clip to size. Unfortunately there seems to be no way to set the WM_NORMAL_HINTS on the javafx stage, so there's no way to make it size to cells properly.

I did a bunch of benchmarking and profiling. One thing I tried was another test to compare to xterm - now at full-screen. Running "find . -name '*.c' | xargs cat" from the root of the linux 3.19.8 source tree. After a couple of runs this is about 25 seconds on termz, and 16 minutes in xterm. Well. Yeah it's a silly test but all those ls -l's during the day add up.

Looking at the memory profiler it doesn't really use too much heap during operation, just a few MB and most of that is the image and javfx. I mean nor should it, there isn't data structures to maintain. But having multiple compilers in ram (jvm, OpenCL), their generated output, and all the added overhead of the runtime support needed for those really adds up so it's very very fat in practice. I guess if multiple terms ran on the same jvm it would be ok.

enough yet?

So at this point i'm not really sure what i'll do with it. I'll probably poke at the edges when i'm bored and eventually when I get around to it I will dump what I have to my software site as the result of a weekend-and-a-bit-hack.

After that I'm not sure. It's actually quite functional and robust already (well, compared to the effort in) and wouldn't take much more work to turn it into a usable terminal for me - add some scrollback (pretty simple), mouse selection stuff (not that hard), and sort out some of the keyboard details (reading obtuse documents and testing). So maybe that will happen.

If I got that far adding the "10x20" typeface would probably be on the cards. Fixed-size outline fonts would be possible by just pre-rendering them but to me they just aren't terminal fonts.

Anything further such as a re-usable term component (which might actually be of use to the world) would require substantially more work on the i18n side of things and I don't feel like learning enough to do that properly.

Make

I did a bit poking at the java makefile stuff and it's to the point where i'm using it for out-of-ide builds of termz and zcl and will look into using it on other projects. That's the best way to find bugs/what works and what doesn't and previous attempts never got that far. For all it's gnumakefile obtuseness it's really rather compact at under 200 lines excluding comments and I didn't put any effort into making it particularly small. And that includes targets for javadocs, source jars, and binary builds.

GNU make and java

Today I had another go at looking at using meta-rules with gnu make to create a 'automatic makefile' system for building java applications. Again I just "had a look" and now the day is all gone.

This is the current input for termz.

java_PROGRAMS=termz

termz_VERSION=0.0
termz_DISTADD=Makefile java.make \
 jni/Makefile \
 jni/termz-jni.c

# compiling
termz_SOURCES_DIRS=src
termz_RESOURCES_DIRS=src
termz_LIBADD=../zcl/dist/zcl.jar

# packaging (runtime)
termz_RT_MAIN=au.notzed.termz.TermZ
termz_RT_LIBADD=../zcl/dist/zcl.jar

# native targets
termz_PLATFORMS=gnu-amd64
# native libs, internal or external.  path:libname
termz_RT_JNIADD=../zcl/jni/bin:zcl jni/bin:termz

# Manually hook in the jni build
jni/bin/gnu-amd64/libtermz.so: build/termz_built
 make -C jni TARGET=gnu-amd64

clean::
 make -C jni clean

include java.make

make (jar)

This builds the class files in build/termz, then the jni libraries via the manual hook. In another staging area (build/termz_dist) it merges the classes and the resource files (stripping the leading paths properly). It uses javapackager to create an executable jar file from this staged tree which includes references to the RT_LIBADD libs (moved to lib/). Finally it copies the RT_LIBADD jar files into bin/lib/ and the native libraries into bin/lib/{platform} so that the jar can be executed directly - only java.library.path must be set.

Not shown but an alternative target type is to use java_JARS instead of java_PROGRAMS. In this case jar is used to package up the class and resource files and in this case no staging is needed.

make tar

This tars up all the sources, resources, and DISTADD into a tar file. No staging is required and all files are taken 'in-place'. The tar file extracts to "termz-0.0/".

make clean

Blows away build and bin and (indirectly) jni/bin. All 'noise' is stored in these directories.

I'm using metaprogramming so that each base makefile can define multiple targets of different types. This is all using straight gnu make - there is no preprocessing or other tools required.

java.make does all the "magic" of course. While it's fairly straightforward ... it can also be a little obtuse and hairy at times. But the biggest difficulty is just deciding what to implement and the conventions to use for each of them. Even the variable names themselves.

There were a couple of messier problems that needed solving although i'd solved the former one last time I looked at this some time ago.

class-relative names

For makefile operation the filenames need to be specified absolutely or relative to the Makefile itself. But for other operations such as building source jars one needs the class-relative name. The easiest approach is just to hard-code this to "src/" but I decided I wanted to add more flexibility than this allows.

The way I solved this problem was to have a separate variable which defines the possible roots of any sources or resources. Depending on what sort of representation I need I can then match and manipulate on these roots to form the various outputs. The only names specified by the user are the filenames themselves.

For example when forming a jar file in-place I need to be able to convert a resource name such as "src/au/notzed/terms/fonts/misc-fixed-semicondensed-6x13.png" into the sequence for calling jar as "-C" "src" au/notzed/terms/fonts/..." so that it appears in the correct location in the jar file.

I use this macro:

# Call with $1=root list, $2=file list
define JAR_deroot=
    $$(foreach root,$1,\
 $$(patsubst $$(root)/%,-C $$(root) %,\
  $$(filter $$(root)/%,$2))) \
   $$(filter-out $$(addsuffix /%,$1),$2)
endef

I'll list the relevant bits of the template which lead up to this being used.

# Default if not set
$1_RESOURCES_ROOTS ?= $$($1_RESOURCES_DIRS)

# Searches for any files which aren't .java
$1_RESOURCES_SCAN := $$(if $$($1_RESOURCES_DIRS),$$(shell find $$($1_RESOURCES_DIRS) \
  -type d -name CVS -o -name '.*' -prune \
   -o -type f -a \! -name '*java' -a \! -name '*~' -a \! -name '.*' -print))

# Merge with any supplied explicitly
$1_RES:=$$($1_RESOURCES_SCAN) $$($1_RESOURCES)

# Build the jar
$$($1_JAR): $(stage)/$1_built
        ...
        jar cf ... \
          $(call JAR_deroot,$$($1_RESOURCES_ROOTS),$$($1_RES)) \
        ...

At this point I don't care about portability with the use of things like find. Perhaps I will look into guilified make in the future (i'm a firm believer in using make as the portability layer as the current maintainer is).

And some example output. The last 2 lines are the result of this JAR_deroot macro.

jar cf bin/termz-0.0.jar   -C build/termz . \
 -C src au/notzed/termz/fonts/misc-fixed-6x13-iso-8859-1.png \
 -C src au/notzed/termz/cl/render-terminal.cl

A lot of this stuff is there to make using it easier. For example if you just want to find all the non-java files in a given directory root and that contains the package names already you can just specify name_RESOURCES_DIRS and that's it. But you could also list each file individually (there are good reasons to do this for "real" projects), put the resources in other locations or scatter them about and it all "just works".

How?

Just looking at the JAR_deroot macro ... what is it doing? It's more or less doing the following pseudo-java, but in an implicit/functional/macro sort of way and using make's functions. It's not my favourite type of programming it has to be said, so i'm sure experts may scoff.

  StringBuilder sb = new StringBuilder();
  // convert to relative paths
  for (String root: resourcerootslist) {
    for (String path: resourcelist) {
      if (path.startsWith(root+"/")) {
        String relative = path.replace("^" + root + "/", "");

        sb.append("-C").append(root)
          .append(relative);
      }
    }
  }

  // include any with no specified roots
resource:
  for (String path: resourcelist) {
    for (String root: resourcerootslist) {
      if (path.startsWith(root+"/")) {
         continue resource;
      }
    }
    // path is already relative to pwd
    sb.append(path);
  }

  return sb.toString();

This is obviously just a literal translation for illustrative purposes. Although one might notice the brevity of the make solution despite the apparent verbosity of each part.

Phew.

going native

Native libraries posed a similar but slightly different problem. I still need to know the physical file location but I also need to know the architecture - so I can properly form a multi-architecture runtime tree. I ummed and aahd over where to create some basic system for storing native libraries inside jar files and resolve them at runtime but i decided that it just isn't a good idea for a lot of significant reasons so instead I will always store the libraries on disk and let loadLibrary() resolve the names via java.library.path. As it is still convenient to support multi-architecture installs or at least distribution and testing I decided on a simple naming scheme that places the architecture under lib/ and places any architecture specific files there. This then only requires a simple java.library.path setup and prevents name clashes.

Ok, so the problem is then how to define both the architecture set and the library set in a way that make can synthesise all the file-names, extensions (dll, vs so), relative paths, manifest entries in a practical yet relatively flexible manner?

Lookup tables of course ... and some really messy and hard to read macro use. Oh well can't have everything.

Libraries are specified by a pair of values, the location of the directory containing the architecture name(s), and the base name of the library - in terms of System.loadLibrary(). These are encoded in the strings by joining them with a colon so that they can be specified in a single variable. The final piece is the list of platforms supported by the build, and each library must be present for all platforms - which is probably an unnecessary and inconvenient restriction in hindsight.

This is the bit of code which converts the list of libraries + platform names into platform-specific names in the correct locations. I'm not going to bother to explain this one in detail. It's pretty simple just yuck to read.

#
# lookup tables for platform native extensions
#
# - is remapped to _, so usage is:
#
#  $(_$(subst -,_,$(platform))_prefix) = library prefix
#  $(_$(subst -,_,$(platform))_suffix) = library suffix
#
_gnu_amd64_prefix=lib
_gnu_amd64_suffix=.so
_gnu_amd32_prefix=lib
_gnu_amd32_suffix=.so
_mingw32_amd64_prefix=
_mingw32_amd64_suffix=.dll
_mingw32_amd32_prefix=
_mingw32_amd32_suffix=.dll

# Actual jni libraries for dependencies
$1_JAR_JNI=$$(foreach p,$$($1_PLATFORMS), \
  $$(foreach l,$$($1_RT_JNIADD), \
   $$(firstword $$(subst :, ,$$l))/$$(p)/$$(_$$(subst -,_,$$p)_prefix)$$(lastword $$(subst :, ,$$l))$$(_$$(subst -,_,$$p)_suffix)))

Thinking about it now as i'm typing it in a simpler solution is possibly in order even if might means slightly more typing in the calling Makefile. But such is the way of the metamake neophyte and why it takes so long to get anywhere. This is already the 2nd approach I tried, you can get lost in this stuff all too easily. I was thinking I would need some of this extra information to automatically invoke the jni makefile as required but I probably don't or can synthesise it from path-names if they are restricted in a similar fashion and I can just get away with listing the physical library paths themselves.

Simple but restricted and messy to implement:

termz_PLATFORMS=gnu-amd64
termz_RT_JNIADD=../zcl/jni/bin:zcl jni/bin:termz

vs more typing, more flexibility, more consistency with other file paths, and a simpler implementation:

termz_RT_JNIADD=../zcl/jni/bin/gnu-amd64/libzcl.so \
  ../zcl/jni/bin/mingw32-amd64/zcl.dll \
  jni/bin/gnu-amd64/libtermz.so

Ahh, what's better? Does it matter? But nothing matters. Nothing matters.

Ok the second one is objectively better here isn't it?

After another look I came up with this to extract the platform directory name component:

$(lastword $(subst /, ,$(dir $(path))))

make'n it work

One other fairly large drawback of programming make this way is the abysmal error reporting. If you're lucky you get a reference to the line which expands the macro. So it's a lot of hit and miss debugging but that's something i've been doing since my commodore days as a kid and how I usually work if i can get away with it (i.e. building must be fast, i.e. why I find it so important in the first place).

And if you think all of that looks pretty shit try looking at any of the dumb tools created in the java world in the last 20 years. Jesus.

damn work

It seems i hadn't had enough of the terminal after the post last night - i was up till 3am poking at it - basically another whole full-time day. I created a custom terminfo/termcap - basically just started with xterm and deleted shit I didn't think I cared for (like anything mouse or old bandwidth savers that looked like too much effort). But I looked up each obtuse entry and did some basic testing to make sure each function I left in worked as well as I could tell. Despite the documentation again there are a lot of details missing and I had to repeatedly cross checked with xterm behaviour. Things like the way limiting the scroll region works. And I just had a lot of bugs anyway from shoddy maths I needed to fix. I then went to look at some changes and netbeans had been too fat so I went down the rabbit-hole of writing meta makfiles so I could do some small tests in emacs ... and never got that far.

I really needed a proper break from work-like-activities this weekend too, and a lot more sleep. At least I did water the garden, mow the lawn, wash my undies, and run the dishwasher.

About Me

Tags

big whoop, another year down

making dough