About Me
Michael Zucchi
B.E. (Comp. Sys. Eng.)
also known as Zed
to his mates & enemies!
< notzed at gmail >
< fosstodon.org/@notzed >
On liver n stuff n shit.
So one of those things you never really liked as a kid. Once in a while I think 'man i could go some liver', even mum's cooked-to-hell-and-back version that just turns each sliver into a piece of copper-tainted rubber. I have some lamb's fry in the fridge so that might be (part of) dinner tonight. It's usually kind of 'yeah did that, not sure why' but you know, it has to be done sometimes.
Made some lime cordial today, had a couple of g&t's while i was bringing it to the boil and continued thereafter. Went with the straight ABC recipe (hmm, seems the original page is gone) this time even with it's 1.5Kg of sugar although I added a lot more citric acid - love me a good bit of tart and it's better to be safe than sorry on the preservation side of things. The last few batches didn't have enough punch and i've been thinking it might be due to a lack of sugar so this is part of the experiment to confirm that hypothesis. I had to buy the limes this time :( but they were cheap because they were old, and they've been in the fridge for weeks - but they still have a nice sweet ripe taste so i'm hopeful of a good batch. I usually make my own 'soda' with it: 500ml glass of really cold water + ice, some cordial 'to taste', 3/4 teaspoon of citric acid and 3/4 teaspoon of bi-carb, stir till it fizzes then slam it down; a really fast (and tasty) way to re-hydrate after an afternoon in the sun. Doesn't work if the water isn't super-cold - it just fizzes over and goes flat.
Trying not to think of the week ahead. The 200km of cycling I did last week at least shaved a couple of kilo's off my spare tyre and turned my legs into iron but i'm still recovering after 3 days rest - wildly out of practice and alas not 20 any-more. At least it's a short week coming up with Easter next weekend. The biltong I made last week is just about ready - taking a while to dry in this weather and I didn't find the 40W incandescent 'heater' till a couple of days ago (normal globes around here don't produce enough heat anymore, this is a narrow candle flame shaped one which still has a tungsten filament). Forgot to dip it in vinegar and the spice mix might have a touch too much fennel but it's still pretty tasty, i'll jot down the recipe on here in another post in the not too distant future. Made it straight from some corned silverside which was on special, so it both cheap and easy to make.
End of intermission ... back to the g&t's ... and maybe some liver if i can be bothered ...
the eze-thing
After the last week I felt the need for a bit of coding ... i worked a bit on the eze-library, deciding on a directory structure and using it, filling out some missing bits and working on the build system. I learnt a few make and gcc things along the way - which is always nice.
I included and/or implemented the various bits and pieces of mentioned on the last few parallella related posts - things like the global loader-defined barrier memory, async dma (via a queue and interrupts), and the startup routine that can pass arguments to kernels, track the current running state and set a return code.
I decided to just allocate the barrier memory for the whole workgroup before the code address every time even if barriers aren't being used. It can always be changed. Still yet to test the implementation.
The start of the memory-map (excluding the isv entries) now looks like this:
+----------+------
| 0028 | extmem
| 002c | group_id
| 0030 | group_rows
| 0034 | group_cols
| 0038 | core_row
| 003c | core_col
| 0040 | group_size
| 0044 | core_index
+----------+------
| 0048 | imask (short, but here so it can be loaded as int)
| 004a | status (short)
| 004c | exit code
| 0050 | entry
| 0054 | sp
| 0058 | arg0
| 005c | arg1
| 0060 | arg2
| 0064 | arg3
+----------+------
| 0068 | barrier, group_size bytes
| |
+----------+------
| ≥006c | .text .data .bss
| | .text.bank0 .data.bank0 .bss.bank0
Not sure if it'll work but i experimented with a @workgroup "tag" on section names. If present the allocation is multiplied by the workgroup size - this was whilst working on the barrier stuff before I realised that wont work because the barrier location has to be the same across all work-items in the work-group even if they're running separately linked code. Something I can play with later anyway.
After getting the most basic test running i'm to the point of being able to debug the new features. And I just got the async dma interfaces to function (yay?) before writing this up. Actually it works out pretty nicely. I define the async dma queue inside the isr handler code so that by using the c functions which reference the queue it drags in the isr and isr vector automatically, which the loader tracks so that the new sync isr automagically sets the correct imask too.
Once i've got everything going i've got a bit of housekeeping stuff to deal with before it can go further. But for now I've settled on two libraries:
-
libezehost.so
-
The host-side library (surprise). This includes a fork of the adapteva esdk 'e-hal' as well as the elf-loader stuff. The on-core runtime interface has been changed to accommodate the new features so it wont work with pre-linked binaries.
-
libezecore.a
-
This is the on-core support library and equivalent to e-lib. Most of the functions are inline calls which generates smaller code-size and more efficient compilation of leaf functions.
I have assembly versions of almost every non-inline routine too; they save some code-space but maybe not enough to be worth it. I might include it as another library option. Perhaps.
I'm probably also going to look at different runtime mechanisms such as a "job queue" mode rather than the current "one shot" mode. This will be changed by specifying a different crt0.o file. Already the crt0 implementation I have allows one to restart the core by just using e_start() without requiring a reset first because the exit routine just idles rather than trapping.
developer tools and tasks vs non-developer tools and tasks
So I started on a project that uses maven for it's build / test and deploy 'script'. Or some part of it; it's complicated.
Like ant (or maybe worse), maven doesn't seem to do dependency checking on targets. All it does is build a dependency graph of the tasks and executes them in an order which satisfies that graph. Or if it fails mid-build you can add some command line arguments to continue from a specific place. Or you can manually cd to a specific part of the system and build that in isolation - but that's one-way ticket to fragility and is error prone even if you've got a really good idea of how the project fits together. So basically you're forced to recompile the whole shooting match every time even if nothing changed in most of the application. This is probably a direct result of both ant and maven's design as a task-based dependency tree rather than a goal based one.
Whilst talking to someone about how useless maven is they mentioned it sounded more like tool more suited to qa or production builds.
Which makes a lot of sense to me.
A developer needs:
- Fast;
- Simple;
- Reliable;
- Repeatable;
- Robust.
But for configuration management for QA builds or production, the operator only needs:
- Reliable;
- Repeatable;
- Robust.
Whilst there is considerable overlap there doesn't seem to be enough to me to justify the overhead in developer time - unless you're paying them peanuts having them wait Dx10xN minutes for D developers to re-build stuff that has already been built every time becomes an incredibly costly exercise.
I think this is similar to my main problem with git. Git was written by a configuration/qa manager (Linus), not a typical developer. His needs are that of a configuration/qa manager and not a typical developer. Apart from having an offensive name, it forces every developer to become their own configuration manager as well and whilst that might be a lofty goal it's taking away time from actually getting work done too. This costs money.
What's remotely agile about Agile?
They also use some sort of 'agile' team management process. I really can't say anything good about that at all.
Apart from some pretty shitty choice of words ('sprint' being the most egregious, but also 'backlog item' for all tasks, not just those beyond due-date; all designed to subtly force people to work harder for the same pay) the whole point of 'agile' seems to be to force all of the team to manage the team together. Which is ... a bit backwards. Firstly there's the problem of specialised skills and experience. It also exposes everyone to the responsibilities that managers "get paid the big bucks" for in compensation.
But ultimately it's just another overhead which interferes with actually getting work done. The point of a manager/team leader is to grease the wheels to let the other developers get work done as efficiently as possible and in a cohesive way. And people are different so more than one approach may be necessary. They usually never get to do any 'real work' themselves but without them (as in agile) nobody else does either. I sat through a 2 hour 'planning' meeting in abject horror as they seemed to mostly just group-operate a bug tracker. Something which could have taken the team leader 15 minutes of his own time (and the bigger items could have been done properly rather than just a hand-wavy guesstimate). I'm sure my disgust was visible because i don't even try to hide it and i was too tired to care.
Anyway I did a little reading about it and it just seems like someone went and interviewed (or imagined) a bunch of 'highly productive' teams about their "process" and wrote it down in a book to make money - creating a whole new language in the process as a way to start a new religion (and more profits).
But they forgot the key element: teams are made of people. And people are ultimately what make the team work regardless of (and often in spite of) any processes in place. But this has probably been forgotten on purpose, books like this are for management who wants to treat individual employees as identical resource units to be reallocated and 'consumed' at will.
I can see agile working ... but only in environments and on projects where it was never needed to start with and in those it would only add unnecessary and considerable overheads. This kind of magic cannot be bottled into self-help books which is all these types of books are and why these things are so faddish. Like self-help books if they actually worked the whole self-help publishing industry would collapse.
The whole two-week lock-step milestone thing is just nonsense too. There's never time to do any big architectural changes should they be required so they are never even suggested. And even little tasks just keep bouncing along across 'sprints' if they need to anyway it just means more wasted group-management time rather than letting them sit in the bug tracker till they're ready to be done.
I think like maven/ant they also forgot that a product which keeps the customer happy is the ultimate goal, not adherence to processes and racking up the count of passing short and arbitrary milestones.
But it's my day off ... but i'm too tired to do anything so I might just try to catch up on sleep and let my legs recover from the cycling. I haven't done this much for years and I was pretty much at my limit getting home yesterday; at least that's something that should improve quickly.
linaro arm gnueabihf cross compilers on Slackware64
I've been trying to get a full cross compilation setup going for the parallella but had way more trouble than I should have getting the cross compiler for arm going. I'm using the linaro sources but can't use the binaries I saw because Slackware64 isn't setup for multilib by default and I didn't want to turn it on. It turned out to be easy but I kept getting wrong arguments from old web pages or got the wrong info from others that weren't doing quite the same thing.
It's very well possible there are binaries elsewhere but I didn't see them; and in any case getting this to work is a lot more useful for me. The linaro build tool 'crosstool-ng' seems to expect debian and/or ubuntu and simply didn't work.
I started from linaro-toolchain-binaries page, and downloaded the full sources sources part 1 and part 2 (although part2 is just the linux kernel source which isn't necessary).
First, some variables/setup which makes it work:
top=`pwd`
prefix=/home/notzed/cross
mkdir -p ${prefix}
export PATH=${PATH}:${prefix}/bin
prefix is the install location.
tar xvjf ~/Downloads/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src-part1.tar.bz2
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/gmp-5.0.2.tar.bz2
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/libiconv-1.14.tar.gz
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/expat-2.1.0.tar.gz
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/gdb-linaro-7.6.1-2013.10.tar.bz2
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/zlib-1.2.5.tar.gz
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/isl-0.11.1.tar.bz2
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/gcc-linaro-4.8-2013.10.tar.xz
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/linaro-prebuilt-sysroot-2013.10.tar.bz2
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/binutils-linaro-2.23.2-2013.10-4.tar.bz2
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/mpfr-3.1.0.tar.xz
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/pkg-config-0.25.tar.gz
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/cloog-0.18.0.tar.gz
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/md5sum
gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/mpc-0.9.tar.gz
Then untar the interesting ones in a way that the configure scripts expect. I use a script for this but this is what it executes:
mkdir -p build
tar xJf gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/gcc-linaro-4.8-2013.10.tar.xz -C build
mv build/gcc-linaro-4.8-2013.10 build/gcc-linaro
tar xjf gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/binutils-linaro-2.23.2-2013.10-4.tar.bz2 -C build
mv build/binutils-linaro-2.23.2-2013.10-4 build/binutils-linaro
tar xjf gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/linaro-prebuilt-sysroot-2013.10.tar.bz2 -C ${prefix}
tar xjf gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/gmp-5.0.2.tar.bz2 -C build
ln -s ../gmp-5.0.2 build/gcc-linaro/gmp
tar xzf gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/mpc-0.9.tar.gz -C build
ln -s ../mpc-0.9 build/gcc-linaro/mpc
tar xJf gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/mpfr-3.1.0.tar.xz -C build
ln -s ../mpfr-3.1.0 build/gcc-linaro/mpfr
tar xzf gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_src/zlib-1.2.5.tar.gz -C build
ln -s ../zlib-1.2.5 build/gcc-linaro/zlib
The
Then the building, just two stages: binutils then gcc. The sysroot from linaro is used and contains libc and so on.
Binutils is straightforward and only takes a minute or so to build.
top=`pwd`
mkdir -p ${top}/build/binutils-build
cd ${top}/build/binutils-build
../binutils-linaro/configure \
--prefix=${prefix} \
--target=arm-linux-gnueabihf \
--disable-nls --disable-werror \
--with-sysroot=${prefix}/linaro-prebuilt-sysroot-2013.10
make -j8
make install
Then gcc. This was the painful one ... but to cut the long story short this is what I did to get it to work. It takes a few minutes to build.
mkdir -p ${top}/build/gcc-build
cd ${top}/build/gcc-build
../gcc-linaro/configure \
--prefix=${prefix} \
--host=x86_64-linux-gnu \
--build=x86_64-linux-gnu \
--target=arm-linux-gnueabihf \
--with-float=hard --with-arch=armv7-a -with-fpu=vfpv3-d16 \
--disable-nls --disable-werror \
--enable-languages=c,c++ \
--with-sysroot=${prefix}/linaro-prebuilt-sysroot-2013.10
make -j8
make install
*shrug* I can't really say whether it's right or wrong but - but it worked well enough to cross compile a working binary.
The ezesdk
So mostly why I did this was so that I could do some re-arranging of the elf-loader code into a more wide-reaching library and keep working on it locally on my workstation rather than having to do it on the parallella. Not that it isn't fast enough to do the little bit of compiling I need - it certainly is - it was just all done on a whim really (one that drew out to a good few hours although not particularly intense ones, in between making another batch of biltong, cooking some corned beef, drinking, and surfing the web).
Another whimsical decision was to fork the epiphany e-hal library. This ... isn't really something I originally intended to do because it just means more to maintain and also there may be future functionality that would be best kept there - such as multi-process arbitration and so forth. But I guess one small but real justification is that right now I need to access some private (-local) data structures which are presumably 'private' for a reason; and if they ever really went private things would break.
Anyway ... I've just been extremely tired the last few days and a bit distracted by some stuff (to somewhat understate it) so it may well just be a bad idea but I suppose i'll see how far I get. It's taken long enough just to decide on a directory layout. At this point i'm going to have libezehost.so which will be the host driver (e-hal + elf-loader, etc), and libezecore.a which will be the on-core api (i.e. e-lib equivalent) although much of it is inline.
I unfortunately have to start working on that other project tomorrow which means a long commute on cyclist hostile roads (more so than average) so tonight better be an early one. Just as well i'm totally knackered and might not even make 10pm - although daylight saving just ended today so it's "really" later than that.
Quick swing by JavaFX
Had a chance to poke around a small JavaFX application for the customer for a couple of days. Unfortunately i've been 'booked' for other work for the next few months so I'm still not getting as much of a chance to play with JavaFX as i'd hoped. Might be the last too (on that s/w) as this new work thrust on me is basically an unrelated graduate-level task in a graduate-level work environment, and that's the least of the issues with it (not too happy right now but i'll have to see how it transpires).
Anyway ... I'm still having issues with layouts although in my particular problem I think it came down to not realising that the scale pivot point is the centre of objects. I had inconsistent/jumping ui's as things changed so it just looked like a layout bug.
I had a few hours left after adding the required functionality so I thought i'd tart it up a bit with clipboard and then drag and drop support. Nice ... it's nice. Just simple and hides all the data-type negotiation and whatnot; something that made sense in 90s era computers without the memory (and also high hopes of extensive functionality) but now there's no real need for it (and dnd is mostly used for stupid things like editing text where it doesn't make sense because of it's clumsiness, and not for file opening where it does). One of those things were a really good style guide might have helped rather than a free-for-all.
As an example this function pointer (or method handle, or whatever it is) can just be added to any ImageView to make it draggable as a picture - e.g. to drop into a picture editor.
static EventHandler<MouseEvent> dragImageView = (MouseEvent event) -> {
ImageView iv = (ImageView) event.getSource();
Image im = iv.getImage();
if (im != null && im.getProgress() == 1) {
Dragboard db = iv.startDragAndDrop(TransferMode.COPY);
ClipboardContent cc = new ClipboardContent();
cc.putImage(im);
db.setContent(cc);
event.consume();
}
};
// any image view not in a list:
iv.setOnDragDetected(dragImageView);
Dragging from lists has to go on the list not the imageview, though isn't much more work.
The clipboard is even a bit easier although I did some strange behaviour with it losing track of the content of the clipboard from another application if it the copy was done when the JavaFX application wasn't already running.
It doesn't seem to support the primary selection though which is a bit of a bummer. Not that most applications that use it do it properly any more :( You do have to stop highlighting the selection when you lose it!
I realised the last time I properly looked at cut and paste was about 15 years ago with gnome-terminal ... hmm.
Compositing
I'm still needing to use BufferedImage for I/O but one of the things I was worried about was whether I would still need to use it for composition. Looks like I don't and depending on how one feels about the javafx layout and blending stuff it offers "all the power" of that - as well as CSS. You just set up an off-screen scene and either snapshot that or one of it's components.
Scene s = new Scene(root);
WritableImage wi = s.snapshot(null);
If you want alpha to make it through to the output:
s.setFill(Color.TRANSPARENT);
root.setBackground(Background.EMPTY);
(actually that is pretty nice, and about time you're able to unify off-screen and on-screen layout and composition. And the api doesn't prevent the implementation using hardware to speed it up either).
One thing I found a little frustrating is the fact that none of the layout objects clip their content. This makes sense but it ended up not quite working properly when I added it to what I thought was the right place - suddenly all my alpha compositing simply stopped working. I had to wrap the container inside another one and just set the clip on the outer container that then when into a BorderPane or whatever it was. Around this time was when I was also having the strange jumping layout issues so that artificially inflated the frustration level.
So just as i'm starting to get the feel of it after a few short hours ... i'm pulled away again.
(Not that I can't work on something at home ... which i probably will once this parallella has run it's course).
Update: So the "real" customer had a look-see this week and apparently were pretty impressed. Not that the JavaFX had much to do with it but it didn't hurt i'm sure. As I said to the PM afterwards and I quote "they're probably used to seeing some matlab piece of shit that takes days to run" ... anyway I guess we'll see if any more money comes along. If so it could be quite interesting indeed - although again, not particularly for JavaFX related reasons.
async copies
I had a play with this idea far too late into the night last night. A dma memcpy is a fine idea and has some benefits but isn't really taking advantage of the hardware features nor providing much opportunity to deal with some of it's problems.
You can resort to manual dma but that is pretty fiddly and bulky code-wise and it's such a common operation it makes sense to have it available in the runtime. And once it's in the runtime the runtime itself can use it too.
My current thoughts on the api and implementation follow.
// incase an int isn't enough at some point
typedef unsigned int ez_async_t;
// Enqueue 1d copy
ez_async_t ez_async_memcpy(void *dst, void *src, size_t size);
// Enqueue 2d copy
ez_async_t ez_async_memcpy2d(void *dst, size_t dstride,
void *src, size_t sstride,
size_t width, size_t height);
// Wait till dma is done
void ez_async_wait(ez_async_t aid);
// Query completion status
int ez_async_complete(ez_async_t aid);
To drive it I have a short cyclic queue of dma header blocks which are written to by the memcpy functions. If the queue is empty when the memcpy is invoked it fires off the dma.
The interesting stuff happens when the queue is not empty - i.e. a dma is still outstanding. All it does in that case is write the queue block and increment the head index and return. Ok that's not really the interesting bit but how the work is picked up. A dma complete interrupt routine checks to see if the queue is not empty and if not then starts the next one.
If the queue is full then the memcpy calls just wait until a slot is free.
I haven't implemented chaining but it should be possible to implement - but it might not make a practical difference and adds quite a bit of complication. Message mode is probably something more important to consider though. This api would use channel 1, leaving 0 for application use (or other runtime functions with higher priority) although if it was flexible enough there shouldn't be a need for manual dma and perhaps the api could support both. It would then need two separate queues and isr's to handle them and would force all code to use these interfaces (it could always take an externally supplied dma request too).
This is the current (untested again) interrupt handler. I managed to get all calculations into only 3 registers to reduce the interrupt overheads.
_ez_async_isr:
;; make some scratch
;; this can cheat the ABI a bit since this isr has full control
;; over the machine while it is executing
strd r0,[sp,#-2]
strd r2,[sp,#-1]
;; isr must save status if it does any alu ops
movfs r3,status
;; Advance tail
mov r2,%low(_dma_queue + 8)
ldrd r0,[r2,#-1] ; load head, tail
add r1,r1,#1 ; update tail
str r1,[r2,#-1]
;; Check for empty queue
sub r0,r0,r1
beq 4f
;; Calc record address
mov r0,#dma_queue_size-1
and r0,r0,r1 ; tail & size-1
lsl r0,r0,#5 ; << record size
add r0,r0,r2 ; &dma_queue.queue[(tail & (size - 1))]
;; Form DMA control word
lsl r0,r0,#16 ; dmacon = (addr << 16) | startup
add r0,r0,#(1<<3)
;; Start DMA
movts dma1config,r0
4:
;; restore state
movts status,r3
ldrd r2,[sp,#-1]
ldrd r0,[sp,#-2]
rti
When I wrote this I thought using 32-bytes for each queue record would be a smart idea because it simplifies the addressing, but multiply by 24 is only 2 more instructions and a scratch register over multiply by 32 so might a liveable expense. The address of the queue is loaded directly so that saves having to add the offset to .queue when calculating the dma request location.
The enqueue is simple enough.
// Pads the dma struct to 32-bytes for easier arithmetic
struct ez_dma_entry_t {
ez_dma_desc_t dma;
int reserved0;
int reserved1;
};
// a power of 2, of course
#define dma_queue_size 4
struct ez_dma_queue_t {
volatile unsigned int head;
volatile unsigned int tail;
struct ez_dma_entry_t queue[dma_queue_size];
};
struct ez_dma_queue_t dma_queue;
...
uint head = dma_queue.head;
int empty;
// Wait until there's room
while (head - dma_queue_size >= dma_queue.tail)
;
... dma_queue[head & (dma_queue_size-1)] is setup here
ez_irq_disable();
// Enqueue job
dma_queue.head = head + 1;
// Check if start needed
empty = head == dma_queue.tail;
ez_irq_enable();
if (empty) {
// DMA is idle, start it
ez_dma_start(E_REG_DMA1CONFIG, dma);
}
return head;
The code tries to keep the irq disable block as short as possible since that's just generally a good idea. With interrupts this is basically equivalent to a mutex to protect a critical section.
The journey. It continues.
So I spent the last couple of nights staying up way too late poking at my epiphany runtime library. Last night I basically finished the first cut of the feature complete version with a C implementation backend where i've finally settled on specific implementations of each function tuned mostly for size.
Most of the functions are intended to be inlined and only those more than about 10 instructions are worth putting into their own separate function - this in-lining is something the C compiler can do that isn't possible with assembly language. In every case these functions produce measurably smaller overall code by being inlined either because the function invocation itself takes more to form than the total work done and/or because it allows the callee to become a leaf and thus completely changes the way registers and stack can be assigned and used.
I spent a lot of time swearing at the compiler. Probably if i hadn't been up so late the night before I would merely have been puzzled by it. I played quite a bit with barrier() but ended up with something that is basically identical to the assembly version I came up with, but a good bit bulkier because the compiler doesn't try to use the small registers (0-7) for the whole function. It also does some redundant testing, but I think this is about as good as i'm going to get and it does pick up a couple of optimisations. I go into detail below.
I tried implementing the barrier of the the previous post whereby it tests 4 states per inner iteration but I decided to stay with the byte version due to the code-size increase. I also decided that because a workgroup barrier has to be workgroup wide, there's no need to ever allow the developer to assign memory for it and i've included the barrier memory as part of the runtime setup code (or it can be created by a linker script) which guarantees it will always be in the same location across all cores in the workgroup even if they are running different binaries. This can go somewhere fixed - either before or after the executive (aka kernel aka bios) or after the stack at the end of memory.
One of the 'interesting' loops was where the remote cores are notified of the barrier continuation. In e-lib it uses an array of pointers to implement this which leads to a trivial inner loop at the cost of extra memory reads. I decided to use the workgroup information to update the barrier directly which means it needs a bit more calculation which increases code-size but the loops are still quite small. This is the final implementation which I created by converting the assembly version literally into C after giving up trying to fight with the compiler for a better one.
volatile unsigned char *root = ez_global_core(barrier, 0, 0);
// Notify from farthest away
int r = ez_config->group_rows-1;
int ce = ez_config->group_cols-1;
do {
int c = ce;
do {
root[((r<<6) | c) << 20] = 0;
} while (--c >= 0);
} while (--r >= 0);
Looking at the listing output (oh boy, finding that option to as really brought back memories, it's -Wa,-a to gcc) one observes that the loop setup takes about half the code with the loop implementation taking up the other. The inner loop is only 6 instructions long.
// build with -O2 -msmall16
6 0000 0305 mov r0,#40
7 0002 C420 ldr r1,[r0,#1]
8 0004 4C010040 ldr r16,[r0,#2]
9 0008 CC210040 ldr r17,[r0,#3]
10 000c 0B800220 mov ip,_barrier
11 0010 9606 lsl r0,r1,#20
12 0012 7F900A24 orr ip,ip,r0
13 0016 9B03FF48 add r16,r16,#-1
14 001a 9B27FF48 add r17,r17,#-1
15 001e 0360 mov r3,#0
16 .L5:
17 0020 DF400608 lsl r2,r16,#6
18 0024 EF040208 mov r0,r17
19 .L3:
20 0028 7A21 orr r1,r0,r2
21 002a 9626 lsl r1,r1,#20
22 002c 9303 add r0,r0,#-1
23 002e 99700004 strb r3,[ip,r1]
24 0032 3320 sub r1,r0,#0
25 0034 70FA bgte .L3
26 0036 9B03FF48 add r16,r16,#-1
27 003a 3B000008 sub r0,r16,#0
28 003e 70F1 bgte .L5
This is almost identical to the assembly I came up with beforehand as the emboldened sections which indicate the non-redundant instructions show.
(untested code follows)
6 0000 0360 mov r3,#0 ; for fragment
7 0002 0325 mov r1,#ez_config
8 0004 C444 ldr r2,[r1,#ezc_group_id]
9 0006 E484 ldrd r4,[r1,#ezc_group_rows/2]
10 0008 0B000200 mov r0,%low(_barrier)
11 000c 964A lsl r2,r2,#20 ; groupid << 20
12 000e 7A48 orr r2,r2,r0 ; root = get_global_address(barrier, 0, 0)
13 0010 B390 sub r4,r4,#1 ; r = rows - 1
14 1:
15 0012 3B360000 sub r1,r5,#1 ; c = cols - 1
16 0016 D6D0 lsl r6,r4,#6 ; rshift = r << 6
17 2:
18 0018 FAF8 orr r7,r6,r1 ; o = (r << 6) | c
19 001a 96FE lsl r7,r7,#20 ; o <<= 20
20 001c 916B strb r3,[r2,r7] ; root[o] = 0
21 001e B324 sub r1,r1,#1 ; while (--c >= 0)
22 0020 70FC bgte 2b
23 0022 B390 sub r4,r4,#1 ; while (--r >= 0)
24 0024 70F7 bgte 1b
The register usage is because I extracted it from a bigger fragment - I may have broken it too but since it's only 4 accountable instructions different to the C compiler it should be ok apart from typos.
Being able to use the lower 8 registers really makes a big difference to code size as you'd imagine. Despite only being 4 instructions shorter this is 38 bytes compared to 64. Due to the ABI it does come at the cost of having to save/restore 4 registers which is 16 bytes of instructions and some time but it doesn't take much code before that is made up.
The other point of note is the direct use of the condition codes after the loop index adjustment. This saves one instruction per loop here plus the scratch register which would otherwise be required (line 24 and 27 of the first listing). This is really something I would expect the compiler to pick up. Perhaps less expected is the double-word load which saves one instruction, and the other instruction is saved by changing the redundant register move on line 18 in listing one into a substraction on line 15 in listing two which means the one on line 14 in listing one can be discarded.
The full barrier implementation needs to use a volatile read of a byte array and I found the compiler (and oddly, the arm compiler as well although only in the signed char case) does some weird and unnecessary data size extension when you read from a volatile char array which adds two pointless instructions for every byte load.
So with that the total implementation in C hits 144 bytes vs 88 for the assembly - which is fair enough I guess but there is room for improvement and only small things can make quite a difference when you're dealing with such small functions.
The e_barrier() code in e-lib compiled with the same options (-O2 -msmall16) hits 236, but much of that is because of the need to perform some multiplies. It's actually much worse than that because e_barrier_init() which is also a necessary part of the implementation compiles to 346 bytes ... :-/. I'm pretty sure the e_barrier_init()/e_barrier() pair has a race condition as well which can't be solved without another ... barrier.
If e_group_config is modified to include group_size and core_index values this drops them down to 132 and 224 bytes respectively which in terms of on-core code is still 2.5x the C one above (and 4.0x the asm). Strangely enough the notify loop uses implicit condition code tests on it's loop counter! But it added a separate counter register to implement this so that is probably why it isn't available on 'normal' calculations. Actually an externally initialised version of this implementation could be made very small; but then the data-size will dominate.
If only that hardware barrier worked properly in this case.
I did some further experiments while writing this and although it's kind of academic because of it needing 5x the runtime memory I fiddled with an implementation that uses an array of pointers assuming they have been initialised by the runtime / loader. I can get it down to 70 bytes for the barrier() call.
a (slightly) better barrier, signals
So I kept thinking about the last point on the last post. That is, having a job-based runtime similar to hsa queuing.
One feature that employs is a signalling mechanism that I think lets waiters wait on multiple events before continuing. I came up with a possible epiphany-friendly primitive to support that.
First there is the sending object - this is what each sender gets to signal against. It requires a static allocation of slots by the runtime so that they can signal with a single asynchronous write without needing an atomic index update that a semaphore like object would require.
struct ez_signal_t {
uint slot;
ez_signal_bank_t *bank;
};
void ez_signal(ez_signal_t s) {
s.bank->s.signals[s.slot] = 0xff;
}
The signal bank is basically just an array of bytes per sender. But also allows for word access for a more efficient wait.
struct ez_signal_bank_t {
uint size;
uint reserved;
union {
ubyte signals[8];
uint block[2];
} s;
// others follow
// must be rounded up to 4 bytes
};
(the structure is aligned to 8 because of the union, it's probably not worth it and i will use an assembly version anyway)
The bank is initialised by setting every byte within size to 0, but those outside of that range which might fall into a word slot are pre-filled with the signal value of 0xff. This allows the waiter to execute a simple uint based loop without having to worry about edge conditions.
void ez_signal_wait(ez_signal_bank_t *bank) {
uint size = bank->size;
volatile uint *si = bank->s.block;
size = ((size+3) >> 2);
while (size > 0) {
while (si[size-1] != 0xffffffff)
;
size -= 1;
}
}
The same technique could be applied to barrier() as well - a tiny improvement but one nonetheless. It does need some edge case handling for the reset and init though which will take more code.
(Hmm, on closer inspection it isn't clear whether the hsa signalling objects actually support multiple writers or not - it seems perhaps not. However they do operate at a higher level of abstraction and may need multiple writers internally in order to implement the public interface when multiple cores are involved. Update: On further inspection it looks like they are basically signal semaphores but with a flexible condition check on the counter and the ability to use opencl atomics on the index.)
Actually another thought I had was that since the barrier is tied directly to the workgroup why is the user code even calling init on it's own structure anyway? It may as well be a parameter-less function that just uses a pre-defined area as the barrier memory which is setup by the loader and/or the init routine. I think the current barrier implementation mechanism may have an issue because you can't initialise the barrier data structures in parallel without it being a race condition and so even the setup needs another synchronisation mechanism to implement it properly. By placing it in an area that can be initialised by the loader/runtime that race goes away quietly to die. And if the barrier is chip-wide or the WAND routing ever gets fixed it could just use a hardware barrier implementation instead.
This thought may be extended to other data structures and facilities that are needed at runtime. Why have the error prone and bulky code to setup a dma structure when the logic could be encoded in a routine with simple arguments that covers the most common cases with just two functions: 1D and 2D? And if you're doing that why not add an async copy capability with interrupt driven queuing as well? Well it may not be worth it in the end but its worth exploring to find out.
gdb sim
So I actually used the gdb simulator for the first time yesterday to step through some assembly language to verify some code. So far I've just debugged using the run-till-it-stops-crashing method. Bit of a pain it doesn't seem to think assembly language is actually a thing though:
$ e-gdb
...
(gdb) target sim
Connected to the simulator.
(gdb) file a.out
Reading symbols from /home/notzed/src/elf-loader/a.out...done.
(gdb) load
Loading section ivt_reset, size 0x4 lma 0x0
Loading section .reserved_crt0, size 0xc lma 0x58
Loading section NEW_LIB_RO, size 0x170 lma 0x64
Loading section NEW_LIB_WR, size 0x450 lma 0x1d8
Loading section GNU_C_BUILTIN_LIB_RO, size 0x6 lma 0x628
Loading section .init, size 0x24 lma 0x62e
Loading section .text, size 0x268 lma 0x660
Loading section .fini, size 0x1a lma 0x8c8
Loading section .ctors, size 0x8 lma 0x8e4
Loading section .dtors, size 0x8 lma 0x8ec
Loading section .jcr, size 0x4 lma 0x8f4
Loading section .data, size 0x4 lma 0x8f8
Loading section .rodata, size 0x8 lma 0x900
Start address 0x0
Transfer rate: 17632 bits in <1 sec.
(gdb) b ez_signal_init
Breakpoint 1 at 0x85a
(gdb) r
Starting program: /home/foo/src/elf-loader/a.out
Breakpoint 1, 0x0000085a in ez_signal_init ()
(gdb) stepi
0x0000085e in ez_signal_init ()
(gdb)
0x00000860 in ez_signal_init ()
(gdb)
0x00000862 in ez_signal_init ()
(gdb)
0x00000864 in ez_signal_init ()
Yay? Had better tools on the C=64.
But I came across this article which provides something of a workaround. Not the best but it will suffice.
(gdb) display /3i $pc
1: x/3i $pc
=> 0x864 <ez_signal_init+10>: lsl r1,r1,0x2
0x866 <ez_signal_init+12>: lsl r3,r3,0x3
0x868 <ez_signal_init+14>: beq 0x880 <ez_signal_init+38>
(gdb) stepi
0x00000866 in ez_signal_init ()
1: x/3i $pc
=> 0x866 <ez_signal_init+12>: lsl r3,r3,0x3
0x868 <ez_signal_init+14>: beq 0x880 <ez_signal_init+38>
0x86a <ez_signal_init+16>: mov r2,0xffff
(gdb)
0x00000868 in ez_signal_init ()
1: x/3i $pc
=> 0x868 <ez_signal_init+14>: beq 0x880 <ez_signal_init+38>
0x86a <ez_signal_init+16>: mov r2,0xffff
0x86e <ez_signal_init+20>: movt r2,0xffff
(gdb)
0x0000086a in ez_signal_init ()
1: x/3i $pc
=> 0x86a <ez_signal_init+16>: mov r2,0xffff
0x86e <ez_signal_init+20>: movt r2,0xffff
0x872 <ez_signal_init+24>: lsr r2,r2,r3
(gdb)
set confirm off may also reduce offence, but otherwise it was all a bit easier than I remember it seemed when I read about it as the few commands above show.
I tend to only use debuggers these days as a tool of absolutely last resort. Most 'bugs' I work on are 'known bugs' like missing or incomplete features where such a tool isn't any help. And debugger features which should be useful like variable watches are often just too much of a pain to setup and work with or trying to step to the 5000th iteration of some loop which also tends to be painful with debuggers even if they have support for it. Obviously i'm not having to debug production code in-place these days, thankfully.
But I remember using it heavily in the bad old days when working on Evolution which uses threads a fair bit - whether each updated revision of gdb would even be able to produce usable stack traces of a threaded application was a bit of a lucky dip. I remember not upgrading for years at a time once I got a combination of kernel and gdb that actually worked. It's probably the single most influential experience that lead me to become very conservative with my tool updates. There's nothing more frustrating than having to fix a breakage for something that wasn't broken in the first place.
Copyright (C) 2019 Michael Zucchi, All Rights Reserved.
Powered by gcc & me!