Asynchronous non-copying message passing

Well I decided to shelve work on the MMU code for now and focus on easier targets. I implemented the basic functions for message passing first. Although since I turned off the memory protection things are quite simple - it just passes around pointers for the most part. The idea for enabling memory protection would be to have each 'message' allocated page aligned in a global virtual address range, and just re-map the page to the target process address space when it is received or replied to, thus enforcing the rules of who owns the memory when; although any data the message references is another issue. I could force messages to include all data they reference (they have a length field), and perhaps special-case io request's buffers. If i force those semantics, then it could probably allow for a copying implementation to be written as well.

Ports are stored inside kernel memory and referenced by handle, rather than by address in user space - so they can go away without fatal problems and can't be corrupted. One problem is that I will probably need to add another query system call to get a copy of objects like these as they contain information a task might want to access - like the signal bit used. Probably useful for other privileged system objects anyway, like library bases or tasks. Anyway, I managed to send messages between tasks 'in the normal way'.

I also had to write some TagList utilities - and I added a new one which can 'pack' a taglist into a structure. Somewhat inefficient but it'll do for now.

And i'm starting to think about how the various `kernel' interfaces might work in practice. Basically everything is accessed through (globally) shared libraries with a few system calls thrown in when necessary. But some of the code wont need to run in supervisor mode at all or go to another server.

So what is the next thing to look at ... well I decided to look at devices, and again checked out the AmigaOS implementation. It's a little different from microkernel models I've read about - io requests can be handled on the user context, for example. And apart from that they are actually implemented as shared libraries - complete with the possibility of public functions directly callable. But long-running i/o functionality runs in a separate task and may interact with interrupts and so on - like a normal uKernel approach.

In a protected environment things will have to change, although the basic ideas seem quite sound and usable.

But before I can get that far there's a lot of mucking about to do. I need to work out the design of the process and thread model and how they'll start and finish.

And how to do libraries; devices are just libraries too. Shared libraries will be objects, not just function tables, and in a virtual memory environment this creates some issues. e.g. should it allow system-wide shared data, per-user shared-data or just per-process/thread state. Also do I want to enforce `pure' re-entrant code for libraries - which saves the hassle when working out what to do with the data segments.

Signals and such

After a couple of late nights I've hacked up a few more bits and pieces for my 'os'.

Signals were pretty easy to get going. These are a bit like posix signals, but also not at all the same. A low-level asynchronous bit-level synchronisation mechanism basically. You can allocate a signal 'bit', wait on it (task goes to sleep) or send it to another process (thus waking it up, and perhaps pre-emptying yourself if it has a higher priority). It took a bit longer than I'd hopped because I aso 'cleaned up' the code - started creating a better directory structure and moving things out of the single c file I had. I rewrote the interrupt handler and system call assembly too - they're pretty trivial so it wasn't much.

I also did a lot of reading about `OS4', the latest AmigaOS. Some interesting ideas there, particularly the extension of the basic object oriented nature of the system further. Hmm, food for thought.

And investigated possible ways to get video output in the future. Sigh. PC video hardware sucks. It's not something I want to get bogged down in, but I might have to if it comes to that. I guess I can get some drivers from some generic library/X/or linux even, if they can be extracted or wrapped somehow anyway.

Being annoyed at the PC hardware I was looking at getting a beagleboard to play with - ARM looks like a lot more fun. But just as I was about to hit the buy button I had second thoughts. The display hardware is completely proprietary and undocumented and I can't really see the point if I can't get that to work. A real pity, as otherwise it looks like nice hardware that would be interesting to work with. Maybe somewhere down the line.

Then the next night I started looking at how to implement message ports and message passing. On AmigaOS these are really just arbitrated queues associated with a (possibly shared) signal bit. Very simple, very fast. Question is how to add memory protection in a way which maintains the simplicity. And since I hadn't done any paging stuff yet I got side-tracked learning how that works.

For message passing w/ memory protection in OS4 they've implemented MEMF_PUBLIC, so it actually means something, but I presume it also means all public memory is public to all processes, which is probably not what I want. That allows all processes to access the shared resources like message ports using the same virtual memory address. I thought instead perhaps I could just allocate the memory in a fixed 'public/shared' virtual address space, and migrate the memory around to the current owner using the paging system. But after reading various comments from kernel developers about the performance, I'm not sure it's a good idea. Then again, maybe it's just the cost of using an MMU and can't be avoided. And then I got so side-tracked with how the VM system should work (do i leave it in kernel space, or implement it as a service, etc) I feared I was getting too bogged down and had to put it aside for later.

And even as far as message ports go I probably need to move to a 'handle' model, rather than passing around pointers. So that meant an allocator that could initialise all the user fields, and OS4 has a generic tag (varagsish) based allocator for those so I had to look into tag utilities and such before even having a chance to go further.

And today i've got a little cold. Hrmph.

More muck'n about

In the quest to find something to keep me busy I've switched tacks again. I've been playing about with writing an operating system, of all things. I guess it will only be a diversion as it would take years of work to get anywhere, but it is nice to play with some low level stuff for once.

I think the idea came after I found some link to Minix 3 and had a look at that. It looked quite interesting, and I always had a soft-spot for micro-kernels. It's surprisingly functional and very small - boots in an instant. Although one problem is you really need to buy the book to understand the code as the documentation is lacking, and even then the code has moved on quite a bit since the book was written, so it isn't terribly useful anyway. The way the various servers inter-communicate isn't very simple either; for a microkernel it seems to have a tighter coupling between servers than I'd have imagined, although there is probably a reason for that. I think the original goal of nice clean readable code has also been corrupted by the bolting on of new features like virtual memory and a vfs layer. And then there are some serious performance issues with it's synchronous message passing design; well not so much the message passing bit (although the overhead is quite large) as the way the services have been implemented. I'm sure it makes the code simpler but it just doesn't scale. And no threads. I do like my threads.

But anyway, the idea of a tiny micro-kernel intrigued me, so I read up on Minix 3, HURD, Mach, L4, ECos. Grand ideas flashed before my eyes. Then all the work involved pulled me back to earth. It took me days just to get a context switch working because of the shitty PC hardware (more on that below). So I definitely want to keep things simple and have pretty low expectations of where things might end up, if not just another long-forgotten directory on one of my PC's (which is pretty much a dead-set certainty no matter what else happens).

The PC isn't nice to play with at all. How disappointing that such an utterly bullshit architecture ever got so common and cheap. First there's the processor, which starts with an ancient and obtuse instruction set and then just goes down-hill from there. Segments and TSS? Enough said. Even the manuals aren't very good; after having read other processor and hardware documentation, they seem to make things far more difficult to find than necessary and leave out too many important details. Then there's the rubbish hardware beyond the rubbish processor. Only one real timing source? And even that isn't very good. Jesus, even the bloody C=64 had far more advanced peripheral hardware than that. Even one of it's timer chips had 2 timers (plus a TOD/clock - which also had an interrupting alarm function), each of which could create interrupts, and they could be cascaded for very long intervals, they even had safe easy programming interfaces - and it had two of these chips! I don't even have to mention the SID chip vs's the PC sound - oh that's right, lets just use another timer channel to make offensive beeps instead.

Sigh. So I should really be targeting the PS3 I guess; maybe if I get the system simulator up I'll look into it. The docs on the hypervisor are a bit slim though.

Well, despite all the rubbish you have to deal with I'm making slow progress; thanks to Grub and some of the free OS's out there and making lots of mistakes in Qemu and Bochs. I have a pretty simple context switch and system call mechanism sorted out, and basic interrupt handlers for timer and keyboard. I got bogged down just writing a very simple memory allocator, which was quite disheartening (I blame lack of sleep; I made it in the end). And instead of Minix or HURD I'm looking to AmigaOS for inspiration instead. It's such a simple and functional design; although I'd like to play with memory protection and similar ideas which don't translate directly and will make things a bit trickier and slower (than they might be otherwise). So that probably means the next thing to look at is signals - which are quite trivial (the only mechanism in AmigaOS used to wait on events or signal them), and then message ports - which aren't much more complex (although separate address spaces makes the whole notion quite different). Once they're done I can probably look at implementing simple devices like keyboard and timing. Although I might also have to look at how I might handle process isolation and virtual memory at some point fairly early in the piece too.

Well, unless I get bored with it all before then.

People actually pay for this rubbish?

I was forced by the project manager to convert one of my documents into Microsoft Word format.

Wow, what an unpleasant journey this turned out to be. First the Lyx/LaTeX to ODF converter didn't work very well. Although I eventually got it to work on a GNUOS machine by removing some ps-tricks diagrams first. The text converted ok, but there were other problems. All the cross references were broken and needed to be fixed up, and the images needed re-inserting as they'd been re-sampled at a lower resolution.

Fighting with OpenOffice 'dot' org's nasty GUI features wasn't much fun. Such as the ones that have the screen bouncing around as you navigate through different document elements; often losing focus in the process -- navigating by keyboard was simply impossible. And finally, after getting the document to a decent state following one last change it simply refused to save in ODF format, so I was forced onto Microsoft Word.

So then came the less-than-perfect conversion to Microsoft Word format. It broke all of the references again. And inserted images, captions, and tables in a way which didn't work very well and needed to be re-done. Microsoft Word crashed twice just trying to re-arrange a table; and had the gall to make out that it was doing me a favour by (very) slowly ``recovering'' my work. Lots of issues with cross referencing (the cross referencing tool is worse than the OpenOffice 'dot' org one, and that's saying something), numbering, tables of contents, maths sections, styles, formatting, and even plain editing functions and the user interface. None of it worked well, and much of it barely worked at all.

Over-all, a few days of a head-ache inducing and hair-greying experience I hope not to repeat any time soon. At least OpenOffice 'dot' org has an excuse, it doesn't cost anything. Although the claimed interoperability is simply not there; but we all know who to blame for that. But someone paid for this Microsoft Word, and it is an utterly ghastly horrible product that only produces amateurish results when it does manage to do what you ask it to.

After having written the document originally using Lyx, and having recently worked with Texinfo as well, even their combined quirks and difficulties pale in comparision to these utterly unproductive, rubbish tools.

Introduction to CBE, Part 2 (chapter 3)

I've uploaded the second part of my Cell BE intro up toEverything Cell. Things are a bit more interesting this time and it gets straight into talking about double-buffered DMA and SPUs and registers and other fun stuff. Should be enough to digest until the next chapter which gets into multi-processing.

Whilst checking that it uploaded ok I did some proof-reading and realised the writing style isn't terribly good. If wordpress didn't force html line breaks all over the place it would look a bit better too. My excuse is that it's only a draft and i'll keep working on it as I have time.

Although now i've got this out of the way i've gone back to playing with some of my Java projects for a while. Oddly enough, using wordpress has given me inspiration to work on my own internet publishing tool again; although it also shows just how far off my ideas are of being realised too.

National Broadband Network

Wow, so Ruddy and Crew want to build a fibre-to-the-home network for the whole of the country through a publicly sponsored company. Well since nobody else would do it, I guess they had no choice, and It's about time we jumped into the 21st century. We've been held back too long by abusive behaviour from our local telephone monopoly.

All the 'pundits' keep focusing on the impact on phone companies and ISPs ... Far too narrow a view -- I think for example that if I was a free-tv television broadcaster I'd be somewhat more worried than anyone else right now.

One negative is that this is the same government who wants to censor the internet. That's a major negative.

Introduction to CBE

Aren't we spoilt today, two posts in a few hours.

Just a quick mention that I've started publishing my introduction to Cell programming - based on the Mandelbrot generator I wrote last year and a few other bits and pieces.

It's on a new site Everything Cell which is a tad bare at the moment, but will hopefully tie together some cell programming resources for free software developers and hackers interested in programming on the PS3 and the CBE CPU.The first chapter is a bit unexciting but sets up some framework before I can get into the nitty gritty. It's surprising just how much work it is getting this stuff together.

PS I cleaned up the theme here too.

About Me

Tags