Why is software so far behind hardware?

For various reasons I was trying to get some binaries from some very large newsgroups - so large that I couldn't find any client that would even list all of the messages without hanging or crashing. Interestingly, I found the mail/news application in seamonkey to be one of the better ones - it worked much better than thunderbird, using far less memory, although it still got stuck very often, and that was only with 200 000 headers rather than the 6 000 000+ in the group.

I saw a few scripts/programmes for grabbing stuff in the repository for this machine so I gave them a go. I can't remember all I tried but `brag' seemed to fit the bill and worked ok on a smaller test group. Well, needless to say it just didn't scale. Either the code is bung or tclsh has a big memory leak or simply isn't up to the task. It managed to download all of the subject lines (meanwhile memory use - ~2.9G!). Well my machine wasn't really swapping so big deal eh? But then when matching the headers memory use continued to climb - even faster. It ran out of 32-bit virtual memory ... well I guess I should've been using a 64-bit OS for such a complex task!

Err, no, that's just bullshit. So I hacked up a quick and dirty C version. Has somewhat fewer features but works more or less the same for what I needed. A bit over 400 lines of code in total, only using libc. No smart algorithms apart from a zero-copy 'read-line', no asynchronous reads or other decoupling of net i/o from disk i/o or processing, etc. Total memory used whilst reading and saving the subject headers of said 6 000 000+ group - about 100K (and most of that is from the C library, I think I malloc about 50 bytes, and have 11K of bss). Total CPU used, about 10 minutes real-time, and about 5 seconds cpu time (top consistently showed 0% cpu usage - well I mean as it should, it's entirely

I/O bound!). It takes about 3 seconds to scan the 780MB cache of titles with a regex when they're in the buffer cache, about 10 otherwise (using the same amount of memory).

Alas, this is the world we live in. This is why a `netbook' isn't big enough to run `real' applications - it will just be destined to run slow scripts which proxy the heavy lifting (when it is quite capable of doing an awful lot of the heavy lifting itself). It will be kept busy struggling to show adverts instead (even something my 4 core machine has trouble with, so I turn off animated gifs and flash and block lots of advert pics). This is why, after all the advances in hardware, we now have shitter apps that run more slowly - e.g. for all it's cracked up to be, compare google-writer to say, abiword, or calc to gnumeric - with no other benefits like stability, security, and so forth.

I was looking at javafx again just recently. Someone hacked (and I mean hacked) up a sort of 'space invaders' game. Except it has no sound (could be my use of oss4), no barriers, the aliens don't move properly or speed up, and unusable controls. And it jerks around in an unplayable fashion, with tearing and an utterly atrocious frame-rate. Flash et al are pretty much the same to me. Slow, jerky - crappy games. The machine in question is capable of emulating real hardware faster than that - you can play the real space invaders (or for me, the C64 version) with proper sound, control, and 'game' all in it's glorious full-frame-rate goodness with no problems (and so it should, it's emulating a 30 year old computer!), but it struggles with a half-arsed rip-off.

Poor algorithm choices. Bad language/framework/environment choices. Poorly implemented language/framework/environment. In short, poor engineering and system design.

Computer Science lecturers always say `we don't teach computer programming'. And it's pretty obvious - no, no they don't.

(TBH `brag' does more processing than just a regex match on the titles. It tries to find the filenames, and count the parts, and tries not to grab duplicates. And it uses multiple connections to improve downloading of the actual messages. Still, there's no reason it should use anywhere near that much memory, loading all subjects into memory at once is not ncessary.)

Coloured Boxes

Was feeling a bit blah and too lazy today so instead of cracking on in the garden I did a bit of hacking. Ok, it's not much but I got a bit of code running on the beagle board that draws random coloured boxes to a framebuffer.

It's being loaded as a kernel image directly from u-boot of course, and then just `banging the hardware'.

I just snarfed the framebuffer init code from the splash screen from an old version of u-boot, with a few tweaks - remove the writes to read-only or unmapped registers and changed the screen size to 1024x768.

Hmm, not sure next. Maybe a serial driver. USB would be nice, but a bit much at this stage. Or the retaining wall!

It beat me, another year on, and what happens next?

I gave dragon age one more go. Started from scratch and spent my upgrade points wisely rather than wasting them on turning a rogue into a fighter (it's been a while since I played a WRPG). Things were going swimmingly for the first 5 hours or so, even managed to do things I couldn't before like get the two horny elves together, but then I hit a too hard part again and I got sick of retrying or wandering back and forth looking for something I could manage. I switched to easy mode.

Well, it's a little too easy but it's ok, at least I can see the story out now (which is playing out differently - this time i'm a human rather than a whiny sickly looking elf) and make some progress.

Ahh yes, another year on. It's not ending very well, I'm barely sleeping and feeling generally pretty miserable. Blah. Reading too much stuff on the intarwebs doesn't seem very healthy, it's all just so depressing. The world is going to shit really and observing that isn't as fun and interesting as I thought it would be, at least for me. Hmm.

I still have work to do in the yard, which I haven't touched for two weeks, but at least I now have a wheelbarrow, so it should go a bit faster once I get back into it. It's been a bit too hot the last couple of days to even consider it - if I had the energy to go that far anyway.

I'm still thinking about projects to work on. It's a tricky prospect however, as all of the interesting stuff is just too complicated to get involved in anymore. Or it's already done. Or it seems like a dead-end waste of time - although perhaps that is what I need, just a hobby again. I'm thinking of just doing some isolated hacking on the beagleboard - at least I have almost complete documentation for it - all 3 516 pages of it, not including the CPU. First I think I'll try and get u-boot booting something which can display to a frame-buffer. That should be doable at least. Still, not holding my breath on that one.

The Puppy Breathes Again

I was talking about ARM stuff on an IRC channel and got interested in playing with my BeagleBoard(s) again. I remembered there was a hardware issue that caused the UHCI port to fail pretty regularly and looked up to see if the designers had found a fix.Fortunately they had.

Subject: [beagleboard] Re: USB EHCI problems
From: Gerald Coley (ger...@beagleboard.org)
Date: Nov 9, 2009 5:57:18 am
List: com.googlegroups.beagleboard

This issue is due to be fixed with on REV C4. It is not clear that it can be fixed on all Rev C3 boards. You are free to try an RMA, but their is no guaranty that it can be fixed. What will be done there is the soldering of the 20uf CAP across C97 and then they will run the board for a couple of days to see how well it works.

Gerald

So, at 3am on a Saturday morning I went looking for parts. I was feeling a bit apprehensive about soldering on such a tiny part, but I didn't really feel like sending it in either. I found a 22uF electrolytic cap on an old motherboard on which all the green caps blew which fit the bill and fit the tiny spot on the board. I worked out which pin was connected to the ground rail and gave it a shot.

So after that I had to re-setup a booting image - previous ones had been wiped to use my digital camera, and a HDD was wiped to transfer tv shows. Setting it up is a bit messy but didn't take too long. Once I got it setup I had it playing video and internet radio and left it running overnight (well over-day, it was about 6am by now).

And it's been up since.

Nice one - before I was lucky to get an hour, and copying some video files across the network would've almost certainly killed it. Well thanks to the BeagleBoard guys for finding a fix. After reading about the technical specs of the OMAP chip I'm almost surprised the thing works, electronics is such a fiddly thing, compared to software where at least 0 always means 0 and 1 always means 1.

Now, to find something to do with it ...

Too many late nights, Dragon Age

Well i've been losing the plot a bit lately - staying up till 5:30 playing games, watching tv, commenting on blogs I should probably not be commenting on. Had a strange episode last night too - possums were running across my roof about 3am making a terrible racket - well that happens almost every night. I went to go outside to shoo them away but caught sight of some guy on the road with what looked like a golf club. I snuck back inside and went back to what I was doing. There was quite a bit of screeching going on before everything went quiet. Must say i've thought of doing the same myself (or what I imagine he did, I didn't look), but they are a protected species. Yard has gone no-where all week, although i've been keeping the plants alive and eating lots of button squash. I did a couple of hours of work this `morning' but tired out pretty fast (should've had breakfast first). I was contemplating doing a bit more this afternoon but after I had some lunch and sat back down on the computer I might just give it a rest for another day.

The 5:30 sessions have been for Dragon Age. It's a strange game, i'm still not sure if I really enjoy it yet, or if it's more like work and I think i've spent more time talking than fighting so far. I guess that's ok in and of itself, and it is mostly done pretty well (it's ALL spoken - most of the voice acting is good too), but it's a bit of a change from other RPG's i've played where the dialogue is just a plot device and not a past-time in itself. Although there's been a couple of times - like when the busty wild mage starts going off about turning into a spider - that were quite confusing, since there didn't seem to be any reason they were talking about it at the time. Also feels a bit funny when, after chatting with some NPC cordially for 5 minutes, you accidentally do one wrong thing and they wont talk to you ever again.

It's big and open and non-linear, although that usually means you don't know where you're going at some points. There also seems to be long-term consequences to how you treat allies and alternative ways to solving things - which is a nice change from 'it doesn't matter what you do/the order you do it in, it always goes through the same steps' (well it seems that way, who knows in the end). Which party members you have active at a given time seems to affect conversations too - but that might've been a coincidence. One problem is that you're often not in the possession of enough information to know if you're doing 'the right thing', and sometimes the decisions are made unknowingly - like turning the wrong corner and getting stuck in a fight you didn't intend. I need to save more often - but why does it take so long to save! It takes about 30 seconds, 10 of which you're paralysed cold and the rest runs mostly in the background. And there's a lot of reading - all the writing (and there's a LOT of it to read if you want to) is too small too for a TV game, and light-on-black which is difficult to read.

The combat system isn't too bad, although a bit clumsy at times - being real-time makes it hard to keep track of all 4 characters and I keep losing them since they don't drug themselves up with health `potions' automatically - I guess I have to investigate the `gambit' system further. Final Fantasy 12 did a nicer job of introduction this programming system I think, made it more accessible, even if it took a lot longer to become useful. For potions, you have to change your current character to the one you want to take the drugs - which is clumsy in the heat of battle. Often when you do, the guy you were playing decides to rush in with his sword when he was quite safely firing arrows - or he just wanders off and forgets to fight! Also I don't know why (nor do I like it) how the current character keeps saying 'yes i will do it', 'as you wish' and crap like that every time you hit X to swing their axe or pick a lock! It completely removes the third-wall for no apparent reason. Isn't this meant to be 'role playing', not 'god playing'?

Early on I didn't get a back-pack at one of the shops (and they're very rare) and now i'm permanently down 10 'weight points' of carrying capacity since that shop has vanished (unless I go back 10 hours of playing) - and it's a constant pain. Mechanics like that are just frustrating and pointless. It adds an unnecessary level of micro-management whereas this is the sort of knowledge you'd expect the players to just know and take care of themselves. Be better if it was just experience and/or strength based. The inventory/status screen seems to overlay on the the live image (whilst pausing it). This seems like a really odd decision, and all it means is sometimes the menu system runs somewhat slower than it should, and uncomfortably so. It is mapped to the controller quite well though, making good use of all of the buttons and sticks - not just a PC port where they map a pointer to one of the sticks, and X to the left mouse button.

Technically it is no Uncharted 2 by any stretch. The stills may look ok (and some look fantastic) but there's a fair amount of popin and the framerate is very unsteady which detracts significantly from how it feels when wandering around. Add some very lo-res textures for things like the ground or tables which are part of canned cut-scenes and conversations which you are guaranteed to look at closely, and it looks more dated than it should. There are very long (10-20s) loading pauses between areas or if you die, even with a big (and painfully slow) copy-to-hdd bit when you first run it. Naughty Dog shows it can be done without the pauses, so it's a pity most developers are too lazy to do it and stick to the crappy 'pc way' of stop, load everything (again), continue, which doesn't translate well to optical storage. Traditionally I put this down to two things - 1, it's easier, a lot easier - and it's not just code, it involves art assets too, and 2, PC hardware can't do two things at once, like loading off a slow disk and keeping number crunching going, but there's no excuse in this day and age and hardware and gigantic budgets for such a naive implementation.

Well over-all it's a pretty decent and involving game so far, even with those criticisms. And it's nice to have a half-decent RPG on my PS3.

Time and Change

Well time has been passing pretty fast lately.

After a visit to a mate who'd done a lot of work on his yard in the last few years I was convinced of the need to think a bit bigger in the back yard department. So i've ordered a shed and a new verandah and have spent the last month working on some site preparation. Well I still have plenty of time but a sense of urgency is starting to creep in since there is still just so much work to do. Originally I was just going to work on the site prep and worry about the rest of the yard later but I do have an awful lot of time and it would save work if I got a few things, like a long retaining wall, out of the way first. Rather than moving piles of dirt and things from one place to another multiple times, I can just move it where it needs to end up and leave it there. Might even be able to get some lawn going by then.

It's been damn slow going though. Weather is too hot or too wet, i'm too hungover, I don't really feel like getting covered in sweat and dirt that day, or lately i've just been extremely tired. I often fall asleep infront of the tv before going to bed. So sometimes I just need to sleep for half a day to catch up a bit. Blah. A few hours, a few days a week, using nothing but a bucket and a spade, well, it's slow and hard work. Disappointingly I haven't lost any weight mind you :-/

The front yard isn't dead but it isn't really flourishing. The thyme lawn is growing very slowly but at least the roses are about to flower again. I think I haven't been watering it enough. The vegetable patch is going pretty well though. Button squash are already fruiting regularly, as are the cucumbers. Tomatoes have a lot of growth and i've had a handful of tiny cherry tomatoes off this week, but they've just covered themselves in masses of flowers so maybe they'll kick in properly by Christmas or New Year. A few more bees around with the weather warming up and settling down too. The sweet potato slips are starting to grow at last - I think they need the ground to warm up a bit more first. I've tons of things in pots too, so just keeping everything alive takes an hour or so every day.

My house-mates just moved out on the weekend so things have gotten a lot quieter all of a sudden. Well it might help me sleep a bit more in the mornings, but the sudden lack of company will take a little while to get used to again. Another visit to another mate gave me the idea of putting in an en-suite for one of the bedrooms and renting it out to a professional woman, so I'm considering that too. Visiting friends is getting costly!

I never finished Uncharted 2 - although I still intend to one day. I'm a fair way through Ratchet and Clank: A Crack In Time too. The last one I played was a bit easy and lacking something new, but I think they did a much better job this time - if you were ever an R&C fan and lost interest, this one is worth a shot. Oh and they finally implemented triple buffering so no tearing and little slow-down when they drop a frame. I also got Dragon Age: Origins but haven't played it very much so far. Partly because others wanted to use the TV, and partly because it starts a bit slowly and feels a bit too 'pc-game' (I can barely read the typeface on most windows!). I think I played more of a neat little free game called 'Widelands' which I found when looking for a 'The Settlers' like game from the Amiga days. I was playing with UAE one afternoon and re-discovered that gem. Widelands has the mechanics all there but the lo-res graphics on the Amiga had a lot more personality and charm (and much better sound fx). Back on the PS3 I got a PlayTV - quite happy with it so far. Certainly a few things could be improved but it works quite well, and it's easier to setup than MythTV. I might get another one just to plug into my PC for MythTV since my USB tuner never worked properly (lots of broken data, particularly if both tuners are on but not specifically so). At $150 it's not bad for something that I KNOW works reliably and with Linux.

Haven't touch Haiku or the beagle boards or any coding at all for ages either. Well now I have an excuse in the yard ... but part of it is that many of the interesting things to do are just problems that are too big for a part-time hobby. Well there's still the idea of a little game, that could still be simple I guess - although my game-designing mate has dropped off the radar lately too. I suspect once I'm working on something interesting again I'll find coding as a hobby more interesting too. Perhaps!

And this period of what I like to call my `unpaid long service leave' finally has a cap to it. I've got some more work lined up early next year which hopefully will be more up my alley than the last few jobs I've had. Using CUDA or I would think more likely, OpenCL to do some image processing. CELL BE would've been fun but with the new PS3's not supporting Linux it would be a hard sell.

Holidays are great ...

... you don't have to do anything.

Oh I've been poking around in the garden and the yard, playing with Haiku (porting emacs, but not much) and reading lots of blogs about the impending catastrophic failure of western civilisation and drinking and eating too much. And even playing some Uncharted 2 (probably the best game i've ever played to date in all respects, technical, graphical, story, acting, action, aural maybe not but certainly up there).

More beagleboard stuff

I've still got some problems with the USB port - although now i'm using a USB HDD for the OS it seems to be a lot more stable. Every now and then it dies and requires a reboot. Hopefully the BB guys and/or TI work out what it is, both of my boards have it. Instead of using the usb OTG port i'm using a PC PSU to run the board - I thought that might help the crashing problem, but it didn't really make any difference.

I haven't had much fun with the default demo build of Angstrom. It's limited in strange ways. I haven't been able to get it to compile either; I got it further by forcing a serial compile, but it still eventually dies, and the build system is shitting me off too.

I gave Android another go - now i've got a screen. Yay, a phone interface. Doesn't really work with a mouse and keyboard, the network wouldn't work and things crashed a lot. Oh well, I think the basic idea is right (throw everything away and use linux as a set of device drivers), but a phone-os has little appeal to me (i'm still using a nokia 8250).

So I tried the Touchbook's OS from Always Innovating ... it's just a custom build of Angstrom. Much better though - the sound even works (sounds quite nice quality to my shitty ears too), although mplayer doesn't now :-/. It's also a bit snappier (it has the powervr drivers and some dsp stuff too). Running off the usb hdd is a lot faster than flash too - it was my old playstation hdd so it isn't even a particularly fast drive. It seems to be set up reasonably nicely although it has a few weird things - like no shadow file (`sudo sh' gives you a root shell). I have to manually bring up the usb ethernet adaptor, otherwise it basically works straight out of the box.

I've also been trying to get haiku to build, but not having much luck. Not that I expect it to work but it can't even get past building the cross compiler. Sigh. Well i'll try with a manual build of gcc rather than the system's and see how that goes. I couldn't register with the haiku site to ask questions since it wanted a 'captcha' that it decided not to display, but they've been making site changes so I should probably keep checking.

Apart from Angstrom's weirdness, linux and the distributions around it are starting to give me the shits somewhat. Progress just seems to be bigger and slower like every other system. My laptop's are running slower now I `upgraded' their distributions. I don't think the 'open saucer' crowd's motto of 'more eyes make bugs shallower' really sums it up, I think 'enough eyes make even the worst software work'.

And i've been reading various ARM & OMAP manuals on and off. There's a great deal of complexity in that little chip. My foot's pretty ok but now I have a sore shoulder. Too much hunched over computers and soldering irons and carrying buckets of dirt.