Maker? WTF is a maker?

By happenstance - as I just don't bother reading the news much these days much less specific papers - I came across this interesting article about the 'hacker culture' this morning in the age.

First i've ever heard hackers wanting to be known as 'makers'. Sounds a bit faffy and overly-pc if you ask me. I had seen 'maker' sites, but I thought that was just a brand-name, and not a movement ...

I have no problem calling myself a hacker, but then again it's not like i'm talking to anyone but myself ;-)

George finishes with the question: could Australia develop it's own 'Silicon Valley'? And I think the answer is simple: No, no it couldn't.

Education

First, there is the education system. Computers are now just mechanisms for running software from Microsoft, Apple, Adobe, or other large US companies. One doesn't start with an 'introduction to computers' covering 'word processing, spreadsheeds, and programming', all you get is: Microsoft Excel, Microsoft Word. If one progresses ever past that point it is merely to 'Photoshop', or maybe avoiding stalkers on the intertubes.

In primary school we had one Commodore 64 in the whole school and as a `bonus' for finishing class work early we sometimes got to type in computer programmes (often from Compute!, which used it's own checksummed number entry system - so it was pages of hexadecimal numbers). High school had a pool of crappy Apple ]['s, but even then word-processing, 'desktop publishing' and spread-sheets were only a small part of the course, which ended with programming. And this was in year 8.

Gone are the days when education was about educating people, now it's just low-level training for stuff anyone who will ever work in the field has already done before they start school: using a mouse, using a crappy 'word processor' (which sadly, is still just as crap as they were back then: i.e. a fancy type-writer), and how to run an internet search.

Hackers of course have always gone beyond the education system, but if you're repeatedly told that computer software systems are locked up private applications which no user serviceable parts, it has to have some impact.

Also this notion of training for specific products that 'industry wants' is utter nonsense. First, you will never cover enough in high school (or even TAFE) to avoid on-the-job training specific to the business needs. And second you are always much better off being exposed to as many different systems as possible. The human brain is an incredible organ, and even more-so during developmental years: like any adaptive system, the more you expose it to, the stronger and more robust it becomes. Intentionally crippling its development by limiting it's exposure to alternatives borders on criminal.

Complexity

Systems are so much more complex than they were back when Silicon Valley got it's start. Not only to write code for, but expectations are much higher.

Back in the day I could spend a whole weekend typing in a bit of BASIC to get an animated lo-res horse-sprite running across the screen and I was impressed enough just with that. And even then; the actual code required just wasn't that much and you were talking directly to well designed simple hardware. You literally could design a whole computer itself on the back of a napkin, let alone the software.

Today you need a bulky IDE to get very far, to learn some gigantic and usually poorly documented toolkit, which goes through another toolkit, through an operating system, then a driver, and if you're lucky to the screen. And you don't feel satisfied until you've come up with the next billionaire-making 'killer app'. And to make matters worse the hardware is devilishly complex; so complex it never really gets working properly and needs to be hidden behind layers of drivers and high level apis - which it necessarily must be for an operating system to function - but makes it that much more challenging to understand what is going on (which as a hacker: you need to know sometimes).

The barrier to entry is huge, and for all intents beyond the reach of most individuals. This leads them to believe there's no point in trying, and creating software is exclusively for the great wizards who doth do such things (and must be paid accordingly).

This complexity affects every level: so beyond the individual then you have large complex systems which take many man-years to create.

Silicon Valley also had a unique combination of hardware and software mixed together allowing both to leap ahead in bounds together - something which would only be possible in China or Taiwan at the moment. But they're not interested in the software or the hardware design, just in making throw-away junk cheaper than everyone else.

Fucked up Legal system

And then we get to patents and so on. It is now impossible for anyone to create any piece of useful hardware or software without stomping on someone's stupid patent in the USA. Which means effectively everywhere in the world as all of our local politicians have kowtowed to the almighty US dollar and wont dare to argue for their or our own interests.

It would simply be impossible for Silicon Valley to become Silicon Valley in today's legal environment. The whole notion was based on a solid foundation of sharing of ideas and people, and a free flow of money. This latter point alone rules out Australia ever being more than a bit-player in the game, things just aren't done that way here.

The USA wouldn't let us

So this is related to the previous point, but the legal system is just one tool they use for this.

The USA is the dominant military, political, cultural, and financial force of the day. They do not let anything get in the way of that dominance. Anyone who thinks otherwise just hasn't been paying attention.

e.g. The H1B visa programme in the USA isn't about US companies not being able to get enough talent at home; it's about preventing any of the talent from being used by competitors.

If Australia ever had a Silicon Valley it would only be at the divine blessing of the USA. It would almost certainly be in the form of a 'technological partnership' with an IBM, or a university. And these are just talent syphoning systems.

Pretty unlikely for that matter - just not enough people, too high wages, that's why they've set-up shop in India and elsewhere.

The tyranny of distance, money, etc

Australia is just too big and too thinly populated to create a high enough density of a specific type of people needed for such a thing to occur. Not to mention too parochial - although a lot of people would travel interstate for work, plenty wouldn't.

The NBN has the potential to remove the distance factor, but even if that ever gets done properly - it just doesn't create enough connectedness for it to work effectively. If people aren't excitedly talking about their latest idea in the pub with competitors the ideas just aren't sharing quickly enough (this will be silicon valley's downfall - already a lot of the exiting stuff is happening elsewhere).

And you need all that excess money to throw at random ideas as well. Which basically means you have some source of free money - imperialism, oil, the world's trading currency, or somesuch.

We will never have that here. And even the great source of free money here at the moment - mining - only seems to produce overweight greedy arseholes (an unusually number of whom are remarkably fugly to boot) who have no interests beyond their own fat arses.

The World Wide Web

So M$ are up to their old tricks again and restricting competitors from accessing their operating systems properly. Nothing ever changes apart from the date. Does it really matter much anyway? (who in their right mind is even going to use that shit to start with?). Mozilla are still pushing their it's-really-an-operating-system angle, but is that really ideal for me as a customer? Or even as a developer for that matter. One only has to look at any application on a tablet and compare it to it's browser counterpart. Even running on a monster desktop workstation the tablet usually beats the pants of it. e.g. google maps or youtube. For all it's "lowly" specifications (well, they aren't really), the tablet i've been using shits all over anything i've ever seen in a web browser, HTML5, or Flash, on much much faster machines. I would really like to see browsers eschew all that heavy-client crap which has turned them into bloated power wasters, and return to a world of delivering information in an open and linked way. This is something they actually do quite well, and then they can leave the heavier stuff to native applications which always do a better job. And with the range of devices exploding - from workstations to tv's to pocket computers - doing anything more than text with a few pictures, the odd form, is only going to become more difficult. The right tool for the right job, and all that.

Abject failure

I spent an inordinate amount of time last night trying to lick the decoder stability issues.

I ported the jjmpegdemo MediaPlayer over to android, and re-arranged it so it was shipping AVPacket's around instead of AVFrames and decoding on a separate thread (this is how ffplay does it). I changed the rendering to copy to a texture synchronously with the decoding (i figured sharing EGL context work can wait). I used multiple textures to implement some buffering.

So when I finally got it running (after wasting a good hour on a silly place-holder mistake when i started), it still suffers from the same problem. Along the way netbeans decided it had too many projects open, and the debug cycle time on the tablet seemed to get longer and longer.

Ho hum.

This morning I poked at using shared contexts, and although it seems to use less CPU time, it still crashes and it's doing something weird with the frame rendering order as well.

So yeah, might have to sit on this for a bit and brood.

Update: As one does ... I persevered. Current guess is that it's something to do with the texture load: if I isolate that out things seem to run fine (but with no repeatability this is hard to judge). More perseverance required.

Speaker-busting screech

So yeah, don't forget to set your endian-ness on your direct ByteBuffers when you're playing with sound files. Poor speaker. This morning I got sound going with jjmpeg/android. This was mostly fixing up the way I was using AVPacket's - I removed the the AVAudioPacket class and added the functionality to AVPacket. I need to look at the new audio_decode_4 api, as that might simplify the usage anyway and remove the need for the special case. I also hit some surround-sound files - i've just never played with audio much (don't even have speakers on my pc) - so have some more api to think about binding. Shouldn't be too hard to fit this into the video player, AudioTrack reports it's position which is enough.

jjmpeg/Android

So yesterday I checked in the work I did last week for the android jjmpeg port. Not heavily tested, but it appears to work.

Then I spent far too long poking around trying to find out about the EGL OES_external_images extension, but finally came to the conclusion that it just isn't exposed publicly (yet?) so it's way more effort than it's worth to me to get it going.

So I suppose it's just glTexSubImage2D for me then.

GLES

I had a long work week this week and although I had intended to get a couple more things out of the way this morning I let it slide, and eventually got sucked in to playing with jjmpeg. I rigged up a version using GL, but still for now performing CPU-side colour conversion. I at least added a separate thread to do the colour conversion, and hooked it all up so that nothing needs to wait for anything else unless absolutely necessary.

The display stuff is a little bit faster - but nothing to worry about TBH. The colour conversion is only about 20% of the time, so it isn't a real big difference either.

I also tried with/without VFP: which made stuff all difference too, barely 5%. I started with 32-bit RGB textures simply because I had that already, and changed those to RGB565 which made another small difference. But all these small differences just weren't adding up to anything but a lot of small differences.

So anyway ... I went and looked at some of the options for skipping frames, multi-threaded decoding and what have you, and I noticed that threading wasn't enabled in the build.

Cut a long story short - beware of copies of copies of scripts one finds on the net: my compile script was basically shit. I also turned on NEON this time ...

Success!

I'm not sure if it's just the NEON, or the threads - or simply compiling it properly - but boy what a difference.

(actually it's all those not-so-little-bits now adding up, even single-threaded it's now much faster, in performance mode 2 threads can nearly handle 720p).

So it's now decoding 720p MP4 fine (taken from the on-board camera), even in 'balanced power saving' mode. Before it was struggling with this, under 10fps. And now the colour conversion is more like 50% of the time, so I will have to investigate using GLES for this since it should be a lot better at it.

Anyway, i'm quite chuffed at this now - I was starting to think it was pretty much pointless apart from perhaps encoding or more control over decoding. But this level of performance opens up a lot of possibilities. I'm also still only using ffmpeg 0.10.0 release, so there might be more there now too.

Update: So I kept going and added the GL colour conversion. Quite a bit better and 720p is fine with 2 threads, but not able to handle 1080P from the built-in camera.

Android Camera & Video, OpenGL

Ahh, somehow I ended up hacking most of the Saturday rather than taking advantage of the relatively pleasant weather to do something better. Addicted I guess. If the weather holds today ... maybe i'll get out.

I was just going to have a quick look at android camera operation, but one thing lead to another ...

After I got a basic camera view working, I wanted to work out how to present it in opengl. So I got pretty side-tracked trying to work out some opengles2 stuff - the examples and suggestions on the net were usually small fragments or for opengl1 - until I found the android sdk sample they all came from which had everything I needed (GL2CameraEyes or somesuch, my workstation is off now). GLES2 is a bit of a pain and requires a lot of scaffolding to get started, but I guess it is what it is.

So yeah, camera to texture, texture to display, etc.

Also checked out taking photos, and recording videos - although the latter requires a preview view, and wouldn't work with a GL 'preview' window so I gave up on that in this context.

The basic objects are a bit bare, but they provide enough functionality to be usable, and are generally fairly simple to use once you know how.

"Broadcast Quality"

One thing that is a bit disappointing is the frame rate achievable. Even with a simple GL render loop it doesn't match the frame-rate properly and you get skips and drops. It's almost like the screen and GL are refreshing at different rates. I tried a dirty-refresh-on-camera-frame, as well as update-continuously-refresh-texture-when-camera-frame, but both ways were simply not smooth.

For fucks sake even a fucking Commodore 64 could generate video-rate-smooth graphics.

(Even something like GL gears, a simple demo which should easily manage a solid and smooth 60fps, is not properly smooth, although it might be the internal animation calculations that are out).

Because of this, and the stupid choice to use a 60hz panel (my laptops run at 50hz for this very reason), one of the few things that might actually be useful on a tablet - watching recorded tv shows or movies - is actually a pretty crappy experience (if you can even get it to work in the first place). Movies on a 50hz display running at 25fps look a LOT LOT better than telecined crap on a 60hz one.

People are so used to totally woeful 'pc graphics' these days nobody even notices or cares (horrah: at least there's no bloody tearing!).

Even tv stations no longer seem to care about 'broadcast quality' any more - I guess the old guard engineers who knew the difference are being replaced by kids who spent more time on youtube than watching broadcast tv). Sport looks like it's been shot with a pocket camera and run through a single-pass of ffmpeg encoding at middling settings - severely blocky from lack of bandwidth and just not very good. And we're often getting American version of movies now - i.e. which have been fucked up by being telecined to 60fps to start with - and here they are just blasted out at 50fps using simple nearest-frame or sometimes average frame 5+6 output both of which looks totally totally shithouse (9, 7, and 10, i'm looking at you). And even when they do have PAL source, sometimes the interlaced frames are flattened to 25fps(AND THEN SCALED!!!), and encoded at 25fps progressive: which is astoundingly bad.

So why did I buy that giant TV for again? Well at least I can read the current score or the news ticker from the couch ...

JJPlayer

Although I'm pretty knackered after a week of hacking on android user interface and REST stuff, a few things fell together near the end of today so I felt like I had a bit more energy to poke around jjmpeg on android.

I converted most of the rest of the binding to use integers as a pointer rather than bytebuffer's and after a bit of debugging managed to read a frame in ok. I'm still holding out on the AVIOContext stuff as it's a bit of a mess.

Using integers and looking up the value as a member variable on the lowest concrete class works pretty well. In some cases it even simplifies the binding as for example I don't need special 'object holder' classes for in/out parameters. I got sick of trying to get the horrid mess of a binding generator to work for all cases and hand-coded these in/out functions as well: it's only 3 calls and a lot less work just doing it by hand than making the generator handle all cases.

Performance doesn't seem terribly hot, but then i'm not au fait on the myriad of ARM compiler options in gcc. And in any event it looks like my build script had a bug and only built using the defaults anyway (ok, so i fixed that whilst typing this and it's a bit faster, but well who cares anyway, it's not the goal to beat the hardware decoder at mp4).

I noticed that I branched for 0.10 before i'd finished off some of the api, so I will have to back-port a bunch of stuff to make it more usable.

I'm also considering simplifying the object model significantly: at the moment 4 (6, if i want transparent 64 bit/32 bit support) classes go together to make 2 objects for each C object; which allows for automatic garbage collection so make the api more java-friendly. It works quite well at least on the oracle and openjdk jvm's, but adds a bit of extra crap to the binding. I could simplify it down to 2 (4) classes and 1 object if I force the developer to manage memory explicitly. But what I have works, so I am in no rush on that one. One doesn't need to create very many objects to decode a video, so it's hardly a big overhead. And although freeing them explicitly isn't that much of an effort either: it's still nice to have it work automatically.

My local code-base is a bit of a mess - I couldn't get netbeans to open the android project as an android project properly until I created a new directory, created another blank project with the android tool, and then copied the source around till it worked. So it might be a bit of hassle copying this all back to a versioned tree and checking it in.

android hacking, jjmpeg

So winter has sort of started here - although we had a it of sun today with a bit of heat in it. It tends to stick from about ANZAC day - the 25th, and we're only a few days early.

Which means: I don't have much to do apart from cook and hack. Still a bit tired from the previous week so I didn't get into much but I had enough time between drinks to have a look.

RTFM

Yesterday I spent a few hours playing around with the Manpages source. I seemed to get a lot done but after looking at what I came up with, it didn't seem to amount to much ... just a bit of GUI re-arrangement to make it take advantage of a tablet. I don't need man pages on a tablet, but I am thinking of how to do a nice interface for accessing documentation as it does seem something the tablet form-factor might actually be useful for. Starting with Manpages just saved me a bit of faffing about.

I first started playing around with the search function: making it use a searchview, and changing it to filter the list of man-pages rather than coming up with a redundant 'shortlist' completion list. And voice search, although it really just doesn't work well for unix commands (grep == brett, great, grit, etc). Probably the stuff that took me the most time was working out how to get the search box to work nicely - getting rid of the onscreen keyboard when it isn't needed and so on. But eventually I have that behaving quite nicely.

The filter stuff isn't very well documented, but it wasn't too difficult to work out. The built-in list filtering has a silly stylistic choice of overlaying the list with a very large label of the filtering string, but it was only a couple of lines of code to remove it by doing it manually. I implemented it using a tree, so an interactive prefix search runs in real-time with no trouble at all.

I'm still only experimenting but if it goes somewhere this might end up another project to divide my time (an android version of 'ReaderZ'?).

jjmpeg

Today I thought i'd instead have a poke at jjmpeg. I'd already had FFmpeg building from source using the NDK, but not gone anywhere with the jjmpeg stuff.

I'm in two minds over this - I don't really need it, and am not particularly interested in writing a video player, although the free ones on the market are pretty crap (and they expect me to pay for ffmpeg? I don't think so tim ...). But it might be useful for work and it's a good opportunity for some self education on mobile hardware, opengles, and so on.

So i've branched jjmpeg, and started work on an android specific port: things are different enough that at the moment that branch will only build the android code. For example I am linking directly to ffmpeg's libraries so I've removed all the dynamic loader stuff.

But anyway, don't expect rapid progress.

GPL3

I've also decided the whole library will be GPL3 - for a couple of reasons.

First, since I have to distribute FFmpeg in the binary, I have to distribute the source now too. If I have to deal with that crap I may as well ask for reciprocation.

Secondly, many android developers seem to be utter leech-tards, and don't mind wrapping anything they can get hold of in a new name and trying to make money off it. I don't really need to contribute to that in any way - after-all I didn't do all the hard work which makes jjmpeg work.

And finally, and related to the previous: the LGPL requires that all the LGPL code be replaceable by the user. But with android this is quite a difficult process which prejudices the user. It's not like GPL will make building with the NDK any easier; but at least it means those capable of doing so end up with more than just a new makefile for ffmpeg which is about all the LGPL requires.