A big bloody rant
Ok it's time for another rant. I had a list of things to rant
about and I lost it ... so I created another one. I may remember
on the way.
Children or pearl clutching nannikins may want to look away.
In short; i'm getting pretty sick of junk on the internet, junk in
software development, junk in politics. There's the other crap
like anti-vaxxers and climate change denialists - but well that's
more of a collective fuckup that we'll all have to reckon with in
the end.
Alright where to begin. Firefox. Fuck firefox. Every time it
upgrades it seems to break somefucking thing. Now the preferences
have been lobotomised for no obvious reason: rather than a click
you have to scroll forever to find what you want. And the default
scroll using a wheel-mouse is so frustratingly slow it's a total
headfuck. Why can't they just include a proper javascript and
image blocker in the base fucking application rather than a buggy
plugin (oh no, extension!) that breaks every upgrade. And then
when you run it it wants to run you through a fucking tutorial
about all the great shit it does. Yeah i've been using a browser
for 20 years, they're just not that special you know. The pocket.
I don't even know what it is but I know I don't want it. Fancy
url rendering: look it's just a bit of text, I want to see what it
is, you don't need to hide the http or the www or the com. It
doesn't make it any more usable or more readable. Multi process;
which just means heavy swapping. Javascript is just such a stupid
language the last thing anyone should be doing is encouraging more
of it.
cmake, got what a load of junk. After years of failures I finally
am able to actually compile some projects with cmake out of the
box. But it's still usually fucked and the only solution seems to
be to go edit the usually non-documented application-specific
configuration files.
C++ is such a shit language. Templates are joke. You
have<to::tell::the::compilerExactlyWhat>youMean.EveryTime(YouUseIt(X)).
And it's still uses gobs of memory and runs like a total pig and gives overly verbose and low-information error dumps.
Everyone uses templates because it inlining absolutely everything
makes the fastest and most efficient code. Actually, no, it
really doesn't. Over-inlining starves registers (particularly on
the terminally register-starved x86 platform), blows out the L1
cache, and wastes memory and disk space. Sure you can use it
judiciously but but the time you've done that you may as well have
just use something else.
Python. Spaces are important. Oh no, actually the number of
white-space characters are important. I haven't got time to wait
for the molasses of a cursor to
step-one-space-at-a-fuck-ing-time-to-get-to-where-the-action-is.
I saw a comment on reddit yesterday about how python is so cool
because of the indenting. It really isn't. With C or any other
sane language you can type whatever you want and the compiler can
work it out. The bit that python is missing is you can then
automatically format it in various more readable ways that show
exactly how the compiler will treat it. With python you're stuck
with what you see. Add a block? Now you how to indent everything
else and make sure you do it right, now just add a couple of
braces and hit re-fromat. Ugh.
This isn't even the real problem with python. I recently had to
try to compile pytorch, a 'lightweight' (ROFLMAO what the fuck?)
artificial neural network engine and the abomonation that was once
caffe. The webpage says use anaconda. Ok 4-fucking-gigabytes
later I get something that doesn't work; this version isn't
compiled with the features I need (probably because of fucking
software patents and ffmpeg). So I spent a couple of days trying
to get it to build from source (actually I tried that first but
had to go back to it). Fighting with cmake and whatnot I finally
get it compiled. But then the python doesn't run. Just a
meaningless backtrace (which is pretty much my experience with
anything python). Ok so some package isn't there. Ok so ... one
uses pip or whatever to install it like you might with apt or yum.
Except it isn't really like apt or yum at all. It isn't even like
slackpkg. All it does is download a tar and drop in on the
filesystem. Dependencies? What are they? Versions? Hah, surely
you jest! It's just a bunch of files on the system.
So you run the script again. Again it breaks. Again you install
another package. Then lmdb wont even bloody install because
setup.py is broken. The internets claim this is a pip bug but
then you fuck up and end up with 3 copies of pip installed (/usr,
/usr/local, ~/.local) ... and well you fuck it all off and start
again. Eventually it turns out lmdb is a compile option hidden
somewhere inside the cmake scripts of the fucking custom OpenCV
build you had to compile when you started on this journey a couple
of days ago.
Javascript. I mean I can't even manage to encompass the size of
the fuckup of this conceptually let alone put it into words.
Web page feedback. You go to AMD or wherever to look up some
hardware or software details and before the front page has even
loaded the page goes white and an overlay asks you to provide
feedback for your 'web' experience. Here's some fucking feedback:
/"\
|\./|
| |
| |
|>~<|
| |
/'\| |/'\..
/~\| | | | \
| =[@]= | | \
| | | | | \
| ~ ~ ~ ~ |` )
| /
\ /
\ /
\ _____ /
|--//''`\--|
| (( +==)) |
|--\_|_//--|
Jesus, even that was hard to find: nobody knows what the fuck
ASCII is anymore. It's all windows-cpfuck-you-too or some other
junk.
Highly related is a web-site that keeps asking you to subscribe.
Or keeps giving you 'helpful hints' on how to use it. You know
what, I can read, if it's not obvious it doesn't fucking matter. I
don't need to be stepped in turn through every single feature that
is so unique it's on every other bloody website. I'm sure you're
very proud of getting it to work, well good for you. Now you just
fucked it up by bugging the fuck out of me.
Or bloody achievements. I just don't bother with any forums
anymore but I am on the FSF member forum although it's pretty dead
and I just don't fit in (story of my life). I mean I get a
happy-face-stamp for writing my first post. Ok thanks I guess?
What am i, in grade fucking one? There's enough trouble with the
world infanticising everything, do we really need it on every
bloody forum we visit? I know i know, it drivers user engagement
and all the metrics. Well fuck the metrics.
Speaking of metrics, why the fuck can't the yanks get with the
programme? Even when they do use them they can't spell it
properly. It's a bloody metre, a meter is something with a dial.
Fuck the freedom units.
Modern user interfaces. aka the hamburger menu. aka the stack of
shit menu. Yeah so apparently a small row or column of easly read
text is too much for the kool kidz these days and it's all hip 2D
iconography and animations and shit. It's a shit style that will
age poorly. Every time a site or application updates menu's go
missing or get moved somewhere else. Jesus fucking stick to
something you don't have to actually change this this just for the
sake of it. You know that people will just get used to whatever
you have; the cognitive overload of having to relean it every time
massively increases the real costs of any 'improvement' you might
make.
Glaring white pages. Well this goes back to the original internet
explorer, M$ windows, M$ word (yes I said itt! There's good solid
reasoning behing it!). I regularly spend 12+ hours a day on a
computer, if I was being blasted with that white shit I would be
blinder than I am already. I don't get headaches for a reason.
Constant updates, rolling releases, fluid apis. It's just too
much. Sometimes things are just done and don't need updating
anymore. But if you do that you dissapear off the google search
results. You're no longer part of the zeitgeist! Heavens! Can't
have that! No you need a fancy logo and an annual rebranding
`event' and need never maintain anything again, just throw out a
new release when you've got enough new buttons to make it look
like it matters! Gotta rush to the end of that sprint!
If you can't keep up, well you've just been out-evolved by the
equivalent of a telephone sanitiser.
There was more i'm sure but that'll do for now.
Oh hang on, one more rather important one in my notes. Back to
firefox. I normally run firefox which is finely tuned to be
actually usable. I don't use many plugins - disable javascript,
disable fonts and colours and that's about it. But I customise
the fuck out of everything else: no fucking animations of
smooth-this-or-fucking-that, middle-click to open a url, etc etc.
I even have a userContent.css that hides a whole of shit on sites
I commonly use (like the banners on stackoverflow and so on). But
I tried a naked profile in order to test out the stylesheets i've
worked on for code and docs. Boy what a fucking joke. All the
reminders about shit I don't want. They even fucked up ctrl-tab
so it goes forward but wont return where you came from (isntead it
opens a menu with very different keyboard navigation) - i mean
what's 25 years of convention for anyway?
But the most alarming one was when I went to download my own
fucking sourcecode. Apparently a tar.gz file is now a dangerous
download.
Yeah fuck off mozilla you bunch of stupid cunts.
Now I just feel like shit, this has been really depressing.
Sad face.
dez 2.0 for DEZ1 is here!
I just released dez 2.0 for the DEZ1 binary delta format.
Consider this a 1.0 release! The format is now frozen.
It was just going to be 1.3 but I noticed that 1.2 came out 4
years ago so I thought 2.0 was in order. I did prepare 1.3 in the
repository but then I just tagged it again with 2.0.
There aren't really many tunables left so I dropped the flags
field and made the OP_SINGLE_SPLIT value tunable. It doesn't make
much difference anyway.
This is now a modular Java 11 project using my common java.make.
I'm also using it to work on adding junit(4) testing to the
java.make system. I may split out java.make into jni.make,
java.make, junit.make, and so on.
tuning it so haven't rolled it back into the base yet.
As usual the Delta-Z home page
has download links and other
snot. code.zedzone.au
houses a browsable copy of the repository; which now uses git.
q
Update: I decided to update the javadocs, as they are.
The interesting page is
the DEZFormat
class which defines and describes the format.
These initial links may not be long-term stable as I will possibly
look at a unified javadoc build across all my java projects. And
I really need to do something about the blinding default
stylesheet. My eyes! My EYES!
Update: I couldn't put up with it so I fixed the
stylesheet. This time I used some Workbench 2.0 colours but
without the 3D borders. Came out quite nice. I filled out the
api documents somewhat as well.
Big Easter Software Bundle!
After basically spending the entirety of Easter (a 4 day holiday
here in Australia) plus half of today I've packaged up and
released a big ol bunch of stuff.
NativeZ 1.0 is an updated version
of CObject which is
portable and modular. I've moved my other projects to use it
rather than their own copies which had evolved and diverged over
time.
jjmpeg 4.0 is updated to use
NativeZ as described on the previous post. There are also a bunch
of bug fixes and small improvements. It now includes a module for
some demos.
zcl 1.0 has been converted to a
modular project and mostly to using NativeZ apis. The tools
package has moved to a demo module.
They are all now capable of cross platform builds and they all
generate platform specific .jmod files as one of the outputs. To
do this they all use a new common java.make which is
a modular and updated version
of an earlier java.make
as covered in the last post (and I made further improvements).
Apart from modules it supports parallel builds, generated java
sources, native code libraries, and cross platform builds using a
very simple configuration mechanism.
They're also a bit smaller due to compression used for some
initialisation tables.
Another project I have gone quite far on is an incremental java
compiler. By scanning the .class files and .java timestamps i'm
able to generate complete and accurate lists of both files that
need to be rebuilt and files that need to be deleted across a
multi-module source compilation. I have all the parts and I think
they should work but I need to tie them together into a usable
system, and then extensive testing.
Sigh, but I need a holiday. I might take most of the week off - like
pretty much everyone else is anyway.
Development Ongoing
I've spent all of Easter so far just hacking away at a bunch of
stuff. Today looks nice so I think I will try to get out of the
house, but who knows. My sleep apnoea has been pretty terrible
lately and when I foget to use my MAD (mandicular advancement
device) i've been wiped out all day, which has been most of this
week. Sigh.
I've primarily been working toward a new modular and portable
release of jjmpeg and zcl, but i've been distracted along the way
and also had some difficulty just making basic decisions.
java.make
I spent way too long on trying to perfect the build system and
things like where intermediate files are stored. There are still
trade-offs that might mean another change, sigh.
But it does work pretty well. The main makefile isn't
particularly complicated even if some of the macro quoting makes
it messy to read. The driver makefiles are trivial - like a
simpler version of automake.
It works by the master makefile including config.make, defining
some variables, and then including java.make.
This for example is the complete Makefile for nativez.
dist_VERSION=1.0
dist_NAME=nativez
dist_EXTRA=README \
build.xml \
nbproject/build-impl.xml \
nbproject/genfiles.properties \
nbproject/project.properties \
nbproject/project.xml
include config.make
java_MODULES=notzed.nativez
include java.make
It's a trivial case so it has a trivial makefile. I added a make
dist target to build up an archive of the source so that meta-data
is really most of the file.
The makefile also builds the native libraries and that is covered
by another makefile
fragment src/<module>/jni/jni.make which is
automatically included if detected.
This is likewise quite trivial for such a trivial example.
notzed.nativez_JNI_LIBRARIES = nativez
nativez_SOURCES = nativez-jni.c nativez-$(TARGET:-amd64=).c
nativez_CFLAGS = -Os -Wmissing-prototypes
nativez_HEADERS = nativez.h
This includes the main .c file and either nativez-windows.c or
nativez-linux.c depending on the current target.
So make does this:
- Compiles all .java under src/notzed.nativez/classes to .class files.
- Generates all native binding headers (from javac -h).
- Automagically creates .d dependency files for all .o files.
- Compiles all .c into .o objects.
- Copies nativez.h file into a staging area for the .jmod.
- Links the .o files into libnativez.so or nativez.dll depending on the target.
- Copies all the .class files into a modular .jar file.
- Copies all the .class files, the .so file, the .h files and license files into a .jmod file.
And only does each step if necessary. For example step 2 doesn't
just take the output of javac which would force every .c file to
be recompiled any time any .java file changes regardless of what
it included. So I use install -C to only copy
changed headers from a staging area. I also use some tricks to
avoid building all the .c dependencies (which requires first
compiling all the java) when you do make clean or make dist.
I'm still debating whether i blow away some of the javac output
directories to ensure an absolutely correct build, or let that be a
separate step and just rely on using the per-module compile flag
which is much faster if less accurate.
I'm also still debating whether it builds the .jar and .jmod files
by default or as a seprate target. For these trivial projects it
doesn't really take any time but if it did it could become a pain
as the code can be executed without either.
jjmpeg is a somewhat more complex case which includes
generated java sources, external dependencies, and (now) generated
c sources and it isn't much more complex to define.
Initially I had everything in the same project but decided to
split it out. The former did allow me to hone down the details on
the build system though and i'm pretty happy with it now; it's
very accurate, simple, flexible, and easy to use.
nativez
I mostly just moved over to the new build system (trivial) and
added a few new functions.
However due to some of the latest in jjmpeg I will probably
revisit some of the underlying ideas here.
zcl
I converted this into a modular project which mostly meant moving
some stuff around. I also changed it to use NativeZ as it's
GC/binding mechanism and to use nativez utility functions where
appropriate. Being modular lets me split of auxilliary demo code
more cleanly so i've done that too.
zcl is all pretty much implemented in a single large (5KLOC) file.
I actually spent a couple of hours splitting it apart into
multiple files because I thought it was getting a bit unwieldy
(and modern emacs struggles with the size of the file
unfortunately). But then I changed my mind and threw it all away.
zcl uses a generated pointer table and typedef casts to manage the
OpenCL function linkage and while it works i'm thinking of
revisiting it again.
I haven't used OpenCL for a while and don't need it at work at the
moment and I can never really think of anything useful to do with
it. So it hasn't been tested much.
jjmpeg
I already had a modular jjmpeg so this wasn't a big change, until
I decided to keep going. First I moved to the new build system.
I already had moved it to NativeZ but there were some lingering
functions that I hadn't cleaned up.
Then I wrote some demos. A simple example to read video frames.
One that saves frames to image files via Java2D functions.
Another that displays video frames in a resizable window at about
the right speed (i.e. a mute video player). And then a music
player, which lets you adjust output parameters.
The music player needed some more bindings and converted enums so
I added that. And it needed a way to get to the samples in a
format I could play.
First a bit of background. In earlier versions of jjmpeg I just
exposed the data pointer array of an AVFrame as a ByteBuffer and
provided entry points into libswscale to allow one to convert
frame data formats. This is very messy and error prone and really
quite pants to use. So copying the ideas from
the PixelReader
from JavaFX I've created a similar AVPixelReader which is flexible
enough to implement MOST of the PixelReader api and hide all the
details of libswscale. The current implementation works by
creating an AVPixelReader targeting the size you want and then you
can write it various Java targets in whatever pixel format you
want. I've got some nice helpers that wrap this to write to some
Java2D BufferedImage (via the DataBuffer) or JavaFX WritableImage
(via a PixelReader).
So I decided to do the same thing with libswresample and samples.
But I don't need to support other pixel readers/writers and only
just dump out arrays of bytes so I made a much simpler interface.
Although libswresample can stream I basically enforced that a
whole buffer needs to be converted at a go otherwise the logic of
tracking the frame gets a bit messy.
At this point I was cleaning things up and thinking about a
release but got side-tracked again ...
jjmpeg - bindings
jjmpeg has a couple of different ways that things are
semi-automated. All simple parameter accessors are created using
C macros. Java JNI handles are loaded via a table which describes
the type, name, and signature. Library functions from
libavcodec.so et al are defined by a table and loaded by a simple
platform-specific function. The tables are all const'd, the
variables are mostly static, so everything should go in read-only
memory and be fairly efficient to utilise.
The tables are handrolled. As are the variables which hold the
function pointers - which means i'm copying them directly from
header files and turning them into variables. This is error prone
and provides no compile time safety confirming that the function
matches the prototype.
And with 64-bit pointers the tables themselves are quite large.
Each entry has pointers to strings, pointers to pointers, and some
flags.
So I thought i'd revisit that.
With a simple code generator I create a struct with all function
variables in it, extracted from the headers at compile time. I
have a compact descriptor - essentially a stream of bytes - to
encode enough information in order to fill out said structure.
It's a mess of shit perl but it works.
The resolution of java objects for jni use is also likewise table
driven, so I did the same for that. It makes the assumption that
jclass, jmethodID, and jfieldID are all pointer sized but that
seems a pretty safe bet.
In short I shaved off about 20% of the size of libjjmpeg.so doing
this.
This is an example of the driver file which defines things of interest.
header avutil libavutil/opt.h {
av_free
av_dict_free
av_set_options_string
av_opt_serialize
av_opt_copy
av_opt_next
av_opt_find
av_opt_set
av_opt_get
...
}
java AVOption au/notzed/jjmpeg/AVOptions$AVOption {
<init>, (JLjava/lang/String;Ljava/lang/String;IILjava/lang/String;)V
p, J
}
I'm not particularly enamoured of the Java definitions but it
simplifies the generator. Hmm, looking now I can use javap to get
this info so maybe I can simplify this somewhat whilst performing
additional verification checks at the same time.
The generator creates a couple of static anonymous structs and a
descrption string, and as a bonus includes the required header
files. There are also some #defines so I didn't have to change
any of the code that uses them.
/* This file was autogenerated on Monday 22 April 09:35:47 ACST 2019: */
/* src/notzed.jjmpeg/jni/extract-proto.pl src/notzed.jjmpeg/jni/extract-proto.pl -f /opt/ffmpeg/4.0 src/notzed.jjmpeg/jni/jj-avoptions.def */
#include <libavutil/opt.h>
static struct functable {
/* libavutil */
void (*av_free)(void *ptr);
void (*av_dict_free)(AVDictionary **m);
int (*av_set_options_string)(void *ctx, const char *opts, const char *key_val_sep, const char *pairs_sep);
int (*av_opt_serialize)(void *obj, int opt_flags, int flags, char **buffer, const char key_val_sep, const char pairs_sep);
int (*av_opt_copy)(void *dest, const void *src);
const AVOption *(*av_opt_next)(const void *obj, const AVOption *prev);
const AVOption *(*av_opt_find)(void *obj, const char *name, const char *unit, int opt_flags, int search_flags);
int (*av_opt_set)(void *obj, const char *name, const char *val, int search_flags);
int (*av_opt_get)(void *obj, const char *name, int search_flags, uint8_t **out_val);
} fn;
static const char *fn_names =
"#avutil\0"
"av_free\0"
"av_dict_free\0"
"av_set_options_string\0"
"av_opt_serialize\0"
"av_opt_copy\0"
"av_opt_next\0"
"av_opt_find\0"
"av_opt_set\0"
"av_opt_get\0"
;
static struct {
// au/notzed/jjmpeg/AVOptions$AVOption
jclass AVOption_classid;
jmethodID AVOption_new_jlliil;
jfieldID AVOption_p;
} java;
#define AVOption_classid java.AVOption_classid
#define AVOption_new_jlliil java.AVOption_new_jlliil
#define AVOption_p java.AVOption_p
static const char *java_names =
"#au/notzed/jjmpeg/AVOptions$AVOption\0"
".<init>\0(JLjava/lang/String;Ljava/lang/String;IILjava/lang/String;)V\0"
",p\0J\0"
;
Rather than have a single global table I have one per file
(roughly on per class). This leads to a small amount of
duplication but keeps everything simple and isolated and also aids
incremental compilation. By having each table static to the
compilation unit also allows the optimiser to remove any address
calculation - it just goes to the linker via the .bss section.
I'm still thinking of other ways to reduce the binding size and/or
overhead. For example the accessors are pretty small ... but
there's many of them. The problem is if I use tables I need to
keep them in sync between Java (e.g. field ids), or if I use
strings then I need table lookups. ffmpeg seems to be moving
toward the 'libavutil/opt.h' mechanism for accessing everything
anyway (probably because of python influence?) so perhaps I should
just use that or at least evaluate it.
But I will first have to move these changes to nativez and perhaps
zcl.
Other
I fixed a few bugs along the way. I wasn't creating a few objects
correctly so they weren't being tracked by the recovery process -
mainly a holdout from earlier implementations which tracked
objects differently. I wasn't freeing an AVFormatContext properly
- these can be created in two different ways and need to be freed
differently. Sigh, even the free functions take different
argumements (some of the avcodec api is such a mess). Anyway I
use the opaque pointer to track this so I could do it properly.
There are probably other leaks but it's rather much harder to test
than a standalone C program like playerz.
I also looked very briefly at writing a Vulkan binding as well.
Hmm, i'm not sure on this one, it's a pretty big API although as I
learn more it will probably shrink in apparent size. But
regardless it's a bit more than a weekend project that OpenCL was.
It will probably need a code generator to make it managable, at
least for certain objects like the construction parameters. I'm
mostly concerned with the overheads so will be looking into
different mechanisms.
And these might trigger a reworking of everything else along the
way.
Still, i'm not sure yet on it, I don't really have a need for it
or know it well enough to do a good job either.
