Java Scripting?

Well to "relax" I did some reading up on JavaScript from Java ... and to be honest it's "bloody awesome". I've done plugin languages and extensions before in C and Java so i'm no stranger to the binding issue, but this makes it almost "zero work" if you already have well defined objects - which should be a goal anyway.

Anyway, it's tidy and 'included', which is nice - no wonder Oracle is trying to modularise Java, it's getting big.

As one might know from reading this blog i'm not a huge fan of writing applications in JavaScript ... but it does have it's uses and this one seems like a perfect fit.

One of the last major components of the original Dusk I haven't already gutted is the scripting language - but it's now on the chopping block too. One reason is the code is pretty opaque and difficult to maintain, another is that the 'scripts' are the whole game, the finally it is a reasonable enough implementation (compact pre-compiled code) and still works. But the thought of extending it to support planned new features is a bit unpleasant.

I think breaking the whole existing part-game is worth it in the long run. And efficiency isn't much worth if it isn't usable - and computers are somewhat faster than they were 15 years ago.

It's also what I would call a "marketing" or "management" feature: JavaScript has a huge following and a ton of resources, whereas trying to learn the arcane and obtuse syntax of Dusk's script engine is not going to be high on anyone's list of things to do before they die.

I played with it a bit more and realised you can basically do everything you could do in Java from any JavaScript script, including file and network i/o ... so it looks like i'll have to delve into SecurityManager. I've always avoided it as the documentation is obtuse and bereft of any useful examples - and we all know how complexity is such a winner when security is at issue. I'll also have to work out using it in a multi-threaded context, if that's possible.

Well I suppose life wasn't meant to be easy.

Time passes.

Ahh on threading it looks like everything has to be from the same doPrivileged() call. I chose to create a threadpool and submit jobs to that, and use Future to get the results if required. Seems to be a reasonable compromise. I've implemented a watchdog timer too, but because the ScriptEngine has no way of interrupting or stopping execution (should be something on the Context?) I'm forced to use the dangerous "Thread.stop()". Well what can you do eh? I guess it's more than a "few lines of code", but on the other hand I have a nice easy to use - and probably safe - script mechanism.

This deserves more of a write-up but I was supposed to have an early minute today so it will have to wait.

Update: I had more of a look at scripting including the SecurityManager aspects and documenting the current script system and how it's used.

Unfortunately the sandboxing turns out to be much more involved than I wanted. I can fairly easily do a basic sandboxing such as preventing disk and network i/o outside of prescribed locations (no thanks to the documentation; it's unreadable shit), but beyond that things get quite tricky. One reason is the rhino engine itself needs to do things like reflection in order to execute any script.

I experimented with some nasty stuff like scanning the stack-trace for 'legal' accesses, but I dunno, it seems complex enough that I wouldn't trust it. So for the moment my initial solution will stick with the simple high-level solution of restricting file and network access and put some trust that the players that reach a high enough level to create scrips that will behave themselves - as one did in the past.

I've still yet to come up with the solution i want for how the scripts should be structured. I've thought about everything from a simple global-context scripts to having the entire mob persistent state exist within local variables on objects within the JavaScript engine and all scripts merged into a single pre-compiled programme. Something in-between will probably be the result - as usual - but I need to implement at least a couple of ideas and see how they work in practice. One desires as much flexibility and simplicity as possible, but you can't maximise both at the same time. I initially thought of simply having global-state inline code, but that doesn't allow the use of 'return'. So now i'm thinking more of defining a function within the code that matches the filename. This also allows pre-compilation. And certain objects such as things or items require a number of related scripts, so it probably makes sense to wrap them in an object and just call methods on that.

The existing script language is pretty weird. Prefix notation for operators and effectively everything is just a function (which parses itself - the syntax is not parse-able without semantic information). Might take a while to grok and convert the scripts - a compiler which converts the scripts probably isn't worth the effort. One detail I hadn't previously been aware of is the number of general player commands that are implemented as scripts - and the complexity thereof. Most of the other scripts are fairly simple or at least straightforward. It's not an absolute necessity to convert all of them or even any of them - but having an existing game working is a good test-bed.

There are also some scripts which don't really do much and simply define global "game-theme" behaviour - yet need to run very often. I will look into whether these can be instead be implemented by Java code and supplied at run-time via a plugin/extension mechanism. This still allows for customised behaviour without recompiling the whole application, together with better space and time performance.

Tall tiles

Had a little hacking session this morning and implemented "tall tiles" into the client. The server doesn't need to know about them although I had to write a converter for Tiled format for the test map I generated.

Tall tiles allow for a great deal more flexibility in design for oblique or isometric projected maps without complicating any of the game logic. Well, ... it seems a pretty obvious feature for anything but a top-down game.

If one looks closely one can see the partially obscured player sprite. Even things like the tree in-front of the path/building/or other trees is completely impossible with the original Dusk engine - without creating separate tiles for every possible background.

As I need to store multiple images and so on I decided to store them in a jar file, with a manifest describing some meta-data about the files therein. In tiled I create a separate set of tiles for each tile size, and the converter creates one packed image for each tile set and stores it in the jar. I intend to use the same mechanism (and jar) for player and mob sprites too, allowing a lot more flexibility there as well, and possible even use tiled to manage those too (allowing for example symbolic names for sprites rather than using numbers which are hard to keep synchronised).

Rendering is straight forward from top to bottom and I render the map then the player. I put all the player labels and highlights above everything else because well it just seemed like the right thing to do.

Server

I decided to change the way tile actions are defined. Rather than have a separate script for each tile by number, I instead have a property file (why write my own parser, again) which defines which script is executed for each tile that needs one. By default tiles cannot be walked or seen through so values only need to be set for open tiles. This makes it a lot easier to manage, and I will probably hard-code some constants like "true" as well. I'm using Tiled to set the properties on the tiles which define which scripts are run, and the converter dumps these to a file the server understands.

Actually last night I also started working on a binary protocol for the server. I did the map update and a couple of others. But then I kind of got hung up on trying to encode some of the more detailed messages and just wasn't feeling the love. Probably a more re-usable approach using simpler "property lists" might be more loveable.

Source?

Some licensing queries I have should be sorted soon, and then i'll put the code somewhere once I've fixed up the files.

Anyway i'm pretty pleased with my progress this week (considering i've never worked on any game code ever before), and this little 'win' is probably a good opportunity to take a break, before I get overly keen and try to tackle multiple maps or layers and sink my last weekend of leave down the toilet!

Tooling and stuff

Did a bit more refactoring and bug fixing this morning in the server. Removed a bunch of Vector types, and improved the code a bit. Found and fixed a concurrency bug.

Now i'm starting to get the codebase under control - and learning enough about it - i'm starting to get more wild ideas and think about other issues.

One is tooling - the bundled editor isn't too good, and there is already a java-based editor for tile maps called Tiled (although it's moved to C++ now - ugh - the original still works ok and doesn't need a pile of dependencies installed). So I had a bit of a play with jaxb and jxc and created an exporter for the current map format to Tiled's native format.

It works ok although bogs down (on this laptop) at this sort of scale - the map is pretty big.

Writing an exporter isn't much more work, although since i've gone this far i'm starting to think of other ideas to include at same time ...

Taller tiles

Taller tile images which sit on the same base square allow for things like trees and forests and posts you can walk behind. i.e. pretty useful for a designer.

I think this should be pretty easy to implement as a first cut, the rendering code only needs to align based on the height of the tiles, and the image storage needs fiddling with. A slight further complication is to avoid pop-in, but the server can send more of the map than is shown.

Taller sprites

This again is fairly simple and solving the problems for the tiles should help solving it for the sprites.

Multiple maps

I think that huge map is pretty unwieldy, 'rooms' are just defined as teleports to other closed sections of the map, and obviously that doesn't scale particularly well. Obvious solution is to have a separate map just big enough for each 'scene', and add the map id as a move/teleport argument. I think this loosely connected graph of separate regions has a more 'muddish' feel to it too.

I think that most of the game logic doesn't really need to change, as most entities will be tied to a given map so it doesn't need to turn into a 3d problem.

Layers

Another idea is to have multiple layers of rendering as a mostly cosmetic feature. I initially thought this would be pretty easy but to do it well will require some extra meta-data.

To keep the problem in two dimensions the entities will either have to be locked to one visual layer which might be a bit limiting, or some extra meta-data provides that information. e.g. a non-visible plane which indicates which layer objects are rendered on. Currently there can be behaviour attached to a type of tile that you're standing on and although this layer-indicator system would also work with that it might be useful to have a separate plane which defines the behaviour - although that is more design work.

But it could certainly be interesting, and allow for some interesting effects like parallax.

Binary protocol

Currently the protocol is a mix of binary (char) and strings. Actually it's a bit of a mis-mash of mis-match - each end is writing to a DataOutputStream, and each end is reading from a StreamReader. Although it works it isn't really supposed to, and having a DataInputStream on the other end would allow binary interactions.

Initially the layer stuff sounded good but the more I thought about it I wasn't so sure. I played with some taller tree tiles in Tiled and I thought it looked pretty nice. For an angled-down view just using a taller image gets you much of what you might want layers for without the extra overhead.

de-awt-ed

``I'll just fix up a few little things'' turned into ``Another full-day of hacking'' on my fork of Dusk.

I finally managed to de-awt everything. Well there are a couple of missing features (text viewer/editor), but the last major window to convert is in:

To make this a bit nicer I think I need to modify the protocol so the server tells me more about the items such as item counts as it does with the inventory command. And there are other possibilities such as graphics, and "detailed description". It should probably also split up equipment and general inventory in some way or another ...

Well at least the layout didn't take any time today even if it looks it.

What took all the time was rewriting another big chunk of the client code map tracking and abstracting away the user interface from the client logic.

For the map code I again abstracted it into a separate structure and provided accessors and indices of various things, although now i'm reflecting on it I don't think some of the stuff I coded is even used since the algorithms that used them are gone. For example there was code to find entities by name - but internally i'm just using unique ids now so having an index by name it isn't required anymore. One major change I did was to just keep track of entities (players, mobs etc) using absolute coordinates rather than trying to keep them forever up-to-date with the current viewing window. Apart from being a great deal easier to use it is obviously more efficient. There were a lot of "x - offset1 + offset2" type things everywhere anything was accessed, and instead I just made the map to be relative to the top-left corner rather than the centre, and track the player location separately.

I also changed entity based actions to use id numbers instead of indexed names. Basically if there is more than one object of a given type on the screen then some messy code gives each one a unique number. The problem is the client and server do this independently, and I couldn't be stuffed finding out how it works - one of the benefits of the map abstraction was killing a pile of ugly code, and it was buggy anyway. Since the server and client already uses unique id's for everything I just added support in the server for doing targeted actions based on id instead. The client can always do the lookup if the command line needs to support it, and I guess I can now clean up that mess in the server too.

And finally the entire user interface and system interaction is hidden behind a re-usable interface, for the obvious sub-java/sub-gnu operating system option.

So far the de-awting seems to have made is more stable too. Although I also fixed some thread issues in the client as the game runs on a separate thread to JavaFX.

Shopfront

Given the day started raining I spent most of it hacking on Dusk again. Coded up a shop-front.

Yeah the styling is naff, but it isn't meant to be better. The main window blurs when it's shown.

I gotta say the layout was a lot messier than I would have thought. VBox and HBox only allow consistent alignment for all items in the stack - which seems pretty pointless to me. I'd have thought a common use-case for a VBox would be a stack of things with different alignments. And I couldn't get GridPane or BorderPane to work very well either, although perhaps I could now with the following knowledge.

Basically to perform arbitrary alignment and expansion of cells within a vbox or hbox, one has to embed a hbox or vbox for each containee, set it to fill, and then set the containee alingment on that. It's just rather clumsy. As for the table view I had to hard-code everything to make it behave at all.

And learning a new toolkit is always a pain - probably took 5 hours to do this one simple window even though i've been working with layout based toolkits since gtk 1. And even then it seems to have some nasty javafx bug in which the table view in the Sell tab doesn't render properly. I really must try this stuff on another box to see if it is just something wrong locally like the shitty Intel IGP or it being 32-bit.

Playstation 4

Ok so I guess I do have a passing interest in the Playstation 4 hardware after all. Although I have only read a couple of articles about it and can't be bothered watching the press conference.

The unified GDDR5 memory sounds really awesome - I think it would make a great workstation. Memory bandwidth is a big bottleneck on any current PC, particularly with the separated memory spaces and the PCI barrier. Pity it's going to be a locked down proprietary system though, so unless it can run GNU/Linux completely then it really just isn't going to interest me this time. I really wonder if the time of the proprietary console hasn't passed ... it's hard justifying the expensive games and expensive blu-rays if most people are happy with flash games and youtube. For entertainment i'm more inclined to read a book, write software, or make something with my hands than play a game - i've got a few unopened PS3 games as it is (and no it isn't ironic that my latest foray into software is a game ...).

On the other hand, the hardware itself isn't particularly proprietary this time - maybe there are some customisations but it is basically just off-the-shelf AMD hardware. The unified memory is one thing mentioned with AMD's HSA plans, and is critical to their performance road-map, so with any luck we'll see comparable APU motherboards available for other OEMS. In the SteamBox perhaps? AMD are hinting at some 'good news for gamers' coming up at any rate. It's about time PC architecture got a bit of a rev-up, Intel has held it back for too long with their over-engineered-junk-as-a-competitive-strategy approach. Well fingers crossed anyway, although this is just a bit of hopeful optimism at this point.

And as an aside it's sad to see CELL BE go. Or it would be if you could ever have gotten a PC with one in it, and they kept up the development to keep it competitive. I learnt a lot about hetereogenous processing, SIMD, and parallel processing from my short stint with it.

On the software side - what's this obsession with "social" junk? Gah. No thanks. Wasted enough of my life on that shit in the past, and when I do want to interact with people I want to be with them. And when I don't i'd rather be properly alone. I can understand the Google's and other people-spying companies of this world wanting everyone to perform all interactions on line (whilst some rentier takes a cut of the cost!), but physically meeting friends is much more satisfying, private and totally free in all meanings of the word.

Early dawn of a new dusk.

Well I basically spent the whole weekend hacking on the Dusk server code. A lot of refactoring and code cleaning, fixing messy logic, abstraction and information hiding. Getting rid of the nasty hungarian notation, moving to modern java types and constructs, and so on.

Broke a lot on the way, but managed to fix some of it and found a few bugs here and there too.

Some of the 'new' Java features that have been particularly handy:

Switch on string. I didn't think much of the feature when I heard about it, but with some of the code having "symbol" tables with 50+ tests implemented using if(), the switch statement fit in very well. I did try converting the command parser to use a hash table but it required so many classes it blew out the code-size too much;
Generic containers just to clean up a lot of unnecessary casts;
The container classes other than "Vector". This replaced some custom container classes or eliminated a pile of code;
Iterators. Let me do nice things with the map (more below);
Subroutines. Quite a lot of code was just cut and pasted from place to place, and simply moving the snippets to parametrised functions just cleaned up a lot of clutter. Don't underestimate the power of subroutines;
foreach came in handy in many places;
inner classes allow for information hiding and better abstraction.

The Tile Map

So apart from just general code cleaning I also replaced big chunks of code with simpler or better implementations. One of the bigger pieces was the TileMap class.

Previously the code had a couple of 2D arrays to hold the tile numbers and the entities on the given tiles. Every bit of code (from anywhere in the class structure) that needed it just accessed it directly - sometimes with a lock, sometimes with the wrong lock, and sometimes with no lock at all. So apart from having a public array being accessed willy-nilly, most of the uses simply fell into the category of scanning the player-viewable area for information. i.e. ripe for code re-use.

As an aside, much of it just wasn't very well done code, it looks like whomever wrote it had some ArrayIndexOutOfBounds exception problems and kept poking it till it didn't crash then cut & pasted the result everywhere. For example:

int i,
    i2,
    i3,
    i4;
DuskObject objStore;
                
i=0;
if (thnRefresh.intLocX-viewrange < 0) {
    i = -1*(thnRefresh.intLocX-viewrange);
}
i2=0;
if (thnRefresh.intLocY-viewrange < 0) {
    i2 = -1*(thnRefresh.intLocY-viewrange);
}
for (;i<mapsize;i++) {
    if (thnRefresh.intLocX+i-viewrange < MapColumns) {
        for (i3=i2;i3<mapsize;i3++) {
            if (thnRefresh.intLocY+i3-viewrange < MapRows) {
                objStore = objEntities[thnRefresh.intLocX+i-viewrange][thnRefresh.intLocY+i3-viewrange];
                while (objStore != null) {
                    ... process ...
                        objStore = objStore.objNext;
                }
            }
        }
    }
}

And this example is synchronized on this - which is the wrong lock.

So apart from the curious convention of post-fixing almost every reference with "Store", and giving loop indices meaningful names like 'i2', the logic is rather convoluted. Took me too much effort to realise it is a simple 2D scanning loop over (x-viewrange, x+viewrange) and (y-viewrange, y+viewrange) with some bounds checking; which can obviously go outside the loop.

As this was repeated in a few places I decided to encapsulate it into the TileMap and provide an Iterable interface that lets me share that logic. It also allows it to hide implementation details (e.g. rather than a 2d array, it is a 1D array - and it should probably just use a hash-table anyway) and handle locking reliably.

Of course the primary benefit is that it just simplifies every use:

for (TileMap.MapData md : map.range(refresh.x, refresh.y, viewrange)) {
    for (DuskObject o : md.entities) {
        ... process ...
    }
}

MapData includes both the tile info and the entities on that tile, and may in the future include per-tile scripts and so on (i'm not sure yet as the script stuff needs a context - which is currently the main game object). The object itself (MapData) and the list is only allocated once on the iterator, so it is cheap and relatively gc friendly to iterate.

But wait, there's more! There were two other main users of the map which required poking around it's internals: visibility testing (i.e. who can see what) and path finding. So I also implemented these inside the map, and took the opportunity to improve the algorithms (although they might need tweaking).

For looking I used Bresenham's algorithm for testing of obstructions through the line of sight. It does give 'enhanced vision' in that it will allow for looking diagonally through gaps which was not possible before, but that can be tweaked if it is a problem.

For path finding I used an implementation of A* (I started with this one) which is restricted to moves in cardinal directions. Since traverse-ability testing requires script execution a callback is used so the map code doesn't need to know about such things. Actually it is perhaps a bit too powerful - it will find a route to any tile you can see even if it has to traipse all the way around the map to get there. But again this can be tweaked without having to change every bit of code that uses it.

In both cases the meat of the code is somewhat smaller than that which it replaced and it is much easier to re-use, at only a modest cost of a bit of plumbing. e.g. a very simple example which prints the directions required to get from start to end, only moving on empty tiles:

MoveListener callback = new MoveListener() {
    public boolean canMoveto(MapData md) {
        return md.tile == 0;
    }
};
for (TileMap.MoveData md : map.move(startx, starty, endx, endy, flags, callback)) {
    System.out.println("Move: " + md.direction);
}

Next?

Well there is still on-going renaming, hiding, and abstraction work in the server. Most i/o isn't implemented reliably (e.g. delete file + write RandomAccessfile). The script compiler/executor uses non-symbolic integers as opcodes. There's too much use of a try-catch to hide bugs rather than fixing them. And there's some bugs in visibility syncing with the client which i've had troubles with - probably ripe for a complete re-do because after too many failed attempts at understanding the code, maybe it's the code at fault and not me.

As for the client, it still has some awt windows which need replacing. The shop and equipment screens are the primary ones, and it could always use an inventory screen as well. Once i've had enough of poking the server I might look at that. One problem on the client is that I fairly regularly get some exception inside javafx which stops it rendering - requiring a restart of the application. Seems to be a javafx bug on this system (i'm coding on my laptop, which I normally only use for C).

And for efficiency's sake at some point the server protocol should be converted to binary, or at least have a binary option. Well, actually a more robust protocol needs to be created.

Then at some point there will also be work on a code release, a couple of issues to sort out first and I think the client should be awt-free before then too. But i'll try to do it before I lose interest and find something better to occupy my time!

Update: Just found out i'm back to work in a week so I might get a bit more in while it's hot in my mind but I need to wind down to prepare for work again too.

About Me

Tags