The Ancient Armies Blog

Firstly I’d like to welcome two more new testers: Frank and Jon! Welcome on-board!

Only managed to squeeze in just over 4 hours of coding this week.

One of the rules I have is that I don’t code at home on the same day that I’ve been coding at work. If I did, I would be coding Ancient Armies whilst my brain was still fatigued from coding professionally – probably not a good thing.  

Also I don’t think it would be fair to the company that I work for, as I would be turning up the next day without giving my brain a sufficient rest period – again probably not a good thing.

In my previous job this didn’t matter too much because I was multi-skilled as a Systems Designer and a Software Engineer so I was never coding everyday. However in my new job this is no longer the case! For me this is a good thing, not a bad thing, after all doesn’t everybody want to do a job that’s also their hobby too?

Although I only did 4 hours I made the most of my time…

First up, I added my new optimized height finding system to the last two remaining shapes, namely rhomboids and circles. 

Here is a screenshot of my system’s calculated test points for the first Rhomboid under test (click for full-size):

As can be seen, my system appears to have calculated a good set of test points. One thing that can’t be seen from the above image, is that I have made an alteration to the system so that it adds an extra test point at the geometric centre of all shape types. Probably not really needed, but I’m just playing safe.

Next I chucked a super-huge rhomboid at my system with some 4000 chariots which resulted in (click for full-size):

Here we can clearly see the additional test points that the system has calculated. So far so good!

Now on to circles (click for full-size):

Again the system seems to have coped with that circle quite well! Next I gave the system a larger circular unit to play with (click for full-size):

Calculated test points are once more spot on!

That’s all the unit shapes accounted for using my new test-point system. I don’t have to worry about unformed units, because they inherit their test points from the shapes from which they originate.

Testing at this stage also proved that the unformed unit bug had already been resolved by just this singular optimization alone!

However, I was on a roll and decided to put a few more optimizations in…

When looking for further optimizations I realised that the one thing that could be optimized was the algorithm for automatically generating broken 3d shapes from a fixed shape. The original system used a kind of cheat to make calculations easier. Consider the broken Companion unit below (click for full-size):

Can you see where the ‘cheat’ is? No?

I’ll come clean, the ‘cheat’ only reveals itself when you take the unit off map to see what is hiding under the terrain (click for full-size):

Hopefully the ‘cheat’ should now be obvious!

Whilst this ‘cheat’ made the generation algorithm a lot simpler, it did mean that the unformed version of the unit had a lot more triangles in it than it really needed. In this case over double! Plus in terms of aesthetics it’s a bit of a failure.

So I hit the ‘On’ switch in my brain and came up with a more complex algorithm, an algorithm that only creates the triangles that are actually required (click for full-size):

Pretty Cool Eh?

Only managed to reduce the triangle count by more than half!

Any other optimizations?

A few more…

Whilst standing outside – yes I do go out! – I noticed my faint shadow on the pavement.

I then decided to do an experiment…

I took a few paces to the left and checked my shadow. Next I rotated my body by 45 degrees and again took note of the shadow. Then I took a few paces back and again looked at my shadow. At this point I decided to do one more final check and moved a few paces to the right and once again observed my shadow. I suspect that anybody looking must have thought I was missing a few marbles

While this wouldn’t be the first time(!), my experiment did yield a very valuable result. I had realised that when a light source is very far away, the rays of light that reach you are essentially travelling in parallel straight lines with the source appearing to be fixed in space even when you move. This is in direct contrast to a nearby light source!

I did the experiment because I wanted people to think that I was crazy remembered that I had modelled the sunlight in the game to replicate a very distant light source with parallel rays. As such, its rays always hit the map in a parallel way from a seemingly fixed point. I now smelt an optimization here!

I realised that if the direction of light was always the same regardless of where you are on the map, then shadows for a model need not be calculated for simple movements (translations) because the light incident angle would always be the same! The only time the shadow volume would need recalculating is if the nefarious unit decided to do something underhand… say like rotating itself!  

The system as it originally stood calculated shadow volumes every-time a unit so much as sneezed! A few code tweaks later resulted in the shadow volumes being calculated on unit rotations only. 

Woot! Another optimization added to the system. This optimization really made quite a big difference to the measured figures for moving units! So who says that standing around is a waste of time?

Next, I added further optimizations to all unit models to prevent them recalculating their vertices unless they are going up or down a slope. Small undulations in the ground would be ignored. Again another boost to performance!

For my performance tuning finale I was going to add a throttle to the main render loop within the control section. However, this made very little difference so I removed the test code and left things as they were.

Despite the last optimization not working, I’m a very happy bunny. Moving units have never had such a high frame rate! And the unformed unit issue is now well and truly solved!

What’s next?

Based on the feedback I have been getting from my testers I’m going to spending the next few weeks adding additional functionality to map editor. 

The map editor uses the maps library which is also used by the scenario editor. I figured that as some of these changes would be in the maps library, it would be a good idea to address these enhancement requests prior to continuing work with the scenario editor. Of course, thanks to my cunnng architecture the scenario editor will automatically get all the enhancements added to the map editor!

That’s it for this week – I’m off to make a Sunday Roast!

Laters

RobP

Firstly, I’d like to welcome José to the project as a new external tester. Hope you have fun!

Work this week has again been concentrated on resolving outstanding issues. Many of the issues were quite trivial – if a little hard to diagnose. However, the one that has been burning most of my time this weekend is unformed units.

These dastardly units work exceedingly well on my laptop, but when the same build is installed on my main and more powerful machine, they exhibit big performance issues and other weird artefacts like random height changes!

This issue is unlike any other that I have faced because the symptoms only occur on my main machine(!) despite the version under test being identical on both machines. I was expecting that it would be my laptop having all the problems as it is a truly ancient machine. But in a bizarre twist of fate and possibly design, the laptop has no issues whatsoever - instead it’s my uber-mega-system which is having the problems.

What could be causing my faster machine to perform worse than my very slow machine?

I had a theory….

I suspected that the faster machine had a much higher throughput within the main render loop of the game. I further theorised that if there was a task involving a lot of processing power going on, the faster machine would theoretically call it more often than the slow machine – in fact, much more often! So in this case it pays not to be so fast!

After running Ancient Armies through my debugger it became obvious that there were a few places where the code was getting bogged down.

Whenever a unit is moved in Ancient Armies the software checks every vertex point of that unit against the underlying map to get the height information for that vertex. In addition, the system effectively recreates the unit’s vertex buffer based on the newly gained height information. It is this system that allows units to ‘terrain-follow’ when being moved around and to extend their heights when they are on the edge of a slope.

Two things occurred to me here:

1. Checking every vertex of a mesh is highly inefficient. Firstly the mesh has to be locked and unlocked – an expensive operation. Secondly, why bother testing so many points? There is no real need to. Unformed units are especially susceptible to this as they have a much higher polygon count than their formed unit siblings.
2. Also why bother recalculating the vertex buffer every time a unit is moved? After all, we only really need to do it when the unit experiences big changes in ground elevation, such as going up or down a hill. Again this is an expensive operation as buffers need to be locked and unlocked and once again our unformed unit fairs badly in this respect due to its high polygon count.

So what is my solution?

It would be a multi-pronged solution. The first thing I wanted to do was create a height finding technique that doesn’t require as many sample points and has no requirement to lock and unlock the unit’s vertex buffer. Basically, I wanted it to work with no dependency on the number of vertices within a unit. Next I would prevent the unit vertex buffers from being calculated unless transitioning hill elevations and finally I would ensure that the main render loop has a governor installed on it to limit its maximum throughput.

On to part one. How was I going to make vertex height checking more efficient on my units?

The cunning scheme I hatched was to rely on the fact that all my unit shapes have concave polygonal shapes. Concave polygons have no vertices whose coordinates fall inside the bounds of the other vertices within the shape. In plain English terms this means that I only need check the height at the few critical points of the shape in order to fully define its external boundary.

What is more, these points could be stored outside of the vertex buffer, which means that I would no longer have to take the performance hit of locking and unlocking the vertex buffer! Woot!

I set about modifying my code to dynamically calculate these  boundary points. The screenshot below shows the points that my new code calculated as the boundary sample points for a standard square formation (click for full size image):

Cool Eh?

With the new algorithm the above unit only requires that four height samples are taken, a vast improvement when compared to the original system’s twenty one. So even for a unit that is not unformed, the savings in performance are huge – especially when one considers the number of calculations that go on under the hood to calculate each height precisely. It should be noted that the red sample points are only there for debugging purposes - they will obviously not be around once coding has finished!

So is that it?

Nope.

I also took the liberty of making my sample point algorithm intelligent. Not intelligent as in HAL2000 intelligent, but intelligent as in being able to take a unit’s overall size into account! To show this in action here is a screen shot of a much larger unit – the Macedonian Hypaspists (click for full size image):

Here one can see that my algorithm has realised it is dealing with a relatively long unit and as a result it has calculated more sample points on the x-axis of the unit! Of course my cunning algorithm also works in depth (click for full size image):

Next I moved on to wedge units (click for full size image):

And of course my algorithm takes unit size into account for wedges too as shown below (click for full size image):

I’ve also started work on Rhomboid shaped units too, although these are not yet coded to dynamically allocate points based on size. (click for full size image):

I still have more work to do to completely resolve the unformed units issue. Obviously, Rhomboids need to be finished off and circular units need to be addressed. In addition to this I will need to put code in place to only update a unit’s vertex buffer when necessary, IE when going up or down a hill. Plus I would need to add a governor to the main render loop. All in all quite a lot to keep me amused!

What else have I been up to this week?

Well, I managed to find some time to setup my own email server and integrate this server with my issues system. As a result whenever work is carried out on an issue or a build is released, the relevant people automatically get an email informing them what’s going on!

I have also carried out further work on the tester’s forum in terms of stylistic design and providing enough documentation to get the testers going.

In terms of my fight against the issues raised by my testers, I think I’m winning – *cue cheesy grin*  (click for full size image):

As one can see the number of outstanding issues as indicated by the blue line has taken a substantial dive!

That’s it for this week.

Laters

RobP

This week I have been busy fixing issues and enhancing my home development environment.

The first thing I did is migrate my code forensics and issue management systems over to a MySQL database. Doing this is recommended by the company that produces this software and gives me the ability to directly tweak the tables. Although this generally isn’t recommended I have already used it to fix some issue resolutions after I changed my work-flow process.

Next up, I have been spending time increasing the integration between Fish-eye, JIRA and Perforce. The results of my efforts now mean that Fish-eye is now ‘Issues’ aware and JIRA is now linked into my source code in such a way that I can now see the source files that got changed for each issue and even be able to carry out code forensics simply by clicking on them – the image below shows the new functionality (Click to see full size):

Another thing I have been looking at this week is my release process – or lack there-of. Up to now I have simply been developing code and building as and when I needed to. These builds overwrote previous builds, so I only ever had access to the latest build. What is more there was no real versioning – being the only person on the project I didn’t see the point.

However, now that I have some external testers, I decided it was time to implement a proper build and release process. This has now been implemented and now handles such things as when each build should be released and what should be in it. I have also put in a version numbering system in place and a process to ensure that I can build at any version I so choose. This means that issues can now be reported in a specific version and fixed in a specific version.

The eagle eyed amongst you have probably noticed the new link on the right to the testers forum. That is another new addition that I have been working on. It will soon be live as a private forum for all testers and myself. Hopefully it will be useful for discussing issues and generally enabling communications between many geographically separated parties.

Right now my intention is to continue with resolving issues as the highest priority and using my limited time to try and stay ahead of the issues raised. The graph below shows how I’m doing in this particular area:

As can be seen the fight between me resolving issues and them being raised is pretty much even at the moment. The blue line at the bottom shows how many issues I have unresolved at any one point in time. The blue line shows that I’m ahead of the curve – but only just.

What’s the breakdown on these issues? After all 15 issues sounds quite a lot!

Well the breakdown is as follows:

So in reality I only have 6 bugs – it is these 6 bugs that are my highest priority. Once they are nailed a few of the improvements will be tackled – some in parallel with the bug fixes. Once it becomes apparent that I’m on top of things I will then continue with brand new functionality.

One interesting bar graph that’s worth a look is the spread by component graph. This can be seen below:

The above chart shows a break down by component of every issue ever raised. It should be noted that many issues affect more than one component – hence the elevated issue count. From this chart I can see where the potential problem hot-spots are and which components are pretty stable. Some of my more stable components are not even on this chart because they have no issues against them.

The great thing with my home development setup is that I can obtain a wealth of highly accurate project management information at the touch of the button with no real overhead due to my tight tools integration!

In terms of the fixes I’ve been looking at, the big one this weekend was a memory leak in the map editor – yes another one! This one differs from the last one in that it only ever occurs every time a new map is loaded into the same instance. This one took quite a few hours to nail, but was important because the testers were reporting the editor crashing after loading just a few maps. The graph below shows memory usage for the pre-fixed version and the post-fixed version. As one can see – the leak has been well and truly plugged!

Other than the above, most of the other issues I have been looking at have been pretty minor – though I do have some juicy ones to look forward to this week.

It seems that over the past few weeks my role on the project is moving from just merely a technical one to a much more project manager orientated role. I think this is a good thing for the project as it will focus what I need to do and hopefully result in the better use of my time.

Now on to the title of this post!

Yes – I am looking for one or maybe two additional testers. I want to pick these people fairly randomly, so I will settle my choice based on a competition!

All you need to do is watch the video below of me practicing on my guitar and tell me at robertapollard@hotmail.com what song you think I’m playing  and a small description as to why you would like to test on this project. The winner will be picked by picking a person with the correct answer and the person that convinces me that they are the one for the job.

There is no game as such at the moment. Just a Map Editor, Army Editor and Scenario Editor. So the testing will probably take a while, but at the end of it all testers will get a full copy of the game – when I eventually finish it – just remember this is a home project

On to the Video! Good Luck!

Laters

RobP