## Keep it really really stupid, stupid

Nov 21, 2016

One of the biggest problems we had when I worked on TOME: Immortal Arena, was ensuring Data Integrity in our data files. An outline of the situation:

1: We were working as fast as we possibly could, and adding new features that needed us to continually extend the data model.

2: Our game was highly data driven (xml).

3: We had thousands of XML data files, that referred to each other by ID.

4: A broken build was very expensive, because our team had 70+ people on it.

5: Our game data was mostly just used by the server and client, but we put too big a premium on it being serializable. In the end, serialization wasn’t really that important.

6: We built a lot of fragile tooling in Unity so that anyone could edit data without having to edit XML by hand.

Our data was relational, and that’s how we basically consumed it. Unfortunately, we didn’t also build an Object Relation Mapper (ORM) (like Ebean in Java, or Active Record in RoR) because they’re complicated, fraught with peril, and basically orthogonal to game development. Because our datas were linked by Ids, this resulted in us having to manually resolve relationships at runtime the gross way that most people are probably familiar with.

var abilityData = DataStore.GetInstance().FromId<AbilityData>(this.Data.Abilities.Ultimate);

var subAbilities = ability.SubAbilities.Select( p => DataStore.GetInstance().FromId<AbilityData>(p)).ToList();

//oh god, make it stop
var projectile = ......

//hit me with hammers
var timelineData = DataStore.GetInstance().FromId<TimelineData>(ability.Timelines[0]);

//the boilerplate, oh god, the boilerplate


This is bad because it’s almost all boilerplate, accomplishing very little. Because this pattern doesn’t make any guarantees on data integrity, there’s the possibility that data can be null, and this could take down your server or throw your client into an irrecoverable loop.

Because we had many people working on the game relentlessly, we had many data breakages, and poor data integrity. Situations like Character datas referencing Ability datas that no longer existed were surprisingly common. We tried writing tests to validate our data integrity, but with a constantly evolving data model, it was a sisyphean task.

It was with grim determination that I set out to create more or less the same thing for Black Future, because I had the stupid notion that this is simply how grownups do things.

## But Wait

Because creating shitty and fragile tooling in the above scenario is a lot of work, I wasn’t eager to dive in when I’m only really trying to prototype a game and standup a character. I instead put together a super minimal data system that I fully intended on replacing. Except….it kind of rules.

Presenting, my really stupid data pattern:

2: The model is relational… but also a document….

3: Relationships are resolved and checked at compile time. Oh, you deleted that ability for that character? Now your game doesn’t compile. This is good.

4: It’s all in code, so even a broke-ass IDE like MonoDevelop is a good enough tool for refactoring and editing.

5: It does not over-value serialization, which is great because Black Future does not have the same networking component as Tome did.

6: It’s all in code, so it can be scripted. There will be a future post on my AI system that shows how useful this is.

7: For a game in development and under constant iteration, I cannot overstate how valuable it is to be able to easily search for references to an ability/buff/item before deleting or changing it.

Ok, so here’s what it looks like as applied to defining inputs into World Generation

MapData.cs

public static class MapDataTable {

public static MapData Map1 {
get {
var ret = new MapData();
ret.ResourcePath = "Maps/Map1";
ret.Id = 1;

return ret;
}
}

//scriptable!
public static MapData Custom(params ZoneData[] zones) {
var ret = new MapData();
ret.ResourcePath = "Maps/Map1";
ret.Id = 666;

return ret;
}
}
}


ZoneData.cs

public static class ZoneDataTable {
public static ZoneData ZONE_1 {
get {
var ret = new ZoneData();
ret.Id = 2;
ret.Name = "New Parish";
ret.Tag = ZONE_TAG_NEW_PARISH;

ret.Flags = ZoneData.ConstraintFlags.NormalSecond;
ret.PracticeAble = true;
ret.MaxRooms = 25;
ret.Difficulty = 8;
return ret;
}
}

//.... zones zones zones

}


Some stats: This simple data system supports the following in Black Future:

1 Map

4 Map Zones

24 Rooms

12 AI Trees

100 Abilities (including buffs)

20 Enemy Characters

14 Room Fixtures

7 Active Items

12 Music Tracks

12 Pickups

18 Projectiles

2 Playable Characters

32 Weapons

Some other less interesting data types

I expect all of these numbers to balloon upwards as development continues. So far, it’s been a really usable system that’s easy to debug and generally has more answers than problems. Having data exist in a pristine version that can be looked up, as well as having all relationships resolved when I request an object is just really nice. This is the referential + document benefit mentioned above. IDE’s are already great tools for the most common data tasks, like quickly finding an object, or checking where a data file is used, and the compile time resolution of the graph has it’s own obvious benefits. This pattern, while dumb, completely eliminates entire classes of bugs related to data integrity.