Entity Component System architecture (C#)

In this post I’ll go over how I’ve laid out the parts of my ECS framework, with some emphasis on issues related to the .Net framework. Check out some previous posts here and here.

By all means this is not meant to be authoritative. In fact, I’ll be outlining several problems I see with the way I’ve done things.

Overview

I’ll start with a few top level diagrams. First, the EntityManager:

EntityManager block diagram

The EntityManager contains the list of all the existing entities (which are just light wrappers over a few ids), and a ComponentManager for each type of component. So the components are stored separately from the entities, each in their own list.

A particular entity is associated with various components, like so:

The entity is composed of the 3 highlighted components.

The other main piece of the pie is the SystemManager:

SystemManager block diagram.

The systems (e.g. RenderSystem, AnimationSystem, SoundSystem) contain the bulk of the logic. A system provides its logic by inheriting from a base class and overriding a few key methods. A system declaratively states which components it is interested in, and the base class is responsible for keeping the system’s entity list updated with all entities that have exactly the components in which the system is interested. As a result, the system can just loop through this list while doing its processing.

Memory Layout

One of the motivations for storing the components separate from the entities is that like components are usually processed together. If we store all the components of one type in a single list, then we can (theoretically) access them sequentially in memory as we process them. This results in good locality of reference. So it is mainly a performance consideration.

If, instead, the components were located in a collection off of each Entity object, then many of our operations would involve touching scattered memory. While this is certainly a valid way to go (I’ve seen several implementations like this in languages with managed memory), you won’t achieve any of the theoretical performance benefits of ECS frameworks.

In C++ you have strong control over memory allocation patterns. In C#, not so much.

Ideally, we want all the components of one type to exist in contiguous memory. The obvious way to do that in C# is to make the components value types (structs). Unfortunately this is too cumbersome – among other things, it prevents us from having a common base class or interface for components, and it is restrictive to deal with references to structs (for instance, you can’t return a struct by reference from a function).

The good news is that we can still put things in contiguous memory if the components are reference types (classes). I mentioned in a previous post that allocations made one after the other are generally allocated sequentially in memory too. I actually couldn’t find this mentioned/documented anywhere, but my testing shows it to be true (at least on my desktop computer).

Of course, the CLR is free to move things around in memory. But in my testing, my arrays of components always had each component object located adjacent to each other in memory. (Note: the CLR Profiler shows this is not true. But I believe this is because when running with the profiler enabled, the allocation logic is different).

This is all only useful if we access these components in order. Unfortunately with my current implementation, I do not.

The details

As the game runs, entities and components are frequently being added and removed. How do I manage these lists so that enumerating through them is efficient? How do I ensure adding and removing is efficient?

My current implementation has as one of its goals the efficient adding and removing of entities and components. This means that we avoid any O(n) operations (such as shifting all the components down in the array when one is removed). Add/remove operations are done in linear time.

To accomplish this, I need to keep 3 lists: The pool of entities themselves, a “compact” array of indices for lookup into the entity pool, and a reverse lookup array which I’ll call the roster.

If I allocate one entity, it would look like this:

Entity Pool [LiveId]	LiveId [RosterIndex]	Roster [LiveId]
EntityA	0	0
EntityB	1 <— NEXT FREE	-1
EntityC	2	-1
EntityD	3	-1

Now let’s add two more:

Entity Pool [LiveId]	LiveId [RosterIndex]	Roster [LiveId]
EntityA	0	0
EntityB	1	1
EntityC	2	2
EntityD	3 <— NEXT FREE	-1

At this point, it’s worth explaining how I can enumerate through all entities. I use the LiveId list, which is indexed by RosterIndex. I simply loop from 0 to “NEXT FREE” and look up the LiveId. I then use the LiveId to index into the Entity Pool.

Now let’s remove a Entity A. The result will be this:

Entity Pool [LiveId]	LiveId [RosterIndex]	Roster [LiveId]
EntityA	2	-1
EntityB	1	1
EntityC	0 <— NEXT FREE	0
EntityD	3	-1

There is now a gap in the Entity Pool, but it doesn’t matter since I just enumerate through the list of LiveIds (the middle column), which never has a gap. We took the now empty spot for Entity A and swapped it with the last one in the list (and decremented NEXT FREE). The Roster[LiveId] list is just needed so we can perform the swap. It let’s us know the index in LiveId[RosterIndex] where the removed guy lives.

Now when I enumerate through the entities, I’ll encounter LiveId 2 (Entity C) and then LiveId 1 (Entity B). So you’ll note that the enumeration order is different. In fact, the more add and remove operations I perform, the more chaotic the enumeration order becomes.

I use this system (which allows for linear time add/removals, and “compact” enumerations) in 3 different places:

1. For the list of entities in the EntityManager, as just described
2. For the list of components in each ComponentManager
3. For the list of entity LiveIds in each System

Just for kicks, I ran my game for a while and played a few levels. I took a look at how the items were laid out in memory for cases 1 and 2 above.

 

The global entity list on top, and the Placement component list on the bottom.

 

It’s maybe a bit hard to explain this image, but a perfectly sequential-in-memory order would be a smooth gradient from left to right. Instead, you can see chunks of sequential memory, and then random jumps to other places. Over time, this would become more and more randomized.

In summary, I have optimized for a fast adds and removes at the expense of potentially pool locality of reference.

A system that operates over a single type of component will be affected by this.

What about a system that operates over of two types of components (such as the RenderSystem, which needs to know about both Aspect and Placement)? Well, there is already an inherent problem there. Even if we were to somehow fix our spatial locality issue, we are simultaneously iterating over two separate lists. If one were guaranteed to be in nice enumeration order, the other would necessarily not.

So fundamentally, there are three things working against us for achieving good locality of reference:

1. It’s possible (though not likely) that our original list won’t be contiguous in memory, and the .net garbage collector can move things around if it wants. Of course, this is not a problem in C++.
2. My algorithm for storing Components/Entities ends up “randomizing” their order as things are added/deleted. This could get worse as time passes.
3. The fact that most systems enumerate over multiple components means that access of at least one of those component lists will be “random”.

To sum up, I can’t honestly claim that locality of reference is a big benefit to my ECS framework with the way I have it currently implemented. Of course ECS has many other benefits.

It gets worse

I haven’t yet gone into detail about how a system enumerates through the entities it cares about. I’ll do that now.

As mentioned before, each system is automatically kept up-to-date with a compact list of entities to enumerate. It is actually just a list of LiveIds. LiveIds allow us to perform a simple array lookup to get an entity from the EntityManager (thus a LiveId is a number that is guaranteed to be unique for the lifetime of an Entity). Systems aren’t typically interested in the entities themselves though: just in their components. So a system might do something like:

 

For each entity LiveId in my compact list
   GetComponentA
   GetComponentB
   Do stuff with those components.

I need to explain another thing now. We have LiveIds for components, just like we have LiveIds for entities. The component LiveIds are quick lookups into the array in the ComponentManager for a component type. We could get away with re-using the Entity LiveIds, but then each ComponentManager would have to be large enough to hold n components, where n is the max number of entities we have. This could work with common components like Aspect and Placement (which are on most entities), but it would be a waste of space for other ones. So instead, we tailor the size of each ComponentManager to the number of components it is expected to have.

So where are these component LiveIds stored? The EntityManager has a giant array of them indexed by entity LiveId and ComponentId. ComponentId is just a fixed number that corresponds to a particular component type (e.g. Aspect = 0, Placement = 1, SoundLoop = 2, up to the number of component types we have).

So if we have 1000 entities and 20 different kinds of components, this array would be 20,000 elements.

So going back to our enumeration in the system:

For each entity LiveId in my compact list
   GetComponentA

GetComponentA involves the following:

• index (in a non-sequential fashion) into the giant array, using [LiveId, ComponentId] to find the component LiveId
• index (in a non-sequential fashion) into the ComponentManager’s array using that component LiveId

So that’s essentially two random accesses into arrays in order to get the component we need. Hardly ideal. Of course, a system that only cared about a single component type could directly enumerate through that type’s ComponentManager, thus incurring only a single lookup.

Thoughts

I haven’t used my ECS framework for a large enough game world yet to really know the performance implications. So it’s hard to make any conclusions about that, other than I know the implementation could definitely use some improvement.

I’d be interested to hear about other people’s implementation details, regardless of whether or not they are in native or managed code. I don’t think managed code is hurting me too much here.

 

Entity Component System framework, redux

My Windows Phone game Entangled was the first game I shipped that used an Entity/Component/System framework. I used a simplified version of the framework I developed for my much more ambitious still-in-progress game, and it worked fairly well (See previous posts on this topic here and here).

Now that I’m re-skinning and updating the game for a port to iOS (and perhaps Android), I’ve had some more experience with the Entity/Component/System framework. I’ll talk about that in this post, and give some examples of how I attempt to solve some game development problems with it (something that is often missing from other articles).

Brief overview

I’ll just summarize my framework here in a few paragraphs.

The game world consists of Entities. An entity is a class wrapper around some identifiers.

Associated with each entity (via the identifier) is a set of Components that define the entity. Some examples of Components are:

• Aspect: defines what the entity looks like (texture info, color, etc…)
• Placement: defines the entity’s location and orientation
• LaserInteractive: this is something specific to this particular game, and contains information about a core gameplay feature
• Input: defines whether or not this entity fires click/drag events when interacted with by the mouse or touch
• Entanglement: another game-specific component. It is used to link one entity’s position or orientation to another
• FrameAnimation: indicates that the entity should use spritesheet animation, and specifies things like framerate.
• SoundLoop: indicates a sound that is to be played (and volume, etc..).
• Scripts: this is a bit of a special one, and describes which custom scripts are associated with the entity.

Note that Components only define data, no logic. They essentially describe “intent”.

Next are the Systems. This is where the logic lies. For instance, the RenderingSystem loops through all entities that have Aspect and Placement Components and prepares the necessary stuff to draw them. It doesn’t need to do anything special to determine which entities these are. It just states declaratively that it’s interested in entities with those two Components, and those entities’ ids are automatically added to a list in the System.

Other examples of systems are as follows:

FrameAnimationSystem: It tracks entities with both Aspect and FrameAnimation components, and adjusts the Aspect Components with the correct location in the spritesheet, based off the current time and the framerate specified in FrameAnimation Component.

 

Spritesheet animation frames

 

LaserSystem: This is the most complex system, and manages the main gameplay logic that involves entities with LaserInteractive Components.

EntanglementSystem: This is another game-specific system. This looks at entities with the Entanglement Component, and ensures their positions and/or orientations are kept in sync with the entity’s entangled pair. It also draws the entanglement visuals (it is not only the RenderingSystem that is responsible for drawing).

 

These two objects are entangled. EntanglementSystem manages that, and is actually also responsible for drawing the visuals.

 

Winning System: This monitors certain entities’  state to see if the current level has been won.

So you can see that most systems deal with one particular component, but also require a set of other components in order to do their job.

Some systems operate over a similar set of entities (for instance, the LaserSystem and WinningSystem), but contain completely unrelated logic. So while they could be part of the same system, “one class, one purpose” dictates they should not be.

The good things

The benefits of Entity/Component/System frameworks are well-known, and I won’t go into detail about them here.

But I will say it’s been really good for quickly prototyping gameplay ideas for this game, since it’s easy to assemble new types of entities together from existing Components.

ECS also really encourages writing code exactly once. The temptation to copy-paste code really isn’t there, because it just isn’t necessary. And that reduces the chance of bugs.

The hacks

It’s fair to look at examples where I’ve had to hack things into the framework and ask why.

There is the temptation to throw more data into existing Components. For instance, I eventually realized I needed to ensure that things were drawn in a certain z-order (I’m using alpha-blended sprites, so I’m not using a z-buffer). So I added a Layer property to the Aspect component. Is this the right place? Probably not, it doesn’t have anything to do with the Aspect. It may deserve a new Component altogether. But just for one additional property? (Actually, as I’m writing this, I realize that the Placement Component is probably the right place).

I also wanted to support dual textures and a custom rendering effect for some entities. I ended up hacking some stuff into the Aspect Component for this (if a certain bool is set, use the custom effect). I actually think this is sort of the right thing to do, but the part of the Aspect that defines the sprite needs to include this information. Currently it’s just a texture – I need another more complex abstraction there that the RenderingSystem understands.

Bricks need two textures.

Another example is that I needed some entities appear to vibrate back and forth, or slide in from behind other objects. However, I didn’t want to change the position of the entity from the point of view of most code (just from the point of view of the renderer). So I added an “AdditionalVisualOffset” and “AdditionVisualOrientation” to the Placement Component. Only the RenderingSystem looks at this.

That was definitely a shortcut/hack. A better solution would probably be to make the entity that has the visual Aspect be the child of the main entity. I support parent-child relationships between entities, so that one entity can be positioned relative to another. The child entity’s position could be adjusted to simulate vibration, without changing the position of its parent entity (which wouldn’t have Aspect). Unfortunately that would break an assumption I have that the main (parent) entity is the one that has the Aspect component. That assumption is needed for logic that sets the entity’s color (on Aspect) based on the type of laser light it emits (on LaserInteractive). I also have code that swaps the sprite used depending on information in LaserInteractive.

Unique logic (scripts)

The bulk of the game logic is contained in the Systems, but I often need more “one-off” logic. Some examples are:

• The code that scrolls new levels into view gradually, and “introduces” a level by highlighting certain objects, and sliding them into view.
• I have objects in the game that change the color of other objects that are placed inside them.
• I have plants that start to grow when you are in a winning state.
• The name of the level appears gradually, letter by letter, when the level is loaded.
• Some objects in the game world make a sound as you drag them… the fast you drag, the louder the sound.
• Animals pop up and offer you hints if you want.

These types of things are too “small and unique” to create new Systems for. So I have a concept of “Scripts” that can be attached to entities. I actually have a Scripts Component which is just a list of scripts that apply to that entity (scripts do get some special treatment though, so making them part of a Component may not be the best solution).

 

A script is used to slide these objects into view and then spawn another script that makes them “flash”.

 

One special thing about scripts is that there is only one instance of a script, even if it is attached to many entities. One of the original motivations for this was to reduce the heap complexity for my objects (I’m using C#, and garbage collection time increases the more allocated objects you have).

As a result, all state for scripts is attached to a generic property bag that is part of the Scripts Component. Although this limits the kind of information scripts can store, it does have the following benefits:

• Since all scripts attached to one entity are updated together (in sequence), they are accessing the same region of memory in the property bag (better locality of reference).
• I don’t need any custom serialization code for a script (i.e. for save games) – all the script state is serialized in the property bag.
• It will allow me to easily use a separate runtime/interpreted scripting language for logic. I don’t do this for this game, but I do for the other larger game.

Getting/setting variables from a property bag makes the C# code in the scripts a little clunky, but this wouldn’t be an issue if I were using a separate scripting language as I mentioned in the last bullet (since I would then have syntactic sugar that makes interfacing with the underlying property bag look just like normal variables).

 

Clunky way to set property bag variables from C#.
To identify variables I use a unique integer id, similar to that described in this post: http://mtnphil.wordpress.com/2012/01/06/identifying-entities-in-the-game-world/

 

I should also mention that the scripts just assume the bare minimum about the entity to which they are attached; so even though I’m calling them “unique code”, they can be applied to any applicable entity. For instance, the “faster you drag an object the louder it gets” script just requires that the entity have Placement and SoundLoop.

One thing I still need to support is the ability to pass parameters to a script when attaching it to an entity.

Messages

Of course, sometimes you need to communicate between systems, or between scripts and systems. I use messages for that. You can send a message “somewhere”, or to a particular entity.

Systems handle messages (they declare which ones they can handle), and Scripts can also handle them if they are sent to a specific entity (in that case, all Scripts attached to an entity get a chance). This is the one case where the entity framework has special knowledge about the Scripts component.

I don’t have any fixed list of message types; I just add them as I need them. They are basically used to glue together bits of code where there is no obvious way to do it. Probably for several of these cases I could refactor things to avoid having to use messages.

 

A script makes plants begin to grow when it detects that you are winning.

 

Some examples of message use:

• PlaySound: A script or system can fire a message that says to play a sound. I could probably solve this in another more explicit way (e.g. having access to some audio manager or something), but currently from scripts there is no access to any particular game classes. So they just fire a message and hope someone handles it. In this case, the SoundSystem will handle it.
• ObtainWinningState: This gets information about if the player is winning the current level (the WinningSystem answers this). I have a script that makes some plants grow, and it uses this to know when and how fast to grow the plants.
• Click: This is sent by the InputSystem to the entity on which it detected a click. In order for anything interesting to happen, that entity will need to have a script attached to it that handles the click message.
• RevealTitle: Here’s one that I realized was totally flawed and bogus as I was writing this. When the level is loaded, there should be an entity that represents the title text of the level. It’s initially hidden. A level initialization script runs for a varying amount of time, and then fires this message. It’s picked up by the title text entity, which has a script that begins to reveal the letters one-by-one. The thing is, I don’t have a way to broadcast a message to all entities in the world. So the level initialization script actually needs to know which entity to send it to (it enumerates all entities and searches for those with specific Components that identify it as the title text entity). It follows then, that it could just attach the “reveal letters” script to that entity at that time. This is one example of an area that is not well thought out by me, and this is just some glue to hold it together.
• UnloadLevel: Another kind of useless one. The game keeps track of the entity that has the “initialize level” script attached to it. When its time to unload the level, it sends this message to that entity. That’s picked up by the “initialize level” script, which then frees all that level’s entities after a predetermined waiting period. I could just as easily have put the unload functionality in another script, and have the game just add that script to the entity. Instead, I used a message, because I needed to pass parameters (how many seconds to wait before unloading), and I have no way to pass parameters to a new script I’m attaching to an entity.

Other bits of functionality

I segregate groups of entities by their owner id (entities can be owned by other entities, or by a “top level group id”). Each level number has a unique “top level group id” associated with it, and all the entities for that level are created with that as their owner. This allows me to have multiple levels loaded at once. This lets me position levels at different spots in the world and pan between them (so two levels exist for a short period of time).

Panning from one level to another.

I have a way to enumerate all entities owned by a particular owner. So when it’s time for a loaded level to finally go away, I can say “enumerate all entities owned by 1234″, and then “free this list of entities”. Of course this involves scanning the entire entity list, and feels like a heavyweight operation.

Unfortunately a few of the systems care about this too. The WinningSystem is only interested in entities that are associated with the “current level”. And the LaserSystem needs to segregate entities by level so that they don’t interact with each other.

The LaserSystem solves this by the fact that I made a Constraint Component that defines an area that at entity operates within. A bit of a hack, but I suppose it’s reasonable.

The WinningSystem unavoidably (I think) needs to access some global state that indicates the current level entity owner.

Globals Bad

One further thing I can think of. As you travel from the main menu to the levels, I wanted to fade the levels in from a distance. The only reasonable way to do this is to render them to a render target first, and then draw that with varying degrees of opacity.

Of course, I also wanted some of the menu UI to be a part of the ECS framework. It just makes things easier.

But how would the RenderingSystem know that these entities are to be rendered to a different buffer than these other ones? In fact, the RenderingSystem is only part of the entire rendering functionality, and doesn’t manage render targets and such. It just issues draw commands.

The best solution I could think of was to have multiple simultaneous but completely separate entity managers. The level entities operate on their own complete separate “space” than the UI/menu entities.

Unfortunately I had a bunch of global state (or rather singletons) that prevented this. There was a single EntityManager and SystemManager. Global stuff always comes back to bite you. Fortunately, removing global variables is usually fairly straightforward to do, if tedious. So I now have the ability to run two (or more) completely separate entity systems.

Brain dump over.

Re-skinning “Entangled”

As I mentioned in my previous post about MonoGame, I’m in the process of porting my Windows Phone 7 game “Entangled” to iOS (either with MonoGame or by making a Unity version). I’m doing this because I’d like to get experience developing and selling on non-Microsoft platforms, and Entangled is a nice small game to start with.

As part of the port, I’m taking the opportunity to re-skin the game, add new levels and mechanics, and possibly some cute bunnies. So – maybe not quite as small anymore.

Hopefully new visuals will aid the appeal of this small game (it has good reviews but very few downloads despite being free).

 

Entangled (Windows Phone)

The same level in the re-skin (rotated)

 

One of the significant things I’m changing involves the color mixing. A significant portion of the gameplay relies on being able to mix and separate the primary colors. In the original version I use additive color mixing (RGB) since you’re mixing laser light; and well, light mixing is additive.

But I think more people are familiar with subtractive color mixing: The stuff you learned in art class when mixing paints. Yellow and blue make green, etc…

With that in mind, I needed to change from lasers to something solid as the medium. So now I have paint or water (not really sure, but it also looks like flames) flying out of gargoyles’ mouths into planters.

 

Art

 

The “laser beam” made of lots of leaf shapes superimposed in Photoshop.

 

Following from this, I’m trying to decorate the world in a kind of garden theme. Vines growing on things, some kind of vegetation in the background, etc…

Now is a time when I’d really like to have an artist working on the project. I don’t think I’m quite prepared for any kind of profit-sharing arrangement though. So for now, I’m doing the art myself. It’s fun in its own right, but a bit of a nuisance when you’re trying to actually make progress on the game.

Right now I’m going with a “jiggly” hand-drawn animated look. The kind of animation where the artist draws the same (static) object many times and these frames are played in sequence. I’m not sure if there is a name for this technique. I haven’t been able to find one – I’ll call it “shimmering” – but it does impart a unique look. It also quadruples (or more) the workload.

 

A leaf texture with four hand-drawn variations of the same image.

 

For art sources, I’m using a combination of the following:

• Completely original hand-drawn stuff (I’m not very good at this)
• Traces of art from a bunch of clip art books I have
• Hand-drawn traces of images of real objects (the leaf above is from a photo that I printed out and then traced).

This is working for the most part, but I’m finding it hard to have a consistent art style throughout the game. I need some kind of art director or something.

 

Purple planter and some leaves (mentioned previously)

 

There’s still a fair amount to do with the art. I have yet to work on the backgrounds. The individual levels now have a sense of place with respect to each other (you pan from one to the other), so it would make sense that the backgrounds impart some sort of movement too (e.g. parallax). So I’m thinking about some vague vegetation/natural environment background layers.

In general most of the art is black and white, since color plays a very important role in the gameplay (this is why the leaves are so pale in the above image). This does make things easier, especially for backgrounds (I think).

One problem with the backgrounds is that any kind of vague static imagery will contrast sharply with the high contrast “shimmering” foreground images. I did experiment a bit with programmatic methods to simulate shimmering for static images, but I wasn’t super satisfied with the results.

 

Mechanics

I’ve introduced a couple of new mechanics, and created another 10 levels for each. I still have a few other ideas for possibly-interesting mechanics.

 

Some of the art for the new mechanics

 

In response to the increased number of levels, I may remove some older levels that aren’t as interesting or are too frustrating (I may also integrate a hint system). It’s very easy for me to create a difficult level, but they usually aren’t very fun. The best levels/puzzles are those that use the fewest mechanics or pieces, but seem impossible at first. The small problem space is a lot less frustrating and results in less trial-and-error solving.

Audio

Given the change in theme, I’m also completely redoing the audio for the game. I don’t plan to have any music soundtrack (though if I find appropriate music for a reasonable price, I’d reconsider). For now I’m going with ambient outdoor noises that I’ve mostly recorded myself. The sound effects I’ve mostly done myself also.

Naming

I’m not quite sure what to name this re-skinned game. I could name it the same (Entangled), but that could get confusing. Given the plant/vine theme, perhaps… something with “tangled”?

I’m hoping to get this shipped within another month. I’m pretty sure I can finish the game in that time, but I’m still unsure how much work the port will be. If I can’t use MonoGame, I’ll have to try Unity. That will mean that I’ll have to essentially re-implement the entire game, not to mention get up to speed on an unfamiliar engine (which isn’t built for 2D in the first place).

Trying out MonoGame

This post (probably the first in a series) will describe my experiences trying to convert an existing XNA project to MonoGame. My intention is to port my (previously WP7) game to iOS, while at the same time keeping a Windows version working. I don’t really care about the Windows Phone version anymore.

My current understanding is that to accomplish the above, I’ll need 3 projects:

• The original VS2010 XNA project (or perhaps just the content project?)
• A VS2010 MonoGame project for Windows
• A project for MonoDevelop on my Mac

I use SVN for source control, and this is likely how I’ll keep files in sync between my PC and my Mac.

My worry is the work required to maintain all these projects through changes. If I’m on iOS and I need to add a texture or something, the workflow will look like:

• On the PC, add the texture file to the XNA content project and rebuild
• Add the resulting xnb file to the Windows MonoGame project
• Commit to SVN
• On Mac, update the project to the latest version on SVN
• Add the new xnb file to the iOS project in MonoDevelop
• Rebuild an deploy to iPad

This is a pretty ridiculously inefficient workflow. It’ll be almost as much work adding a new code file, except that at least I don’t need to leave the Mac to do that. I should be able to add it and push it back up to my repository  However, I’m worried about how adding new files on the Mac will work with VisualSVN (I use VisualSVN (a VS plugin), which automatically adds any project files to the repository. It’s great!)

Installation

There are conflicting descriptions of the necessary prerequisites on the MonoGame site, but this is what I’m trying (I already have VS2010 and XNA 4.0 installed): Windows Phone 7 SDK (needed for content building in MonoGame!) and OpenAL (an audio layer), and of course the MonoGame libraries and VS templates themselves.

I would kind of like to just modify the XNA project files and convert them into a MonoGame project, but I don’t see anyone doing that. The recommended method seems to be to create a new MonoGame project and re-add all your code files manually.

The first issue I encountered is that the MonoGame VS templates didn’t show up, so I could not create a new MonoGame project. This turned out to be a Visual Studio bug, where it was looking the wrong Documents folder (I moved my Documents folder from it’s default location when I set up my machine 6 months ago. Visual Studio still hasn’t realized this).

New MonoGame project

I created a new blank MonoGame (Windows OpenGL) project and successfully built it.

Next, I manually added links to all my game files to it. I’m lucky that this is a very small game, or else this would be really complicated. As it stands, I still have 7 subfolders for my game files, all of which needed to be added manually. Very tedious.

 

 

Next, I had to delete the Game1.cs that was added by default to the MonoGame project and add a link to the real Game1.cs. Then I had to modify Program.cs slightly in order so it used the correct namespace.

I only had a single build error after this, so that’s pretty promising. Apparently MonoGame adds an Input class to Microsoft.Xna.Framework.Input, which resulted in an easy-to-fix name conflict.

Just for kicks, I ran the MonoGame version to see if it would make it as far as the content loading, and it did! Promising so far.

Content

Time to add content. The MonoGame template has a Content folder in it by default. I also added subfolders under that in order to mimic the file structure in my XNA content project.

Then I added all my content xnb files as links. These are of course files generated by building the original XNA project, and exist under the bin\x86\Debug or bin\x86\Release output directories there. I chose to add the xnb files generated in the Debug version. This whole thing is kind of messy, but I believe the MonoGame folks are working on a fix for this.

I’m also going to do some extra work to ensure all these files get added to the subversion repository, but I’ll worry about that later (once I try to get stuff running on Mac/iOS). Needless to say, it will result in some manual labor every time I add a file.

Next, as per some tutorial I saw, I set the properties of all the content file links to Build Action: Content and Copy to Output Directory: Copy if newer. You can select multiple files at once to do this, so it isn’t too bad.

Then, I built and ran the project, and it “worked”!

First run

Well it worked insofar as it didn’t crash. The immediate obvious issues were that some audio didn’t work, and some of the graphics blend modes (and z-order?) seemed to be incorrect.

 

XNA on the left, MonoGame on the right.

Audio problems

At first it looked like SoundEffect.Play was broken, but SoundEffectInstance was fine (since the only sound that worked was the one played through SoundEffectInstance). But upon further investigation, it seems to be dependent on the particular wav file.

The broken ones (which is all but one of my sounds) have a size of 0 in the SoundEffect.Size, despite SoundEffect._data having apparently valid-looking data.

After about an hour of poking around trying to figure out what was unique about the one sound that worked, I stumbled upon it. It was built using “Best” for the compression settings. So it looks like MonoGame doesn’t work with compressed wav files.

I’ve found some mention that you need to use MonoGame-specific content pipeline processors (MonoGame.ContentPipeline.Processors.dll), but there is no such assembly included in the MonoGame install.

Graphics problems

Next it was time to tackle the graphics issues. I was hoping perhaps it was just some uninitialized state of mine, but unfortunately it’s related to the stencil buffer.

In short, MonoGame doesn’t appear to support a stencil buffer for the back buffer. I’m able to create a RenderTarget2D with DepthFormat.Depth24Stencil8, but not the back buffer. I don’t know if this is a bug, or some limitation of OpenGL. To work around it I had to render to an intermediate render target and then to the back buffer. That’s pushing a lot of extra pixels though (a problem when I port to iOS).

An additional bug (or different behavior to XNA) is that the GraphicsDevice.PresentationParameters retain the default settings for BackBufferWidth and BackBufferHeight at the time of the Game.Initialize() method (480 x 800). They only get set to the actual back buffer size after the call to base.Initialize().

After dealing with those issues, things were still not quite right.

 

XNA on the left, MonoGame on the right.

 

The game elements were no longer showing where I told them not to draw (thanks to the stencil buffer working), but the Perlin screenwipe I’m using still wasn’t functioning properly.

A little bit of investigating showed that it was BlendState.ColorWhiteChannels that was essentially being ignored. Again, I’m not sure if this is an OpenGL limitation, an unimplemented MonoGame feature, or a bug. I was able to work around the problem – but, like before, at the expense of pushing more pixels.

Next up…

In my next post, I’ll talk about my experience getting it up and running on iOS. Should be interesting!

Thoughts on Fez

I recently completed a play-through (actually nearly two play-throughs) of Fez, so I thought I’d write a bit about my experience. I’m a big fan of platformer puzzle games in general, so I knew eventually I’d get around to playing this game, and I knew a bit of the history of its development after watching Indie Game: The Movie.

 

A “Blade-runner- esque” region of the Fez world.

 

Opening bugs

First, I’ll start off with the poor “out of box experience” I had. Right off the bat the frame rate drops were extremely noticeable. Chug chug chug. And the first scene had several speech bubbles that keep re-laying themselves out, making it difficult to actually read the text. I can understand the occasional bug like this well within the game, but I have to admit it was pretty surprising seeing this as an opening experience. Usually you try to polish up that nicely.

 

Text re-laying itself out as it floats around the screen.

 

I was playing the trial version, and when it expired it seemed to give me the option to continue my progress in the paid version. I purchased the game (as I had always planned to do), and any progress I’d made seemed to have been lost. However – later it seemed to me that the trial wasn’t actually identical to the beginning of the game, so I guess I can understand this. At the time though, it seemed like another bug.

I was a bit dismayed at all this. However, as I began to play the game, I didn’t notice too many other glaring bugs. I quickly got used to the frame-dropping so it wasn’t so noticeable.

And actually, having played through the game almost twice now, I haven’t hit any deal-breaking bugs. I’ve had the game freeze up the Xbox 3 times (required a hard reboot) – but I suppose over maybe 12 hours of game play that isn’t too bad.

Puzzles

I wasn’t really sure what to expect here. I knew the game was classified as a puzzle-platformer, and had a cool rotation mechanic. But I’d also heard people claim that Fez wasn’t really a game – just a relaxing environment to walk around in.

The truth is somewhere in between. While it isn’t Braid or Portal when it comes to mind-expanding puzzles, it is definitely its own unique “thing” that works.

The rotation mechanic around which the game… revolves… is neat at first. I had a number of “oh cool” moments. During the first hour or so of the game you are exploring this mechanic.

But after that… it fizzles a bit. There were a few spots later in the game where something sort of new happened. I remember a screen with ladders that took a few minutes to puzzle out. And another where you needed to perform a rotation while in mid-air. I had actually been waiting for that puzzle – it seemed like a natural progression (although I think this was on my second play-through, so it wasn’t necessary to complete the game). In general though, you are rarely left contemplating how to get to a particular place to grab a cube (or treasure, or whatever). Things are generally pretty obvious.

Instead, the game throws things at you that are thematically related to the mechanic, but only tangentially related in terms of puzzles. For instance, the platforms that rotate (essentially this just makes them get bigger or smaller). Or the twisty levers that raise and lower. Some are not related at all (platforms that rise and fall on their own). Some do offer some genuine new puzzles (particularly the twisty levers that rotate blocks and put them on a timer when they will rotate back). Some just test your precision timing, such as the platforms that rotate along the axis that goes into the screen.

Overall, the rotate mechanic seems a bit… thin. There isn’t enough depth in it (or else it wasn’t explored thoroughly enough) to fill a game. Luckily Fez “pads” it with enough other interesting content to make it all worth it.

Theme

Fez is held together with a strong theme. Cubes, and the 2d-3d transformations, are pervasive. This really helps tie the game together. The shape of the characters, the way the stars look, the 2nd/3rd level puzzles involving numbers and alphabets, the world map, the inventory screen. Everything just makes sense.

 

The inventory screen is cube-themed and uses the same rotation controls.

 

I think it’s really important for a game to do this. It helps cultivate a sense of “the world”.

The world

The world of Fez is fun to explore. I kept finding little hidden areas. “Missing” areas are generally marked on the world map. You know there is something there, but often you don’t have the skills necessary to successfully explore that particular section. Or you may know there is a room you have yet to explore – but you don’t know how many other rooms may yet branch off from that one. I missed the nearly the entire “water world” section in my first play-through.

 

Just a part of the Fez world map.

 

Most regions are held together with common themes (common puzzle types) and common colors. This helps with navigation, though I still found myself having trouble identifying areas on the world map.

The game definitely encourages exploration, and it is quite big. There is a lot of stuff to uncover.

There are some very nice touches when moving between screens/rooms. I only realized this partway through the game, but when facing the doors you can usually see the place you’ll end up going in the background. You’ll see that screen/room floating hazily in the background, and it really helps give a sense of place (there is also a small preview in a bubble when you hover near a door, but these are often insufficient to identify a region).

 

That door will lead to the level that’s floating in the background.

 

Second play-through

You can finish the game without really finishing it. That is, there is another set of puzzles and lore beyond the ones required to initially complete the game.

Unfortunately these puzzles are very different from the ones associated with the rotation mechanic. They are basically cipher puzzles, and it’s unlikely that most people will solve them without help from the internet (I certainly wasn’t able to). So they are not fun puzzles to solve. They are not puzzles that can be solved step by step. You either get/guess them or you don’t. It’s poor puzzle design, but I suppose its ok given that it’s really only necessary to accomplish new things in the second (or third) play-through.

It’s basically like an extra bonus game – but also something that helps piece together the lore of the world.

Conclusion

Overall, I really enjoyed the game. Evaluated based on its puzzles, I wouldn’t give it a great review – the puzzles were mostly either too easy, or too obscure. But the world the creator of Fez has created manages to fill out the game nicely and make it an enjoyable playing experience.

 

Tank Negotiator – postmortem

Tank Negotiator was released today on XBLIG. This post will just serve as a quick postmortem about the process.

I made this game several years ago as I was first learning XNA and modern 3d graphics. It was pretty much complete, but it sat “shelved” until I brought it back to life earlier this year with the intent of finally releasing it (proving to myself that I could ship a finished game). I did some work, and put it into an official playtest. But I didn’t really push it, and got no feedback.

I forgot about it again until the beginning of November, when I prepared it for an official peer review. It failed shortly thereafter when I was pointed out that I was forcing signed in controllers to take part in the game during network matches (many thanks to @NVOGames on twitter for the network playtesting).

Originally I planned to just fix that one issue and resubmit as soon as possible (you have to wait at least one week).

To properly fix it though, I had to do a a fair amount of redesign around how the network sessions are implemented. It would really have been nice if Microsoft had provided extensive/official guidelines on how to handle various situations. All you have to go on is any existing AAA or XBLA network titles you may have. They all do things differently, and some of them actually wouldn’t meet XBLIG standards!

I was a beginner

A lot of my network and profile signin/signout code is from when I was just getting up to speed on XNA. As such, it isn’t very well thought out. This definitely made it challenging to make fixes without risking regressions. I haven’t given a great deal of thought as to how to re-factor it, but I know it probably should be done.

Performance

As I was finishing up my changes and preparing to submit to peer review again, I noticed that I had some pretty big framerate drops at times. I decided to investigate this, and that turned into about a week of work making performance tweaks (as detailed in a previous blog post). I have improved things a lot, but they’re still not quite where I want them to be.

Visual and polish tweaks

This “forced probation” period also made me take a step back and try to look at the game with new eyes. What areas lacked polish? Were there an easy and non-risky changes I could make that would improve the look? In the end, some of the changes I made were:

• A better loading indicator at the start.
• Revamped the textures for the playing environments. Some of them just looked plain stupid before. I realized that extremely patterned textures didn’t look good on walls, so most of the wall textures have changed to more subtle textures. I also added a very very dark texture set. This entails absolutely no risk of regressions, yet potentially adds a new gameplay variation.
• A new flag model. When I examined the game with “new eyes”, I couldn’t believe how bad my flag model looked! It was completely inconsistent with the rest of the game – very out of place.
• I changed the UI colors a bit, and improved the look of the player selection screen in the lobby.
• The pause screen allowed you to rotate the board and take a look at a “freeze frame” of the gameplay from different angles. But it left the pause menu up, obscuring what you wanted to see. I polished this up a bit, fading out the menu if the user starts to pan around. So this is now an “official feature”. I also put in a small Easter egg here. Can you find it?
• Added support for MSAA and FXAA anti-aliasing methods. Surprisingly these incur almost no performance hit (despite MSAA resulting in predicated tiling).
• Put a QR tag in the credits screen that links to a webpage that provides more detailed info on the gameplay elements.
•  Added a new recorded demo match that demonstrates a larger variety of power-ups.
• I put in a link to the demos from the “Tank Negotiator is a multiplayer game only”. Hopefully this will help drive a few more sales instead of driving people away instantly when they realize they can’t play the game alone.
• I made several of the environments not available in trial mode. When you choose an environment or maze that isn’t available in trial mode, instead of blocking it, I pop up a message box asking if you want to buy the game. Hopefully this provides a bit more of a carrot for people to buy the game.

New features!

I promised myself I wouldn’t add any new features. I wanted to be done with this game, not give myself more of a testing headache. Plus I didn’t have time to organize more playtesting sessions with friends.

But I couldn’t help myself. I did it very carefully though.

First, I made the choice of gameplay environment matter more. Prior to this, the only thing that was affected was the friction of the floor. One environment had less friction and one had more. But most of the environments were identical in terms of gameplay.

Now each environment is unique. The differences can be subtle, but now there is a reason to choose each one other than just the look. Weapons cooldown, bullet speed, and skill recharge time are a few of the things that are affected. I thought this was an easy low-risk change given the new variety it adds.

I did a similar thing with power-ups. I added two new ones : Turbo (increases speed) and Ice Cube (makes weapons cool down more quickly). But I did it so as to absolute minimize risk. These power-ups don’t add any new game logic, they just add a multiplier to use along with existing numerical game constants.

Generally adding new power-ups was a headache because I had to think about how they work in network play. But these new ones are simply “you have the power up or you don’t”. And there is a re-usable piece of code that manages power-up ownership across the network. So I didn’t need to write a single line of network code to add these.

The low risk of these new features is the only reason I felt comfortable adding them.

Any untested code probably has bugs

Most of the networking stack is testable on your own; but some Xbox Live features – like invites – are not. Realizing this, I decoded to work around it by writing code that simulated invites as much as possible. Basically, it would start a new Live network session from any point in the game. I worked out a few kinks this way, but didn’t find one remaining bug until I actually tried out Invites for real.

Automation

During the past few weeks, I spent several days writing automation that runs through a series of tests of the game. These mainly focus on the menus, network sessions and signing players in and out. I have between 20 and 40 tests (depending on how you count them) that probably take an hour total to run. Half of them can be run locally, the other half require 2 PCs and the Xbox.

In the end, I’m not sure the time payoff was worth it. I probably spent more time writing the tests than I would have spent manual testing. On the other hand, the results are probably more reliable; and if I ever need to do another update, these will prove invaluable to help prevent regressions.

Writing the tests does help you consider different scenarios you might not otherwise think about though. And in the end, I found 2 crashing bugs, one XNA bug, 2 pretty serious gameplay bugs, and 2 visual bugs. So overall, it was worth it.

Stress testing

For the actual gameplay, I have stress tests. This is something I wrote a long time ago, not in the past month. But I put them to good use when I finally tested network sessions between 3 devices, and found 2 pretty serious gameplay bugs.

Unit tests

I don’t have any unit tests for Tank Negotiator. I think they probably could come in handy for testing some of the network code though. I found some bugs in the network code only after sometimes more than an hour of the gameplay stress tests. So I suspect there are still some corner cases where things can go wrong.

Peer review

When I submitted the game for peer review again, the most common piece of feedback was “this really needs a single player option”. That’s a very legitimate piece of feedback, and they’re right. But I don’t want to invest the time to do that now.

I worked on it before, and gave up after about of month of hard-core AI work. Any options I have to provide a single player mode are significant investments.

• I could provide full-fledged AI players that try to simulate human player. That’s what I tried before, but it is very non-trivial. It would require months more work to do a good job at it.
• I could provide a campaign mode where you play through different scenarios that teach you about the game. The AI could then be much dumber and easier to implement, because it wouldn’t have to fully simulate a human player. But then I’m basically making a second game. It would be a big investment to make it as polished as the current experience.

I do offer a single-player experience where you can practice against a dumb computer player. It’s so dumb that it doesn’t move or shoot. Someone commented that maybe it should move around and try to kill you – at least a bit. I considered that – but then I feel like I’m giving the message that this is an alternate legitimate gameplay experience in Tank Negotiator. Except it would suck and not be fun. So that’s why I’ve kept the computer player extremely dumb: so you realize this is not the real gameplay (I’ll display a big yellow “Practise Mode” on the screen).

Known bugs that shipped

Bumping into another unit causes them damage. But it doesn’t work in network play. I don’t recall the exact reason (I haven’t gone back and looked at the code in great detail), but I suspect it’s because I was trying to measure player velocity to determine the bump strength. And player velocity can be so erratic if you’re determining it by the remote player’s position deltas. It’s definitely fixable. At the very least, I should have made it clear that it doesn’t work over the network. Oh well. I’ll fix it if I ever need to release an update.

You can get into a bad state if you try to add a guest profile to a network session prior to the profile of whom this is a guest. I think there is an XNA bug here, but I should have coded up a work-around. It would involve some string parsing of gamertags, but I think it should be doable in a robust manner.

Xbox/XNA performance tuning – follow up

I’ve been mainly working on some visual polish for Tank Negotiator, along with finishing the test automation cases. But when I get bored of that sometimes I work on perf, so this is going to be basically another status report on my performance work.

I’m still missing my 60 FPS goal pretty regularly. I don’t think this is a deal-breaker, but I’d like to inch closer if possible.

In my Update cycle, I’ve managed to reduce the collision testing for tanks vs walls down to about 25% of what it used to be. It was frequently eating up about 3ms of time per frame, so this is a good thing. The improvements came solely with how my quad tree was organized, so that I now test an average of 3 wall pieces per movement instead of 12.

I keep wondering if I’m hitting issues with caches misses on the Xbox CPU, given how all my sequentially-accessed data is scattered across memory. But even though I’ve seen expensive pieces of code that have poor memory access patterns, I’ve never realized any improvement by reorganizing the data. Admittedly I’m flying blind here, since there is no way to actually measure cache misses. But I’ve been surprised how – as far as I can tell – this is not a big performance issue.

On the other hand, I have definitely seen performance improvements by manually inlining functions, turning properties into fields and passing structure parameters by ref.

In the Draw cycle, I feel like the number of draw calls has less performance impact than I would expect, while the amount of data I pass has a greater impact than I would expect.

One of the things I draw is bullet holes in the walls. I draw up to 50 of them (50 quads), fading old ones out as new ones are needed. I figured that sending up to 50 quads (200 vertices) to the GPU every frame would be no big deal. But that ends up being typically about 4KB of data. That single Draw call costs 1.3ms of CPU time! I realized a good improvement here by using a DynamicVertexBuffer and only updating it when necessary.

On the other hand, one of the shocking things I found was that I was making separate draw calls for each segment of the lasers (which bounce off walls up to 12 times)! Typically we might be drawing 10 or 20 segments, so that’s 10 or 20 draw calls, and 1 or 2KB of data. Bringing it back down to 1 or 2 draw calls didn’t seem to result in much improvement though.

Another good example of where CPU-to-GPU bandwidth is an issue is a large screen of text I frequently bring up in trial mode. There are about 500 characters in the blurb, and the single SpriteBatch.Draw call that draws them ends up taking 1.5ms. Text is slow! Each glyph quad is 96 bytes of data. So 500 characters ends up being nearly 50KB.

Another area which wasn’t a huge deal, but definitely showed up on the radar, was a place where I was re-fetching the EffectParameter from the Effect each time. In one place I did this about 100 times per frame. This cost about 0.33ms. Stashing the EffectParameters and re-using them brought it down to about 0.25ms. A minor improvement, but a fairly straightforward change.