The new C++ standard, namely C++11, is here at last; offering many additions to the language’s core as well as in the companion library, the STL. Without doubt it will change the way we think and work but nobody can predict if it is for better or worst. The experimentation period is nearly over, only a handful of features missing from GCC and clang and the C++ engineers will have to learn and master the new tricks in both fronts (core and STL). For those familiar with boost library the second front should be an easy transaction to the new STL, for the first though we need tutorials and lots of them. This little article is one tutorial that extends the already published ones.

One of the new cool features is the variadic templates, simply put, templates with variable number of template parameters. To put it into context this is a variadic template:

template<typename... Types>
struct Foo{};

In this article we will not present the basic usage of variadic templates. You can find great publications online (the wikipedia’s to name one) with an introduction and first steps, its not my intention to repeat the same in here. The present article’s purpose is to expand the wikipedia’s one by presenting a few cool things that now are doable. So, before proceeding please take a look at this first http://en.wikipedia.org/wiki/Variadic_template.

Concept 1: Creating a visitor class

I’m not sure how the C++ scientists name that technique but because its pretty useful for all the examples I will call it “three way” technique. This technique has the word “three” because it utilizes three things. A forward declaration, a declaration and a partial template specialization. It’s purpose is to use the variadic parameters to do repetitive things.

This example explains an easy way to implement the visitor pattern. To construct a visitor we use the “three way”:

// Forward declaration (1st part)
template<typename... Types>
struct ConstVisitor;

// Declaration (2nd part)
template<typename First, typename... Types>
struct ConstVisitor<First, Types...>: ConstVisitor<Types...>
{
	using ConstVisitor<Types...>::visit;
	virtual void visit(const First&) = 0;
};

// Specialized for one (3rd part)
template<typename First>
struct ConstVisitor<First>
{
	virtual void visit(const First&) = 0;
};

The declaration of ConstVisitor is recursive (second part) as the class inherits itself. This recursiveness needs to be stopped so we need a specialization for the single template parameter (third part). The forward declaration is there to satisfy the compiler. The above code defines the pure virtual visitor() methods for a number of types. Now lets see the class Base and its derived MyDouble and MyInt that utilize the visitor pattern.

// Forwards
struct MyDouble;
struct MyInt;

/// Base class
struct Base
{
	typedef ConstVisitor<MyDouble, MyInt> MyConstVisitor;

	virtual void accept(MyConstVisitor&) = 0;
};

/// Double
struct MyDouble: Base
{
	double x;

	void accept(Base::MyConstVisitor& v)
	{
		v.visit(*this);
	}
};

/// Int
struct MyInt: Base
{
	int x;

	void accept(Base::MyConstVisitor& v)
	{
		v.visit(*this);
	}
};

/// A visitor that does something
struct PrintVisitor: Base::MyConstVisitor
{
	void visit(const MyDouble& x)
	{
		std::cout << "I am a MyDouble: " << x.x << std::endl;
	}

	void visit(const MyInt& x)
	{
		std::cout << "I am a MyInt: " << x.x << std::endl;
	}
};

The Base::MyConstVisitor is abstract and it contains two pure virtual methods, one for const MyDouble& and the second for const MyInt&. We used the Base::MyConstVisitor to create the PrintVisitor, a visitor that actually does something. The main may look like this:

int main(int, char**)
{
	PrintVisitor vis;
	MyDouble md;
	md.x = 123.456;
	MyInt mi;
	mi.x = 987;

	Base* b = &md;
	b->accept(vis);
	b = &mi;
	b->accept(vis);
}

And the output:

I am a MyDouble: 123.456
I am a MyInt: 987

The “variandic template way” to define a visitor does not do anything super smart or super useful. Its only a good way to save us from typing and nothing more.

Concept 2: Getting the nth type of the variadic template class

Having a variadic template class we want to access the type of the nth template parameter. Once again using the “tree way” technique this is doable.

template<typename... Types>
struct Foo
{
	// Forward declaration
	template<int id, typename... Types_>
	struct X;

	// Declaration
	template<int id, typename First, typename... Types_>
	struct X<id, First, Types_...>: X<id - 1, Types_...>
	{
	};

	// Specialized
	template<typename First, typename... Types_>
	struct X<0, First, Types_...>
	{
		typedef First DataType;
	};

	// Preferable (template aliases) way but we have to wait for GCC 4.7
	// template<int id>
	// using DataType = X<id, Types...>::DataType;

	template<int id>
	struct Y
	{
		typedef typename X<id, Types...>::DataType DataType;
	};
};

int main(int, char**)
{
	typedef Foo<double, int, std::string> MyFoo;
	MyFoo::Y<0>::DataType a(1.23);
	MyFoo::Y<1>::DataType b(123);
	MyFoo::Y<2>::DataType c("Hello");

	std::cout << a << " " << b << " " << c << std::endl;
}

The Foo::X uses the “three way” to define a type, it takes as template argument an integer indicating the position of the template parameter. As you can see this id is getting decreased starting from N and it stops in the specialized Foo::X where the type is defined. In the main() the Foo::Y::DataType defines the correct type, 0 for the first (double), 1 for the second (int) and 2 for the third (std::string).

The output of main is:

1.23 123 Hello

Concept 3: Getting the position of type in variadic template

Rationale: Imagine we have a variadic template class Foo that accepts a number of types and the typedef MyFoo that accepts tree common types. For example:

template<typename... Types>
struct Foo {};

typedef Foo<int, float, std::string> MyFoo;

What we want is to get the type ID of one of the types at compile time. For example the:

std::cout << MyFoo::getTypeId<int>() << " " <<
	MyFoo::getTypeId<float>() << " " <<
	MyFoo::getTypeId<std::string>() <<
	std::endl;

We want to output this:

0 1 2

To do that we will use the “three way” once more like this:

// Forward
template<typename Type, typename... Types>
struct GetVisitableId;

// Declaration
template<typename Type, typename First, typename... Types>
struct GetVisitableId<Type, First, Types...>: GetVisitableId<Type, Types...>
{};

// Specialized
template<typename Type, typename... Types>
struct GetVisitableId<Type, Type, Types...>
{
	static const int ID = sizeof...(Types);
};

This smart structure given a Type and a list of types defines a const integer indicating the Type’s position from the back of the list. The integer is named ID.

The last step is to add the getTypeId() in the Foo:

template<typename... Types>
struct Foo
{
	/// Using the GetVisitableId get the id of the @a T
	template<typename T>
	static constexpr int getTypeId()
	{
		return sizeof...(Types) - GetVisitableId<T, Types...>::ID - 1;
	}
};

Once again with the recursive “tree way” technique we are able to do magic.

Bottom line

After reading this you will be ready for the second part where it implements a variant-like structure. A concept that is complex and not that easy to understand. All the above concepts are needed to create a very fast and abstract class that behaves in almost the same way as the boost::variant.

To put it more simply imagine that we have a container that it lies in the private scope of some class (aka a private member). We want to expose the container’s data in a read/write manner but we dont want to allow access to the container itself. For the user of this class, we grant him access to the nth element of this container but we don’t allow him to push_back or from the container.

class Foo
{
	public:
		typedef std::vector Container;

		Container& getStrings()
		{
			return strs;
		}

		const Container& getStrings() const
		{
			return strs;
		}

	private:
		Container strs;
};

...
Foo foo;
...
foo.getStrings()[0] = "Hello"; // OK!
foo.getStrings().push_back("World"); // Not Acceptable

The example above is a bad example of how to expose a container. It doesn’t feet our needs simply because we give access to the container itself.

After describing what we want to do and how not to do it lets see the correct way. Once again we will use the boost library and more especially the iterator_range module.

#include <boost/range/iterator_range.hpp>

class Foo
{
	public:
		typedef std::vector Container;
		typedef boost::iterator_range<Container::iterator> MutableRange;
		typedef boost::iterator_range<Container::const_iterator> ConstRange;

		MutableRange getStrings()
		{
			return MutableRange(strs.begin(), strs.end());
		}

		ConstRange getStrings() const
		{
			return ConstRange(strs.begin(), strs.end());
		}

	private:
		Container strs;
};

...
Foo foo;
...
// Iterate, OK!
BOOST_FOREACH(std::string& one, foo.getStrings())
{
	std::cout << one << std::endl;
	one += "...";
}

foo.getStrings()[0] = "Hello"; // OK!
foo.getStrings().push_back("World"); // OK! Error compiler

In the above example we managed to do what we wanted. We can access the contents of Foo::strs vector, we can alter them, we can iterate and more importantly we cannot alter the container at all.

At this point you may ask about the overhead of range. Apparently its very lightweight and equal to two iterator copies.

Special thanks to K.K. for creating the original proof of concept.

The concept of multiple threads offers many advantages in applications that execute many uniform jobs at the same time, for example a webserver needs to serve multiple hosts at the same time and without a host waiting for a previous request to finish. A game application though used to have a very standard and linear flow. For example, we first update the AI, then the physics, then we update the world, we do visibility determination and lastly we render, then we repeat the same again. Sometimes its difficult to execute some of these distinctive steps in parallel because the data of the previous step will be used by the next. I bet high class development studios have found ways to blend these steps but in AnKi we use a more simple approach. We use multiple threads to run uniform jobs in parallel.

One good example to illustrate how AnKi uses the power of multiple threads is the visibility determination algorithm. One step of visibility determination is the test for every renderable scene node against the camera’s view frustum. If we have N nodes to test and M threads we can roughly assign for testing the first N/M nodes to the first thread, the next N/M nodes to the next thread etc.

AnKi features a job manager that executes N jobs in parallel in N different threads. One thing to note is that the threads are being created only once (upon the manager’s initialization) and not for every parallel run. This gives an obvious performance advantage (over OpenMP for example) because creating and destroying threads is time consuming and not efficient at all. The job manager works from the main thread like this:

Assign job to first thread
Assign job to second thread
...
Assign job to N thread
Wait for all threads to finish

The implementation of the job manager is pretty simple. Underneath it utilizes the boost threading library (as always) and it uses:

• N condition variables
• N mutexes
• N threads
• One barrier

If you are not familiar with these concepts please refer to a good textbook about parallel programming.

The condition variables are being used for thread waiting. When the thread is not assigned to do a job it waits using a condition variable. Every condition variable comes with a mutex and that’s the reason we need N mutexes. The reason of using N threads is pretty obvious but the usage of the barrier is not. The barrier is being used for thread synchronization. What we synchronize is the N threads along with the main thread.

According to the boost documentation:

When a barrier is created, it is initialized with a thread count N. The first N-1 calls to wait() will all cause their threads to be blocked. The Nth call to wait() will allow all of the waiting threads, including the Nth thread, to be placed in a ready state. The Nth call will also “reset” the barrier such that, if an additional N+1th call is made to wait(), it will be as though this were the first call to wait(); in other words, the N+1th to 2N-1th calls to wait() will cause their threads to be blocked, and the 2Nth call to wait() will allow all of the waiting threads, including the 2Nth thread, to be placed in a ready state and reset the barrier. This functionality allows the same set of N threads to re-use a barrier object to synchronize their execution at multiple points during their execution.

We actually initialize the barrier with N+1, the number of the N threads plus the main thread.

When we say that we assign a job to the job manager what we actually mean is that we pass to the thread a pointer to a function to execute along with a set of arguments. So lets see how we use the job manager in AnKi’s view frustum test:

VisJobParameters jobParameters;
jobParameters.cam = &cam;
jobParameters.skipShadowless = skipShadowless;
jobParameters.visibilityInfo = &storage;
jobParameters.scene = &scene;
jobParameters.msRenderableNodesMtx = &msRenderableNodesMtx;
jobParameters.bsRenderableNodesMtx = &bsRenderableNodesMtx;

for(uint i = 0;
	i < parallel::ManagerSingleton::getInstance().getThreadsNum();
	i++)
{
	parallel::ManagerSingleton::getInstance().assignNewJob(i,
		getRenderableNodesJobCallback, jobParameters);
}

parallel::ManagerSingleton::getInstance().waitForAllJobsToFinish();

The code for waitForAllJobsToFinish() is the simple:

barrier->wait();

Getting the source

You can download the job manager and an example bellow. The build instructions and usage are inside the readme file.

Download the example archive

In this short article we will discuss a case where auto_ptr fails to do what its supposed to do….

How the auto_ptr is being used:

// Begin of block
{
	MyClass* newInstance = new MyClass;
	auto_ptr ptr(newInstance);
}
// End of block

At the end of the block the auto_ptr calls its destructor and checks if it holds a non null pointer. In our case its not null so it deallocates the newInstance.

Now that we roughly know what auto_ptr let us see why its a bad idea to use it at least in GCC. For our test case we have four files:

ClassA.h

#ifndef CLASS_A_H
#define CLASS_A_H

#include <memory>

class ClassB; // Forward declaration of ClassB

class ClassA
{
	public:
		ClassA();
		// Implicit destructor:
		//~ClassA() {}

	private:
		std::auto_ptr<ClassB> bPtr;
};

#endif

In the above declaration we dont include the ClassB.h at all, the compiler wont complain at all because the forward declaration is sufficient for pointers and bPtr is essentially a pointer.

ClassA.cpp

#include "ClassA.h"
#include "ClassB.h"

ClassA::ClassA():
	bPtr(new ClassB)
{}

ClassB.h

#ifndef CLASS_B_H
#define CLASS_B_H

#include <iostream>

class ClassB
{
	public:
		ClassB() {std::cout << "ClassB()" << std::endl;}
		~ClassB() {std::cout << "~ClassB()" << std::endl;}

	private:
		int x;
};

#endif

Main.cpp

#include "ClassA.h"
//#include "ClassB.h"

int main(int, char**)
{
	{
		ClassA a;
	}

	return 0;
}

The output of the above test case is:

ClassB

If we uncomment the second line (#include “ClassB.h”) the output will become:

ClassB
~ClassB

In the first case the destructor of ClassB was not called at all! The “a” instance is deleted but the ClassA::bPtr destructor is never called. Is this an error or what? Actually in Main.cpp there is no mention of the ClassB destructor at all so a simple dealocation happens without calling the destructor. As you can see this is a VERY VERY bad situation where auto_ptr fails to do what its supposed to do. GCC could and should post a warning but even with -Wall and -Wextra enabled it failed to do so.

Boost library once again shows its brilliance and avoids these kind of logical errors with compile time assertions. In our test case we can use a boost::scoped_ptr to ensure that the ~ClassB destructor will be called. A simple replace of auto_ptr will not do the trick as we have to add a destructor in ClassA.cpp or remove the ClassB forward declaration from ClassA.h.

The proper way to do ambient occlusion is very expensive for today’s hardware and especially without the use of a ray-tracing renderer. For that reason a few new techniques developed that tried to produce the same result using simpler and faster algorithms. One of these approaches is the Screen Space Ambient Occlusion (aka SSAO) that makes the calculations of ambient occlusion in 2D space just like a normal 2D filter. This article will not go deep into what SSAO is, there are many great readings around the web that cover the theory behind SSAO, in this article we will jump into the implementation that AnKi uses.

There are quite a few techniques to produce SSAO, a top level way to group them is from the resources they need to produce the SSAO:

• Some use just the depth buffer (Amnesia, Aliens VS Predator)
• others use the depth buffer and the normal buffer (Crysis, gamerendering article)
• and others use the pixel position and the normal buffer (gamedev article)

AnKi used to implement the second technique for quite some time. The results were good but apparently not that good, so for the past couple of weeks I’ve tried to implement the third technique by reading this great article from gamedev. The present article extends the gamedev one by adding a few optimization tips and by presenting the code in GLSL.

Bellow you can see the old and the new SSAO, rendered at the half of the original rendering size with two blurring passes.

Old implementation of SSAO

New implementation of SSAO

The whole scene with the new SSAO

In order to produce the SSAO factor we need practically tree variables for every fragment, the first is the position of the fragment in view or world space (view space in our case), the normal of that fragment and a random vector that we obtain using a noise texture. The fact that AnKi uses a deferred shading renderer gives us the normals of the scene in a texture. The gamedev article suggests that we need to have the view space positions stored in a texture but in AnKi we don’t do that for any reason. Its very expensive to store the position in a texture and for that reason we use a few techniques to obtain the position from the depth buffer.

To obtain the fragment position in view space using the fragments depth we do:

vec3 getPosition(in vec2 uv)
{
	float depth = texture2D(depthTexture, uv).r;

	vec3 fragPosVspace;
	fragPosVspace.z = -planes.y / (planes.x + depth);

	fragPosVspace.xy = (((uv) * 2.0) - 1.0) * limitsOfNearPlane;

	float sc = -fragPosVspace.z / zNear;
	fragPosVspace.xy *= sc;

	return fragPosVspace;
}

The variables:

uv: The texture coordinates to read from

depthTexture: The texture that stores the depth

zNear: Camera’s near plane

zFar: Camera’s far plane

planes: This is an optimization that we use to calculate the fragment’s z. The original and unoptimized code that gives the z is given by:

fragPosVspace.z = -zNear / (zFar - (depth * (zFar - zNear))) * zFar;

if we do a few calculations we can isolate two expressions that can be pre-calculated in the CPU and then passed as uniforms in the shader. The planes 2D vector is (C++):

planes.x() = zFar / (zNear - zFar);
planes.y() = (zFar * zNear) / (zNear -zFar);

limitsOfNearPlane: This is a 2D vector that keeps the limits of the right and top eye vector (C++):

limitsOfNearPlane.y() = zNear * tan(0.5 * fovY);
limitsOfNearPlane.x() = limitsOfNearPlane.y() * (fovX / fovY);

fovX is the horizontal angle (imagine the cam positioned in the default OGL pos). Note that fovX > fovY in the classic resolutions where the width > height

As may have already noticed by now the getPosition is used to calculate the position using a perspective camera.

The SSAO vertex shader:

layout(location = 0) in vec2 position;

out vec2 vTexCoords;

layout(location = 0) in vec2 position;

out vec2 vTexCoords;

void main()
{
	vec2 vertPos = position;
	vTexCoords = vertPos;
	vec2 vertPosNdc = vertPos * 2.0 - 1.0;
	gl_Position = vec4(vertPosNdc, 0.0, 1.0);
}

The position is the vertex coordinates of the quad, the coordinates are: {1.0, 1.0}, {0.0, 1.0}, {0.0, 0.0}, {1.0, 0.0}. The values are pretty easy to digest and they help to calculate the texture coordinates. With this way we don’t pass a separate vertex attribute for the texture coordinates.

Fragment shader:

/// @name Uniforms
/// @{
uniform vec2 planes; ///< for the calculation of frag pos in view space
uniform vec2 limitsOfNearPlane; ///< for the calculation of frag pos in view space
uniform float zNear; ///< for the calculation of frag pos in view space
uniform sampler2D msDepthFai; ///< for the calculation of frag pos in view space

uniform sampler2D noiseMap; /// Used in getRandom
uniform float noiseMapSize = 100.0; /// Used in getRandom
uniform vec2 screenSize; /// Used in getRandom

uniform sampler2D msNormalFai; /// Used in getNormal
/// @}

/// @name Varyings
/// @{
in vec2 vTexCoords;
/// @}

/// @name Output
/// @{
layout(location = 0) out float fColor;
/// @}

/// @name Consts
/// @{
uniform float SAMPLE_RAD = 0.1;  /// Used in main
uniform float SCALE = 1.0; /// Used in doAmbientOcclusion
uniform float INTENSITY = 3.0; /// Used in doAmbientOcclusion
uniform float BIAS = 0.00; /// Used in doAmbientOcclusion
/// @}

/// globals: msNormalFai
vec3 getNormal(in vec2 uv)
{
	return unpackNormal(texture2D(msNormalFai, uv).rg);
}

/// globals: noiseMap, screenSize, noiseMapSize
vec2 getRandom(in vec2 uv)
{
	return normalize(texture2D(noiseMap, screenSize * uv / noiseMapSize).xy * 2.0 - 1.0);
}

/// Get frag position in view space
/// globals: msDepthFai, planes, zNear, limitsOfNearPlane
vec3 getPosition(in vec2 uv)
{
	float depth = texture2D(msDepthFai, uv).r;

	vec3 fragPosVspace;
	fragPosVspace.z = -planes.y / (planes.x + depth);

	fragPosVspace.xy = (((uv) * 2.0) - 1.0) * limitsOfNearPlane;

	float sc = -fragPosVspace.z / zNear;
	fragPosVspace.xy *= sc;

	return fragPosVspace;
}

/// Calculate the ambient occlusion factor
float doAmbientOcclusion(in vec2 tcoord, in vec2 uv, in vec3 original, in vec3 cnorm)
{
	vec3 newp = getPosition(tcoord + uv);
	vec3 diff = newp - original;
	vec3 v = normalize(diff);
	float d = length(diff) /* * SCALE*/;

	float ret = max(0.0, dot(cnorm, v) /* - BIAS*/) * (INTENSITY / (1.0 + d));
	return ret;
}

void main(void)
{
	const vec2 KERNEL[16] = vec2[](vec2(0.53812504, 0.18565957), vec2(0.13790712, 0.24864247), vec2(0.33715037, 0.56794053), vec2(-0.6999805, -0.04511441), vec2(0.06896307, -0.15983082), vec2(0.056099437, 0.006954967), vec2(-0.014653638, 0.14027752), vec2(0.010019933, -0.1924225), vec2(-0.35775623, -0.5301969), vec2(-0.3169221, 0.106360726), vec2(0.010350345, -0.58698344), vec2(-0.08972908, -0.49408212), vec2(0.7119986, -0.0154690035), vec2(-0.053382345, 0.059675813), vec2(0.035267662, -0.063188605), vec2(-0.47761092, 0.2847911));

	vec3 p = getPosition(vTexCoords);
	vec3 n = getNormal(vTexCoords);
	vec2 rand = getRandom(vTexCoords);

	fColor = 0.0;

	const int ITERATIONS = 16;
	for(int j = 0; j < ITERATIONS; ++j)
	{
		vec2 coord = reflect(KERNEL[j], rand) * SAMPLE_RAD;
		fColor += doAmbientOcclusion(vTexCoords, coord, p, n);
	}

	fColor = 1.0 - fColor / ITERATIONS;
}

• The noiseMapSize is the size of the noise map. The width and height of the noise map is the same
• The screenSize is the width and height of the window. This value is not that relevant and it doesn’t affect the result that much
• The KERNEL is a number of 2D coordinates that form a circle around (0, 0) and we use them for sampling
• The unpackNormal is a function that unpacks the normal from two 16bit floats, dont be alerted by this function its internal

First of all lets see what logging is. Logger is a subsystem responsible of keeping a record of the various engine events, to inform about problems and eventually help us (the programmers) diagnose them.

A simple logging method is to use the printf function for writing in the standard output:

// Report an error
printf("Error: %s\n", "Something went wrong");
// ...or inform the user of a certain event:
printf("Renderer initialized successfully!\n");

The simple printf is more than inadequate. A proper logger should have the following attributes:

• Data abstraction
• Minimal dependences
• Only one instance & self sustained
• Multiple outputs (VERY VERY INTERESTING)
• Error tolerant

Data abstraction

The logger should be able to take as input various data types and be able to process them without much trouble from our side. The std::cout is a perfect example. cout can output C strings, STL strings, floats, ints and other standard data types. So a its a good idea for a logger to have the same functionality as the STL ostream.

// Bad and slow (thanks to boost::lexical_cast)
Logger::getInstance().write("The current line is: " + boost::lexical_cast<std::string>(__LINE__));
// Good
Logger::getInstance() << "The current line is: " << __LINE__ << endl;

Also note that the ostream’s way is much faster because we can use a preallocated buffer instead of invoking malloc to concatenate the STL strings all the time.

Minimal dependences

The logger is an essential subsystem that practically every other subsystem uses. It’s a very bad idea for the logger to depend on other engine subsystems like the renderer or some global variables. The cross dependencies across subsystems should be avoided when possible and especially for logging.

Only one instance & self sustained

In an application only one logger should exist. To ensure the uniqueness we can use the singleton pattern or create a global instance. As we all know globals should be avoided when possible and for that reason the AnKi logger is a singleton class. Also the logger cannot be part of any other subsystem because this makes the unit testing difficult.

In the previous version the logger was member of the global application class and it was initialized by the application. For testing, if a subsystem wanted to use only the logger it had to initialize the whole application. Thats a bad practice for our simple unit tests because it renders the code pretty untestable. For unit testing the preferable practice is to have many independent subsystems with very little dependences with each other.

Multiple outputs

In a game engine the logs should be outputted in various locations. The output can be the standard output (or standard error), the in-game console (eg quake console), a pop-up message box, in a log file(s) or in all of these at the same time. As we already mentioned before the logger should not have any dependencies so a question is how to write to the in-game console, for example, without invoking the renderer? The answer is to use the observer pattern. For those who have worked with a signal/slot framework (eg Qt) this introduction is pointless but for those who haven’t lets explain how AnKi uses it.

The logger has a signal data member, if another subsystem wants to know the logger’s output it can simply “connect” to this signal. When the logger is ready output a message (eg when it flushes) it emits the signal with the message as a parameter. The subsystems that are “connected” to this signal will get the message and handle it according to their needs.

// Connect the signal with a method...
Logger::getInstance().getSignal().connect(boost::bind(&App::handleMessageHanlderMsgs, this, _1, _2, _3, _4));
// ... and the method just outputs to the stdout
void App::handleMessageHanlderMsgs(const char* file, int line, const char* func, const std::string& msg)
{
	std::cout << "(" << file << ":" << line << " "<< func << ") " << msg << std::endl;
}

Error tolerant

The logger provides the means for error reporting but what happens if the logger itself needs to report an internal error? Its very stupid for the logger to use itself to for reporting, the reason is pretty obvious I guess. The only solution is to be fault tolerant or use assertions. For example, in AnKi when we have a buffer overflow we flush the already filled buffer and continue from the top. In another case when the logger’s input is very big and exceeds the logger’s internal buffer (more than 512 bytes!!??) we simply ignore the message.

Conclusion & references

The logger in AnKi is very clean and abstract. One of the disadvantages it that the signals and slots add a certain overhead but the messages in the main loop should be kept to minimum either way.

AnKi logger header file

AnKi logger source file

Boost signal library

Qt signals/slots

After a few weeks of coding I’ve moved the renderer from OpenGL 2.something to OpenGL 3.3 core. This process proved to be a not so easy task mainly because I had to modify a few shaders and change the way the meshes get rendered. One thing to note about 3.3 is the way of rendering geometry. It doesn’t support immediate mode ofcource, it doesnt support vertex arrays and it doesnt support the use of vertex buffer objects (VBOs) outside vertex array objects (VAOs). The changes in the shaders were not that difficult. The modifications were to remove the varyings and replace them with in/out variables, remove the gl_FragColor and add a few layouts for some attributes. The shaders worked without causing major headaches.

With the transition to 3.3 core I found an opportunity to change the organisation of meshes and materials. The meshes now contain only the VBOs and the materials do not have any information about the depth passes. A mesh can now have multiple compatible materials. The container that has meshes, materials, skeletons and animations is called model.

The work in the renderer is not yet complete. I have to rewrite the debug rendering stage and for that I need think of an optimal way to draw debug primitives (mainly lines).

XML files

One more thing is the new file format for materials. I’ve moved away from the custom formats and its now the materials are in XML. To parse the XML AnKi uses the boost property_tree container. The model files are in XML as well and only the lights are left to be done.

Future

For the future AnKi has quite a lengthy TODO list

• Modify the hardware skinning by implement texture buffers and use them as bone transformations in the shaders
• Optimize the animation by the means of transform feedback (details will follow)
• Finish the shredder tool that splits meshes in little octree cubes
• Implement animation blending

This is an example of a material file:

<material>
	<shaderProg>
		<customMsSProg>
			<defines>
				<define>DIFFUSE_MAPPING</define>
				<define>NORMAL_MAPPING</define>
				<define>ENVIRONMENT_MAPPING</define>
			</defines>
		</customMsSProg>
	</shaderProg>

	<userDefinedVars>
		<userDefinedVar>
			<name>diffuseMap</name>
			<value>
				<texture>meshes/horse/horse.diff.png</texture>
			</value>
		</userDefinedVar>

		<userDefinedVar>
			<name>normalMap</name>
			<value>
				<texture>meshes/horse/horse.norm.png</texture>
			</value>
		</userDefinedVar>

		<userDefinedVar>
			<name>environmentMap</name>
			<value>
				<texture>textures/env/env_map_3.png</texture>
			</value>
		</userDefinedVar>

		<userDefinedVar>
			<name>specularCol</name>
			<value>
				<vec3><x>1.0</x><y>2.0</y><z>-0.8</z></vec3>
			</value>
		</userDefinedVar>

		<userDefinedVar>
			<name>diffuseCol</name>
			<value>
				<vec3><x>1.0</x><y>1.0</y><z>1.0</z></vec3>
			</value>
		</userDefinedVar>

		<userDefinedVar>
			<name>shininess</name>
			<value>
				<float>90.0</float>
			</value>
		</userDefinedVar>
	</userDefinedVars>
</material>

AnKi, just like every other game engine, needed a scripting language and the searching/prototyping started almost a year back. We have made prototypes for lua, javascript (V8 and SpiderMonkey), squirrel, Qt script, Python and others. Lets see why we chose boost python over others…

Key elements of choosing a scripting language

Lets analyze what are the key elements of choosing a scripting language for a game:

• Glue code. The embedding is a process that, unfortunately, comes with additional development cost. The developer needs to expose (aka wrap) classes and functions by writing glue code. This is not an easy task and it varies from API to API. For example squirrel was designed to be an embedded language in games and its fairly easy to write glue code. Python on the other hand is not. Using python C API is very unproductive.
• Speed. Some interpreter implementations offer faster responsiveness over others. For example Google’s V8 engine offers JIT (just in time) compilation, a feature that made Google’s Chrome one of the fastest browsers out there.
• Platform support. The language binaries/libraries should be portable among popular gaming platforms like PC and consoles. In most cases this is not a problem because most of the interpreters are written in C/C++ which makes them fairly portable. There is a big “but” though. Some interpreters support JIT compilation, a feature that implies generation of machine dependent code at runtime. One example is Google’s V8 again. In order to use V8 you have to build only 32bit executables of your application. Ok that not a big deal, the big deal is that the generated  code is for x86 and ARM platforms and it wont run in console hardware without major modifications.
• Popularity: Being a popular language means that there are lots of tutorials, documentation and people who can help you. Some languages are fairly new like Squirrel and they lack excessive support. Its a risk to use a language and have to switch later because of lack of support.

Pros and cons of the tested implementations

Lets see what are the pros and cons of all these implementations:

• Lua: Fast, small memory footprint, very popular among game developers, moderate glue code writing, weird syntax (IMHO)
• Qt script: Speed is undetermined, javascript, zero usage among game developers, very easy to write glue code, difficult to port to consoles, Qt dependent
• V8: Smoking fast, javascript, moderate writing of glue code, non portable, bad documentation
• SpiderMonkey family: Fast to smoking fast, javascript, moderate writing of glue code, the SpiderMonkey is portable, the TraceMonkey is portable but the JägerMonkey may not be due to JIT compiler, good documentation
• Squirrel: Speed is undetermined, glue code writing is easy, designed for game embedding, not that many developers behind it
• Python: Not the fastest around, huge in non-game developers and popular among game developers, lots of glue code when using C API but very very easy with boost python

*Note: As we can see javascript is very popular with various implementations but for games it may be a no go due to luck of support of operator overloading. The math library in AnKi uses operator overloading and to map this in javascripts requires different notations. It may be ugly.

Why boost python?

After creating (and still creating) prototypes we choose boost python. Boost python is a very smart wrapper on top of Python’s C API that makes the writing of glue code very very easy. Python is the most popular scripting language around, it may be slower than lua but for the py3k (Python 3.x) there is a JIT compiler under development. On the downside, boost python is template oriented and most of the time you find yourself in front of compiler errors with obfuscated cause. Also the documentation is lucking.

References

http://squirrel-lang.org
http://www.boost.org/doc/libs/1_44_0/libs/python/doc/index.html
http://code.google.com/apis/v8/embed.html
https://developer.mozilla.org/en/JavaScript_C_Engine_Embedder%27s_Guide

For that reason we’ve created a versatile class called BinaryStream. Binary stream inherits the well known C++ iostream which allows the class to behave as a text stream as well. On top of that there are a few methods that extract binary data from binary files. It can extract unsigned integers (32bit), floats (32bit) and special strings. One thing to note about the numbers is the endianness also known as byte order. Byte order is a common problem when it comes to memory representation of numbers. BinaryStream class addresses this issue by taking into account the file’s byte order as well as the machine’s byte order. The strings are a special case of strings that resemble Pascal strings. The first 32bits are the string’s length and the rest is the following characters.

AnKi binary files also contain a header before everything. The header is an 8byte identifier and it ensures that we are reading the right file type. For example meshes have the “ANKIMESH” and the skeletons the “ANKISKEL”.

Here is a sample of how we use the BinaryStream in mesh loading:

// Open the file
std::fstream file(filename, std::fstream::in | std::fstream::binary);

if(!file.is_open())
{
	throw EXCEPTION("Cannot open file \"" + filename + "\"");
}

BinaryStream bs(file.rdbuf());

// Magic word
char magic[8];
bs.read(magic, 8);
if(bs.fail() || memcmp(magic, "ANKIMESH", 8))
{
	throw MESH_EXCEPTION("Incorrect magic word");
}

// Mesh name
std::string meshName = bs.readString();

// Material name
std::string materialName = bs.readString();
if(materialName.length() > 0)
{
	material.loadRsrc(materialName.c_str());
}

// Verts num
uint vertsNum = bs.readUint();
vertCoords.resize(vertsNum);

If you want suggestions of how to load binary files check the BinaryStream in the SVN repository

The C-like way of error handling

The main idea is to return true on success and if an error happens it will be reported by the function that caught it first.

bool doSomething(...)
{
	...
	if(error)
	{
		cerr << "Something went wrong" << endl;
		return false;
	}
	...
	return true;
}

If someone calls the function “doSomething” it needs to check the return flag to check is an error happened. It is possible to report the error again and have a nice backtrace of the error.

Error (Texture.cpp:126 createEmpty2D): OpenGL Err: invalid value
Fatal (Ms.cpp:28 init): Failed to create one MS FAI. See prev error. Bye!

In the above example the function createEmpty2D failed for some reason, reported the error and returned false. The init function, who is the caller of the createEmpty2D, caught the false and re-reported the error. With this simple trick we have a way to keep a backtrace and immediately we know who failed and who started it.

Exceptions

For some subsystems this C-like way of handling errors is ok but its very dirty for library-like subsystems. Exceptions is a nice alternative. We throw an exception and the caller is responsible of handling it. I wont go on with details of how exceptions work, this is not a tutorial but this is how doSomething looks like:

void doSomething(...)
{
	...
	if(error)
	{
		throw std::runtime_error("Something went wrong");
	}
	...
}

Obviously the code is less cause you save the returns, also you wont find yourself in the unpleasant position of forgetting a return or returning the incorrect boolean. There is an issue of a certain overhead when using exceptions. Lets see how much overhead exceptions add.

I’ve created a simple example and tried to read the assembly code.

try
{
	printf("in\n");
	int r = rand() % 2 + 1;
	doSomething(r);
	printf("out\n");
}
catch(std::exception& e)
{
	printf("in\n");
	fprintf(stderr, "%s\n", e.what());
	printf("out\n");
}

The assembly:

			printf("in\n");
0x000000000040263a <main+42>:  mov    $0x405a82,%edi
0x000000000040263f <main+47>:  callq  0x402280 <puts@plt>
			int r = rand() % 2 + 1;
0x0000000000402644 <main+52>:  callq  0x402040 <rand@plt>
0x0000000000402649 <main+57>:  mov    %eax,%edx
0x000000000040264b <main+59>:  sar    $0x1f,%edx
0x000000000040264e <main+62>:  shr    $0x1f,%edx
0x0000000000402651 <main+65>:  add    %edx,%eax
0x0000000000402653 <main+67>:  and    $0x1,%eax
0x0000000000402656 <main+70>:  sub    %edx,%eax
0x0000000000402658 <main+72>:  add    $0x1,%eax
0x000000000040265b <main+75>:  mov    %eax,-0x18(%rbp)
			doSomething(r);
0x000000000040265e <main+78>:  mov    -0x18(%rbp),%eax
0x0000000000402661 <main+81>:  mov    %eax,%edi
0x0000000000402663 <main+83>:  callq  0x402494 <_Z11doSomethingi>
			printf("out\n");
0x0000000000402668 <main+88>:  mov    $0x405a85,%edi
0x000000000040266d <main+93>:  callq  0x402280 <puts@plt>
0x0000000000402672 <main+98>:  jmpq   0x4026f7 <main+231>
0x0000000000402677 <main+103>: cmp    $0x1,%rdx
0x000000000040267b <main+107>: je     0x402685 <main+117>
0x000000000040267d <main+109>: mov    %rax,%rdi
0x0000000000402680 <main+112>: callq  0x402330 <_Unwind_Resume@plt>
		catch(std::exception& e)
0x0000000000402685 <main+117>: mov    %rax,%rdi
0x0000000000402688 <main+120>: callq  0x402060 <__cxa_begin_catch@plt>
0x000000000040268d <main+125>: mov    %rax,-0x20(%rbp)
0x00000000004026d8 <main+200>: callq  0x4022c0 <__cxa_end_catch@plt>
0x00000000004026dd <main+205>: jmp    0x4026f7 <main+231>
0x00000000004026df <main+207>: mov    %edx,%ebx
0x00000000004026e1 <main+209>: mov    %rax,%r12
0x00000000004026e4 <main+212>: callq  0x4022c0 <__cxa_end_catch@plt>
0x00000000004026e9 <main+217>: mov    %r12,%rax
0x00000000004026ec <main+220>: movslq %ebx,%rdx
0x00000000004026ef <main+223>: mov    %rax,%rdi
0x00000000004026f2 <main+226>: callq  0x402330 <_Unwind_Resume@plt>
			printf("in\n");
0x0000000000402691 <main+129>: mov    $0x405a82,%edi
0x0000000000402696 <main+134>: callq  0x402280 <puts@plt>
			fprintf(stderr, "%s\n", e.what());
0x000000000040269b <main+139>: mov    -0x20(%rbp),%rax
0x000000000040269f <main+143>: mov    (%rax),%rax
0x00000000004026a2 <main+146>: add    $0x10,%rax
0x00000000004026a6 <main+150>: mov    (%rax),%rdx
0x00000000004026a9 <main+153>: mov    -0x20(%rbp),%rax
0x00000000004026ad <main+157>: mov    %rax,%rdi
0x00000000004026b0 <main+160>: callq  *%rdx
0x00000000004026b2 <main+162>: mov    %rax,%rdx
0x00000000004026b5 <main+165>: mov    0x205ddc(%rip),%rax        # 0x608498 <stderr@@GLIBC_2.2.5>
0x00000000004026bc <main+172>: mov    $0x405a89,%esi
0x00000000004026c1 <main+177>: mov    %rax,%rdi
0x00000000004026c4 <main+180>: mov    $0x0,%eax
0x00000000004026c9 <main+185>: callq  0x402240 <fprintf@plt>
			printf("out\n");
0x00000000004026ce <main+190>: mov    $0x405a85,%edi
0x00000000004026d3 <main+195>: callq  0x402280 <puts@plt>

For doSomething there is practically no difference, its like a normal call. The try branch does not add any overhead, nevertheless, the catch does.

Lets try another test case. We have the two different samples to compare them. Note that in both cases we wont get an exception, this means that we will never go into a catch block.

for(int i=0; i<BIG_NUMBER; i++)
{
	try
	{
		int r = rand() % 2 + 1;
		doSomething(r);
	}
	catch(std::exception& e)
	{
		fprintf(stderr, "%s\n", e.what());
		break;
	}
}

try
{
	for(int i=0; i<BIG_NUMBER; i++)
	{
		int r = rand() % 2 + 1;
		doSomething(r);
	}
}
catch(std::exception& e)
{
	fprintf(stderr, "%s\n", e.what());
}

The thing to now is where the try/catch block is located. In the first case we have the try/catch inside the loop and in the second outside. After some benchmarks we saw that practically there is no speed difference. This indicates that the try block does not add any significant overhead.

Conclusion

In AnKi we dont care to recover from an error. The only thing that matters is to report the error correctly and nothing else. Exceptions make the code more simple, I guess, and in our case they dont add any noticeable overhead.