Building Complex Functional Tests

So you've written a simple functional test and now you want to scale it up to test large parts of your gameplay code.

This tutorial should help you create a suite of simple tests and helper functions that you can compose into complex functional tests for your game.

But first, let's discuss Unreal's two testing frameworks: Functional Tests and Gauntlet.

Functional Tests vs Gauntlet

The Functional Test framework is great for writing small unit tests, and not bad for larger integration tests.

Gauntlet on the other hand, is Unreal's newer testing framework that focuses on testing gameplay rather than individual units.

Some benefits and drawbacks of each:

• Gauntlet only works on full builds, it does not run in the editor. This means every time you update your game and wish to re-run tests, you have to make a full build in order to run the Gauntlet tests.
• Gauntlet tests can be run on any platform; PC, console, mobile. Functional Tests cannot as far as I know.
• Functional Tests work both in-editor and from the command line.
• Because Gauntlet tests run on full builds, they are perfect for tests that measure performance and throw errors when FPS drops below a threshold.

In this tutorial we will focus on using Unreal's Functional Test framework rather than Gauntlet.

Functional Tests for Gameplay Code

In the Introduction to Testing tutorial, we created a simple unit test that instantiated a class, tested it, and cleaned up.

Now we want to test a larger chunk of gameplay code or an entire game using the Functional Testing framework.

Gameplay code does not always execute instantaneously, there are often animations, delayed callbacks and load times. For our tests to work with these delays we will need to change our tests to be able to wait until conditions are fulfilled. Unreal calls these functions latent functions.

Latent Hello World

Previously we could do all our testing in a single function, but now that we need parts of it to wait for gameplay code, we need to change our approach.

AutomationTest.h has a macro ADD_LATENT_AUTOMATION_COMMAND(), that creates a new instance of the class in the parentheses, and adds it to a list of commands to be performed in-order.

We will be using it to make sure that each part of the test is only started once the previous one has finished.

Searching the Unreal codebase for uses of ADD_LATENT_AUTOMATION_COMMAND turns up a lot of interesting things showing how it is used:

// We can make the test wait for a second before calling the next command
const float ActiveDuration = 1.0f;
ADD_LATENT_AUTOMATION_COMMAND( FWaitLatentCommand( ActiveDuration ) );

const float Delay = 1.0f;
ADD_LATENT_AUTOMATION_COMMAND( FDelayedCallbackLatentCommand( [=] {
	// Do something after waiting 1 second
}, Delay ) );

So now that we know how to wait, why don't we just write something like this?

IMPLEMENT_SIMPLE_AUTOMATION_TEST( FBrokenHeroTest, "Example.Broken", EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter )
bool FBrokenHeroTest::RunTest( const FString& Parameters )
{
	ABUIHero* myHero = NewObject<ABUIHero>();
	myHero->DrinkPotion();
	// Wait 10s so we know it's done
	ADD_LATENT_AUTOMATION_COMMAND( FWaitLatentCommand( 10 ) );
	// Check that the state on hero is correct
	// This will fail!
	TestEqual( "Hero should have full HP", Hero->GetHP(), Hero->GetMaxHP() );
}

DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER( FBUISetupHero, ABUIHero*, Hero );
bool FBUIHeroDrinkPotion::Update()
{
	Hero->DrinkPotion();
	return true;
}

DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER( FBUISetupHero, FAutomationTestBase*, Test, ABUIHero*, Hero );
bool FBUITestHeroAtMaxHealth::Update()
{
	Test->TestEqual( "Hero should have full HP", Hero->GetHP(), Hero->GetMaxHP() );
	return true;
}

IMPLEMENT_SIMPLE_AUTOMATION_TEST( FHeroTest, "Example.HeroLatent", EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter )
bool FHeroTest::RunTest( const FString& Parameters )
{
	ADD_LATENT_AUTOMATION_COMMAND( FBUIHeroDrinkPotion() );
	ADD_LATENT_AUTOMATION_COMMAND( FWaitLatentCommand( 10.0f ) );
	ADD_LATENT_AUTOMATION_COMMAND( FBUITestHeroAtMaxHealth() );
	return true;
}

Nuances of Latent Commands

It's worth going into a little more detail on what exactly is happening when we define and run a custom latent command.

In our example below, we use DEFINE_LATENT_AUTOMATION_COMMAND_TWO_PARAMETER to defines a class called FBUIWaitUntilCountMatches, with two member variables, Start and Target. Note that they are member variables in the class FBUIWaitUntilCountMatches, not function parameters.

The Update() function that is implemented immediately after it is called as the test runner executes commands in its queue. If the function returns false it is re-run again from the start, next frame.

Thus the member variable Start is incremented over and over, returning false which prompts the test runner to re-run the latent command again. Until finally Start matches Target and the Update() function returns true;

DEFINE_LATENT_AUTOMATION_COMMAND_TWO_PARAMETER( FBUIWaitUntilCountMatches, int32, Start, const int32, Target );
bool FBUIWaitUntilCountMatches::Update()
{
	Start += 1;
	if ( Start < Target )
		return false;
	return true;
}

This nuance should be clearer when we see what happens with ADD_LATENT_AUTOMATION_COMMAND. It is calling the constructor of FBUIWaitUntilCountMatches, and setting its member variables Start and Target to 0 and 100.

IMPLEMENT_SIMPLE_AUTOMATION_TEST( FBUICustomLatentTest, "BUI.CustomCommands.Simple", TestFlags )
bool FBUICustomLatentTest::RunTest( const FString& Parameters )
{
	ADD_LATENT_AUTOMATION_COMMAND( FBUIWaitUntilCountMatches( 0, 100 ) );
	return true;
}

#define DEFINE_LATENT_AUTOMATION_COMMAND_TWO_PARAMETER(CommandName,ParamType0,ParamName0,ParamType1,ParamName1)	\
class CommandName : public IAutomationLatentCommand \
	{ \
	public: \
	CommandName(ParamType0 InputParam0, ParamType1 InputParam1) \
	: ParamName0(InputParam0) \
	, ParamName1(InputParam1) \
		{} \
		virtual ~CommandName() \
		{} \
		virtual bool Update() override; \
	private: \
	ParamType0 ParamName0; \
	ParamType1 ParamName1; \
}

Composing Complex Functional Tests

Now we understand how the Functional Testing framework supports latent functions, let's talk about how we can build a suite of tests in a smart way.

After a bit of experimentation, I found it helpful to break down testing code into three categories:

• Single tests
• Composite tests
• Helper functions

Single Tests

Single tests are exactly what they sound like, they test a single value. They must be implemented as latent commands, so they can be placed in a queue of commands to be executed.

Continuing with our generic RPG example, these could be some simple tests:

• Test that the hero's health matches an expected value
• Test if the hero is poisoned
• Test if the hero's inventory contains a given item
• Test if the hero is dead
• Test if the hero has spoken to a given NPC

In order to be able to call TestEqual and other test functions, I always pass in a pointer to FAutomationTestBase as the first parameter.

DEFINE_LATENT_AUTOMATION_COMMAND_TWO_PARAMETER( FBUITestHeroGoldEquals, FAutomationTestBase*, Test, ABUIHero*, Hero, int32, ExpectedGold );
bool FBUITestHeroGoldEquals::Update()
{
	// Wait until the hero is alive again rather than failing instantly
	if ( Hero->IsDead() )
		return false;

	Test->TestEqual( "Hero should have specified gold", Hero->GetGold(), ExpectedGold );

	return true;
}

We can call this function in the following way:

IMPLEMENT_SIMPLE_AUTOMATION_TEST( FHeroTest, "Example.Hero", EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter )
bool FHeroTest::RunTest( const FString& Parameters )
{
	ADD_LATENT_AUTOMATION_COMMAND( FBUITestHeroGoldEquals( this, 0 ) );

	return true;
}

Helper Functions

Helper functions are latent commands that perform no tests. They can be useful for setting the game's state before running tests. For our RPG example they could be things like:

• Add items to the player's inventory.
• Create monsters
• Change the player's stats

As before, these need to be defined as latent commands so they are executed in-order after other latent commands. They are implemented in the same way as simple tests, but there is no need to pass in FAutomationTestBase*.

Composite Tests

We can now use our Single Tests and Helper Functions to put together more complex integration tests.

Using the examples from Single Test, we could compose a more complex integration test that checks that the poison system works in the game.

1. Helper — Create a new hero
2. Helper — Shoot the hero with a poison arrow
3. Single — Test that the hero is poisoned
4. Single — Test that the hero has lost health
5. Helper — Wait for 1 seconds
6. Single — Test that the hero has lost health
7. Helper — Wait for 30 seconds
8. Single — Test that the hero is dead

IMPLEMENT_SIMPLE_AUTOMATION_TEST( FBUICustomLatentTest, "BUI.CustomCommands.Simple", TestFlags )
bool FBUICustomLatentTest::RunTest( const FString& Parameters )
{
	ABUIHero* Hero = nullptr;
	ADD_LATENT_AUTOMATION_COMMAND( FBUICreateHero( Hero ) );
	ADD_LATENT_AUTOMATION_COMMAND( FBUIShootHero( Hero ) );
	ADD_LATENT_AUTOMATION_COMMAND( FBUITestHeroStatusEffect( this, Hero,
		EBUIStatusEffect::Poison ) );
	// ... etc
	return true;
}

You can see that as you add more and more Single Tests and Helper Functions, you are creating a library of tools that make it easier and easier to compose complex integration tests.

Code

All of the code for this tutorial can be found on GitHub.

View on GitHub