Designer Diary: Focus Testing
Author: Tim McCracken, Quality Assurance Manager, Firaxis Games

It’s an understatement to say a great deal of planning went into the production of Sid Meier's Pirates! for the Xbox. With an aggressive deadline, features were measured against production time. The objective wasn’t reinventing the game, rather ensuring the Xbox version had it’s own unique style and delivered a great Xbox gaming experience. A key device in judging the success of our efforts was focus testing. Gamers were brought into our studio to play, while we observed their decision making, actions, and reactions. Across several months, the game was fine tuned in response to these observations.

Our testers ranged from casual to hard core gamers because we wanted to make a game that would be fun entertainment for all types of players.. Before controllers were handed out, each participant would answer a few questions to rate their interests, game genre preferences, and experience.

Single player and multiplayer were the two types of focus tests held. Single player concentrated on the feedback of one tester over a minimum of an hour long play session. Multiplayer sessions recorded the feedback of a group of players testing the Vs. Ship Battle game. While single player sessions were focused and engaging, multiplayer awoke the participant's primal, competitive nature. Every session roared with rowdy cheers, reminiscent of the raucous heard along the shores of the 17th century Caribbean.

Speaking out loud was encouraged, sometimes prodded. When 'speaking out loud', a focus tester announces aloud why they are doing something. This approach helped testers to streamline their thoughts, giving us a glimpse into the intuitiveness and balance of the game's design. I gave them the example ofwalking through my house counting windows. Rather than simply telling someone the number of windows on each floor, I would describe walking through each room; vividly painting a picture of the setting and the process.

Rather than immediately answering questions on the interface, game mechanics, or controls, we would document the questions and answer them only if the player reached an impasse. At the end of each session a questionnaire was handed out asking the following: What did they or did they not enjoy? Was the game intuitive? Was anything confusing? To ensure impartiality, the subjects documented their impressions anonymously.

At first, everyone experienced similar difficulties. As this information was documented and passed on to the design team, changes were made and a new focus group was brought in to test the new version

Fans of Pirates! PC were pleased to see the addition of the button matching game for evening the odds while boarding a ship. They liked the smaller world map, promising a fast paced, action packed game. Using the Leaderboard on Xbox live for comparing fame scores was also a highlight.

The introduction of multiplayer was heartily welcomed. This head to head experience features a rarely seen environment. The physics of the open sea and dynamics of the wind, create a unique battle field. Properly timing a broadside, cutting the sails for a tight dodge, and ferociously racing for a power up, are all necessary skills for survival.

With the statistics system recording a detailed log of each game, competition became fierce. At first, ship battles were simply about winning - sinking your opponent and remaining afloat. After a series of matches, however, victory was not enough. Players were not satisfied until they crushed their opponents, watching the enemy ships splinter apart, and hearing the lamentation of the crew.

With multiple maps and ships to choose from, egos were put on the line as the superior captain was determined. Who could defeat the enemy with the most accuracy? Victories with one ship in one arena did not guarantee the same results in different settings. Teams were formed and rivalries intensified even more. The combatants were not eager to leave the helm - when the focus test ended it would be months before they could return…

After witnessing the endless enthusiasm and genuine excitement of our onsite focus testers, I'm eager to see how the game is received in the Xbox community at large. I'll personally leave a massive wake of splintered ships and defeated crewmen come the game's release, and further the legend of my own infamous character, Captain Jack Kidd!