Test plans

A test plan documents the strategy that will be used to verify and ensure that a product or system meets its design specifications and other requirements. A test plan is usually prepared by or with significant input from Test Engineers.

Depending on the product and the responsibility of the organization to which the test plan applies, a test plan may include one or more of the following:

Design Verification or Compliance test - to be performed during the development or approval stages of the product, typically on a small sample of units.
Manufacturing or Production test - to be performed during preparation or assembly of the product in an ongoing manner for purposes of performance verification and quality control.
Acceptance or Commissioning test - to be performed at the time of delivery or installation of the product.
Service and Repair test - to be performed as required over the service life of the product.
Regression test - to be performed on an existing operational product, to verify that existing functionality didn't get broken when other aspects of the environment are changed (e.g., upgrading the platform on which an existing application runs).
A complex system may have a high level test plan to address the overall requirements and supporting test plans to address the design details of subsystems and components.

Test plan document formats can be as varied as the products and organizations to which they apply. There are three major elements that should be described in the test plan: Test Coverage, Test Methods, and Test Responsibilities. These are also used in a formal test strategy.

Test coverage in the test plan states what requirements will be verified during what stages of the product life. Test Coverage is derived from design specifications and other requirements, such as safety standards or regulatory codes, where each requirement or specification of the design ideally will have one or more corresponding means of verification. Test coverage for different product life stages may overlap, but will not necessarily be exactly the same for all stages. For example, some requirements may be verified during Design Verification test, but not repeated during Acceptance test. Test coverage also feeds back into the design process, since the product may have to be designed to allow test access.

Test methods in the test plan state how test coverage will be implemented. Test methods may be determined by standards, regulatory agencies, or contractual agreement, or may have to be created new. Test methods also specify test equipment to be used in the performance of the tests and establish pass/fail criteria. Test methods used to verify hardware design requirements can range from very simple steps, such as visual inspection, to elaborate test procedures that are documented separately.

Test responsibilities include what organizations will perform the test methods and at each stage of the product life. This allows test organizations to plan, acquire or develop test equipment and other resources necessary to implement the test methods for which they are responsible. Test responsibilities also includes, what data will be collected, and how that data will be stored and reported (often referred to as "deliverables"). One outcome of a successful test plan should be a record or report of the verification of all design specifications and requirements as agreed upon by all parties.

Sony PlayStation2 Game Testing

First of all, SQA testers perform a submission test on title. This submission or 'TRC' check will verify that if the game meets the requirements set by the SCEE/SCEA/SCEI in their Technical Requirements Checklist.
This test run also checks the following elements:


Naming Conventions: this test will cover the 'definitions' and 'namings' in the game in all its different languages to see if there are any violations against chapter "7.0 PlayStation Component Naming Conventions".
Memory Card Operations: every developer / publisher knows that this issue is of critical importance to Sony. SQA testers check all possible memory card messages in all available languages. As a part of this test, they will simulate any user action will be to see if the game reacts correctly to these actions. The messages will be checked for translation, naming convention standards and the correct display of the specified message.
Language check: check if your game is correctly translated into all featured languages.


At the end of the test run, SQA testers send you a detailed report on test results and TRC status.


Besides TRC testing, SQA testers can carry out other functionality tests, such as game play, balance and AI, to make sure that your PlayStation game meets the highest quality standards.

Microsoft XBox Game Testing

tests include:

TCR (Technical Certification Requirements). We will assess the compliance of your game with Xbox requirements. Thus, we will check if your game is ready for submission to Microsoft, helping you save both time and money.

Xbox Live. "Live TCR issues" is a separate chapter in the Xbox Guide . We will check if your game complies with it. The test includes 'Xbox live' requirements, content download, statistics, friends, voice.

Functionality. This test focuses on the general game functionality, and namely:
Menu functionality
General functionality of the buttons, options
Game play and AI
Game stability
Walkthrough


Multiplayer. We will check how your game behaves when played in all featured multiplayer modes. This can vary from multiple players on a single Xbox, system link over to LAN and of course actual online multiplayer sessions.

Microsoft XBox Game Testing

tests include:

TCR (Technical Certification Requirements). We will assess the compliance of your game with Xbox requirements. Thus, we will check if your game is ready for submission to Microsoft, helping you save both time and money.

Xbox Live. "Live TCR issues" is a separate chapter in the Xbox Guide . We will check if your game complies with it. The test includes 'Xbox live' requirements, content download, statistics, friends, voice.

Functionality. This test focuses on the general game functionality, and namely:
Menu functionality
General functionality of the buttons, options
Game play and AI
Game stability
Walkthrough


Multiplayer. We will check how your game behaves when played in all featured multiplayer modes. This can vary from multiple players on a single Xbox, system link over to LAN and of course actual online multiplayer sessions.

Game Testing Methodology

There is no "gold standard" method for game testing, and most methodologies are developed in-house, by individual video game developers and publishers. And, methodologies are refined from time to time and may differ for different types of games (for example, the methodology to test a MMORPG will be different from the testing for a casual game). But outlined below is a typical methodology of game testing.

Identification: Incorrect program behavior is analyzed and identified as a bug.
Reporting: The bug is reported to the developers using a defect tracking system. At minimum, the circumstances of the bug and steps to reproduce are included in the report. Developers sometimes request additional documentation such as real-time video of the bug's manifestation.
Analysis: This step of the process involves the developer responsible for the bug, such as an artist, programmer or game designer. It is outside the scope of the game tester's duties, though if there are inconsistencies in the bug report, the tester may be approached to provide more information or evidence.
Verification: After the developer claims the bug to be fixed, it is the responsibility of the tester to verify that the bug no longer occurs. Not all bugs are addressed by the developer. Reported bugs may be claimed as features (often expressed as "NAB" or "not a bug"), and may also be waived (given permission to be ignored) by producers, game designers, or even lead testers, according to company policy.
Regression testing: As game development progresses, closed bugs are checked again to ensure that they have not regressed, or re-manifested. For example, a programmer might undo a fix, having forgotten why the change was made in the first place. Though tedious, this type of testing is important since unexpected changes to code can cause old problems to resurface in the same or different ways.
Standards testing: Consoles manufacturers such as Nintendo, Sony and Microsoft have standards they require to be met for games for their platforms. Not all manufacturers standards are alike and some testers are dedicated only to ensuring these standards are met.