6 Part 6: V-Modell Reference Activities

6.3 Activities

6.3.5 Evaluation

6.3.5.7 Preparing Evaluation Specification Usability

Purpose

The preparation of the »Evaluation Specification Usability will start during the »System Specification with the definition of scenarios. In this definition a complex task setting of a user in a defined user role shall be described. It will consist of a number of associated use cases or test items that describe the overall scenario.

Such a use case is defined by

• an Identifier characterizing the use case,
• the description of a use situation, in which the user is in his role defined by the scenario at the dialog workstation during the operation of the application system;
• the description of a work task the user is to perform in the described use situation at the dialog workstation;
• the description of a test target that specifies what has to be achieved or tested with the use case and
• the description of discussion points between supplier and user.

Thus the possibility that typical work tasks can be realized by the prototypes developed within the framework of the accompanying human factors engineering will be traceable and can be tested. The prototypes have to be iteratively evaluated and tested with representatives of the users. The results of prototype development should be taken into account in the specification and, if possible, as early as in the requirements specification.

In the following one use case will be outlined as an example.

• Identifier: Prepare new order from the archive.
• Use situation: A new order is transmitted to the user by the central office. He remembers that a similar order exists already in the centrally available archives that contain the orders.
• Work task:

• • Loading the appropriate order for the user from the central archives.
  • Making changes/amendments based on the transmitted parameters of the loaded order from the archives.
  • Starting the new order.

• Test target: Analyze the dialogue order-archive, analyze the navigation in categories, execute activities for amending the order, start the function Order from the dialogue "Order Archives".
• Discussion points between supplier and user: Is the structuring of the archives in categories sufficient, is it necessary to have for each order an explicit name that can be freely specified by the user or is a search function required?

In addition to the user cases, usability tests shall be established and analyzed. The aim of early usability tests is to familiarize the user with the prototypes and to give them an initial realistic impression of the dialogues of the workstation. The usability tests shall be designed in a way that they can be performed at the workstation that will be set up as a prototype under working conditions that will be as realistic as possible. For all tests the test evaluation procedures shall be described.

The usability tests should be specified in a way that the documented evaluation results can be used in the further course of interface implementation.

6.3.5.7.1 Specifying Evaluation Strategy

The evaluation strategy shall be derived from and refined based on the »User Tasks Analysis and the specified general conditions. Afterwards, it shall be documented in the »Evaluation Specification Usability.

Evaluation requirements shall be prepared for every evaluation object listed in the evaluation plan. If relevant, the connection between the evaluation requirements and requirements documents shall be demonstrated.

The evaluation case structure, i.e., the fundamental structure of each evaluation case, shall be specified.

Depending on the case, also safety and security shall be taken into account. Thus, the evaluation method will be based on the safety and security level/action matrix specified in the »Project Manual and the criticalities specified for the respective functional unit.

6.3.5.7.2 Deriving Evaluation Cases

The evaluation cases of the individual evaluation objects shall be established on the basis of the evaluation requirements in the evaluation specification. For each evaluation case the coverage matrix of the evaluation specification shall describe what architectural elements and interfaces and what requirements are verified.

The structure of the evaluation case should be based on the evaluation strategy in accordance with the definitions of the evaluation case structure. For each input value the expected specified reaction shall be described.

When establishing a system, the evaluation cases generated in the self-check shall be considered. As far as this is required, these evaluation cases shall be supplemented or modified.

6.3.5.7.3 Allocating Evaluation Cases to Requirements

Each specified evaluation case shall be allocated to the requirement from which it was derived. This shall be documented in the coverage matrix of the evaluation specification. Frequently several evaluation cases, e. g. good behavior or various exceptional behaviors, will be specified and allocated to one requirement. It may also be quite possible that one evaluation case tests several requirements at the same time.

6.3.5.7.4 Identifying and Determining Protective Measures

See Identifying and Determining Protective Measures in activity Preparing Evaluation Specification Delivery.

6.3.5.7.5 Determining Evaluation Environment

All requirements concerning the evaluation environment shall be defined. This will include both functional and non-functional requirements, depending on the test requirements, methods and criteria. On top of that all additional required resources shall be determined, for example integration environment, line issues or special operating personnel, such as crane operators, or personnel with a safety qualification.

If there are several evaluation environments, the individual evaluation cases shall be allocated to them.

It must be possible to unambiguously identify each evaluation environment by its configuration identifier so that an allocation to the system elements to be tested for reasons of replicability will become possible.

In case the complexity of the evaluation environment is high, the latter should be generated in a special subproject.