8 Part 8: Annex

8.1 Method References

8.1.6 Evaluation Process

Usage

Preparing Requirements Evaluation, Performing Market Survey for Off-the-Shelf Products, Performing Make-or-Buy Decision, Peparing Criteria Catalog for Assessment of Offers

Reference

Kon96, Schw04, Wil75, Wan02, PD99, LMTC01, AF02

Purpose

Within the framework of IT projects, there is an increasing demand for procedures that permit a qualitative and quantitative evaluation of specifications – like the »Requirements Specification, the »Evaluation of Off-the-Shelf Products, or the »Overall System Specification – based on transparent and repeatable criteria. In the course of the past 10 years, some standard modules have been developed for this purpose.

Weighted Scoring Model (WSM)

One of these standard modules is the Weighted Scoring Model (WSM) [»Schw04]. In a first step, this model defines assessment criteria, which are then weighted in accordance with their significance for the overall system (e.g., essential, very important, important, nice-to-have, or 10, 7, 5, or 3 points). In the evaluation, the model will assign scores to the individual criteria, e.g., 70 % degree of fulfilment. The total weighted scores are obtained by multiplying the score with the weights of the individual criteria, e.g., 70 % * 7 points = 4.9 points. The total of all evaluated criteria indicates the weighted score of the subject to be evaluated. The result can than be compared with the results of the other points. In addition, minimum scores may be defined, which lead to appropriate consequences for the overall project (e.g. if the weighted scoring for off-the-shelf products shows that the acquisition of these products is no realistic possibility, the development of individual products is the only way).

Analytic Hierarchy Process (AHP)

A similar procedure is the AHP procedure, which is also based on a decision matrix. The criteria are arranged in hierarchy levels in accordance with their relevance, and the scores can be calculated from pair-wise comparisons (cf. »Kon96 et al.).

Both methods, but particularly the AHP, pose the risk that the overall model becomes inconsistent due to wrong weightings, thus loosing its informative value. The complexity of the model should be limited - also considering the effort connected with the evaluation.

Special Case: COTS Software

The evaluation of standard software and standard software components is intended to develop and apply comparison methods and criteria, which permit the evaluation and selection of off-the-shelf products. The subject has been discussed at international level since approx. 1990. Since that time, the commercial use of IT has no longer aimed primarily at individual system developments, but at the use and integration of standard applications.

Transaction Cost Analysis

The subject was first developed for industrial production, but was soon also transferred to the IT sector: Is it more economic and effective, to produce a sub-product or end product within the enterprise or to purchase it from a third party? For this purpose, the transaction cost theory (TCT) [»Wil75, »Wan02] was developed, which at first evaluates the assets based on their specificity for the respective process: the more specific an asset is, the more recommendable is the production within the enterprise, and the less specific an asset is, the more sensible is the purchase from third parties. Second, the approach evaluates the uncertainties, the risks, followed by the frequency of use and the reputation of the supplier. These are the criteria for the decision as to whether the asset should be produced within the enterprise or purchased from a third party.

Meanwhile, numerous models have been developed, which propagate a combination of different evaluation processes [for a small selection, refer to »Kon96, »PD99, »LMTC01, »AF02].