Fundamentally, costing is an estimation process. Any Reference Case therefore needs to be rooted in the scientific principle of what defines ‘good estimates’. However, as with any estimate, the extent to which cost estimates need to meet any quality criteria depends on the intended purpose for the estimates, which can be complex to define.

So, what is meant by a ‘good’ cost estimate? In statistical terms, the quality of an estimate can be defined along two dimensions:

Accuracy – This reflects the extent of systematic bias in an estimate (how far the estimate is from the true value of the population average cost), often referred to as internal validity. For example, an average cost estimate that systematically excludes overhead cost is biased downwards.

Precision – This reflects the narrowness of clustering of the measurement around the central estimate, such as the mean. For example, if a small sample is used to measure unit costs, then it may have a high margin of error.

These concepts are illustrated in the figure below²⁰.

Clearly, greater precision and accuracy are both desirable. However, defining an agreed minimum level of precision and accuracy is problematic and relies on the purpose behind making the estimate. Unit costs may be used for a range of purposes from routine financial management to estimates of a ‘global price tag’ to scale up global health interventions (see the first section in the Reference Case below). These diverse purposes may require different degrees of precision and accuracy for different levels of aggregation (e.g., total vs. component costs). Moreover, costing itself can be expensive. The ‘cost of getting it wrong’ therefore needs to be weighed against the ‘cost of getting it right’.

There are some formal analytical techniques, such as the ‘expected value of perfect information’, that can help those considering how much to invest in improved cost estimates for specific purposes such as economic evaluation²¹. However, these analyses are expensive and time consuming in themselves, and are thus not widely applied. Given the wide range of purposes and the lack of formal approaches available, there remains no simple way to define ‘a universal minimum standard of precision and accuracy’ for cost estimation in global health.

There are two other important characteristics of cost estimates that may be relevant for specific uses. The first US Panel stated that a core rationale for Reference Cases is to facilitate the comparison of the results of different studies, so that “each study contributes to a pool of information about the broad allocation of resources as well as to the specific questions it was designed to address”¹⁴. This aim is particularly important in global health costing, where resources to collect data are scarce. Thus, three other properties of cost estimation that the Reference Case aims to facilitate are:

Generalizability – the extent to which the cost estimate can be directly applied to other programmatic settings (often referred to as external validity)

Transferability – the extent to which the cost estimate (with adjustments) may be transferred to other programmatic settings

Comparability – the extent to which the features (for example the perspective and the resources included) of one cost estimate are similar to one another

Unlike precision and accuracy, achieving generalizability may not be universally desirable. In some situations, the benefits of arriving at an accurate and precise context-specific estimate (internal validity) may override the benefits of a less precise but more generalizable estimate (external validity). Comparability, narrowly defined as an identical estimation process, may also not always be desirable for the same reasons as generalizability. However, comparability is less problematic, if it reflects improved transparency that permits analysts to adjust estimates in order to compare. So in summary, while generalizability is desirable in most circumstances, as with precision and accuracy, it is hard to set minimum standards in this respect without examining the intended use of the cost estimate. Comparability may, however, be improved simply through improved reporting, without adverse consequences.

In addition, any estimation method should be reliable. Ideally, if carried out by different people it should give consistent results. Likewise, if the estimate is carried out over time results should be consistent. Finally, there should be consistency if carried out across different health interventions, services or outputs.

The focus of a Reference Case therefore is not to set specific minimum standards for each of these characteristics of a ‘good’ cost estimate, but instead to define the ‘best methodological practice’ to support a cost estimation process that is fit for purpose and efficient given the funding and data available. It concentrates on providing a framework for analysts to structure their choices around study design and methods, and to consider how their methods influence the quality of their estimates so they can make efficient choices given their resources available to conduct the cost estimation. In doing so, it aims to improve both the precision and the accuracy of estimate for the funding available.

The Reference Case is more prescriptive, however, in terms of setting minimum reporting standards to improve the transparency of cost estimation. While it does not recommend specific methods to be used, it provides standardized ‘units’ and ‘results tables’, in addition to a methods checklist, to improve the comparability, transferability and generalizability of cost estimates going forward.