Test and Evaluation (T&E)

Test and evaluation (T&E) is a crucial in providing assurance during product acquisition, development and sustainment. In many cases it is easy to test and evaluate performance: one can easily measure the top speed of a car for example. Less easy to T&E reliability, availability and maintainability (RAM) metrics.

The challenges which need to be answered when conducting RAM T&E are:

  • What tyre of test will provide me with the most accurate results?
  • What is the definition of failure? (which of the failures observed during the test are counted as being ‘real failures’).
  • What is the mission profiles to be used during the test?
  • How many products do I need to test?
  • What is the length of testing?
  • From my test plan, what is the risk that the product passes the test, but is still a bad product (Consumer Risk).
  • From my test plan, what is the risk that the product fails the test, but is actually an acceptable product (Producer Risk).

From a data analysis view, the two primary activities in T&E is establishing the test parameters. RAM test are expensive and often have their size and length limited by resources. A limited RAM test however may expose the participants to huge risk of accepting a bad product or failing a good product. This must be quantified in order to bid for more resources, or look at other ways to provide reliability assurance.

The second primary data analysis activity in T&E is the evaluation of the test results. Once the failures have been identified, the data can be used to determine if the product passes the specification. This should be done as routine task during the test. If a product performs exceedingly well early in the test, then sufficient data may be available to prematurely stop the test whilst still providing the desired level of assurance. If the product performs exceedingly bad, again there may be sufficient data to stop the test early, therefore saving valuable dollars and effort in testing.

Acuitas can help you design RAM tests and evaluate the data they generate. Regardless of what characteristic you value the most; Acuitas can assist you in balancing your costs and need for accurate information. After a test, Acuitas can determine if a product passed or failed a specification and quantify the uncertainty in the results: uncertainty that you may wish to dedicate more resources to refining