Traceability

Trace Score™ – An Empirical Way to Reduce the Risk of Late Requirements

By |

trace score

Trace Score™ – An Empirical Way to Reduce the Risk of Late Requirements

Executive Summary

One of the main causes of rework, delays, and cost overruns in product development is the creation of new requirements late in the process. This is a well-known risk in product development, but what management practices can empirically be shown to reduce this known risk?

Using our proprietary database of metadata from over 50,000 complex product development projects, we were able to determine that the Trace Score is an empirical method to reduce late requirements. In fact, teams that maintain a high Trace Score reduce the burden late requirements have on their project by 67% compared to teams with low trace scores.

  • With this knowledge, our recommendation is that practitioners measure and monitor the Trace Score of their projects to resolve issues early and ensure that the risk of late requirements is kept to a minimum.

Dataset Background

Jama Software has the world’s largest, live dataset of engineering process performance with over 50,000 engineering projects updated and growing continuously. Leveraging this dataset, it is now possible to determine empirically which management practices improve the performance of the product development process. To learn more about our benchmarking, please review our Traceability Benchmarking Report.

The Empirical Questions

In this analysis we will explore three key questions:

  1. What are late requirements?
  2. How do late requirements negatively impact projects?
  3. Does maintaining a high Trace Score reduce the risk of late requirements?

What are late requirements?

For the purpose of this analysis, we define “late requirements” as those requirements created after the completion of a project’s requirement decomposition phase which we estimate as spanning the middle 50% of all requirement creation activity (creation and refinement). To illustrate what late requirements look like, we show two actual projects below with requirement activity plotted over time.

Requirement Creation Over Time

 

In the Timely Project, requirement creation occurs in a defined requirement decomposition phase to form a necessary and sufficient set of requirements, with very few requirements being added after the fact (e.g. in fig (a), only 1.3% of requirements created late). In the Late Project’, requirement creation bleeds into future phases of the project, leading to a significant amount of late requirements (e.g. in fig (b), 9.2% of requirements are created late).

RELATED: Requirements Traceability Benchmark

How do late requirements negatively impact projects?

We can measure the outsized burden late requirements have on project teams, which we have illustrated for our two projects below. We define late requirement burden as the total number of requirement activities (creation and refinement) attributed to late requirements as a percentage of all requirement activity.

Impact of Late Requirements on Project Team Activity Burden

In the Timely Project, minimal late requirements enable better forecasting of project completion and limit the rework and cost brought on by late requirements (e.g. in fig (c), late requirements only create an additional 8% burden).

In the Late Project, the high volume of late requirements makes it much harder to forecast project completion as the scope of the project is constantly changing, and project teams need to accommodate the late requirements (e.g. in fig (d), late requirements contribute an additional 31% burden).

Unsurprisingly, this additional burden of late requirements has an impact during testing for requirement validation. In our actual project examples, the Late Project has a test failure rate over 3x that of the Timely Project.

percentage chart

RELATED: Unlocking The Power of Live Traceability with Jama Connect

Does maintaining a high Trace Score reduce the risk of late requirements?

A core theorem of Systems Engineering is that maintaining high requirement traceability from the start of a project reduces the risk of late requirements and negative product outcomes. With our project dataset we can now test this theorem empirically. We define traceability as a measure of a project’s ‘expected’ traceability that has actually been established and calculate the Trace Score as follows:(1)

established over expected

For our example projects, the Timely Project achieved a Trace Score over 6X that of the Late Project; suggesting that maintaining a high Trace Score throughout the project reduces the risk of late requirements.

traceability chart

To further determine if Trace Score correlates to late requirements, we divided our dataset of projects into quartiles based on their Trace Scores (Quartile 1 = bottom 25% trace score, Quartile 4 = top 25% trace score) and then compared the distribution of ‘Late Requirements Burden’ across these quartile groups. What we found is that projects within the bottom traceability quartile had a median Late Requirements Burden 3x greater than those in the top traceability quartile. In other words, the evidence supports that projects managed with higher traceability generally experience less risk from late requirements.

Recommendation

Our analysis has shown that late requirements negatively impact projects and that managing projects through a Trace Score is the only empirical way to reduce the risk of late requirements. Below you can see how one can measure the Trace Score over time as a project progresses to ensure system engineering best practices are being followed. A low or falling Trace Score can quickly identify areas to address to reduce the risk of late requirements.

Here you can see how managing the Trace Score directly as the project is underway would have identified the risk early in the Late Project.

Benchmark Chart

To learn more about achieving Live Traceability™ on your projects, please reach out for a consultation.

Interested in learning more? Download the entire Research Notes: Trace Score datasheet HERE.

 


Marc Osofsky
Latest posts by Marc Osofsky (see all)