Ergonometrics®

Ergonometrics ® (from the greek ergon: 'work' ) is a framework for measuring and analyzing the work of making software: the process by which a team collaboratively makes changes to a shared code base to implement a set of behavior changes requested by users.

The underlying measurement models build on a data structure called the Ergonometric Graph™.  It maps representations of cards and their meta data in a Work Tracking system, to representations of commits and their meta data, from code repositories in a DevOps platform.

The cards and their metadata represent the "why": the business context for why a set of code changes was made. The code commits, and their metadata represent the "how": what changes were made, who made the changes and when they were made.

Polaris maintains this graph in granular detail in real time, and opens up the engineering black box for Continuous Measurement and analysis.
The Framework

Ergonometrics® takes an opinionated stance on how the work of making software should be measured. We focus only on team level KPIs, and only on process-agnostic, customer facing metrics instead of measuring internal activity or the utilization based metrics which are commonly used today.

The customer facing metrics are decomposed into a small set of process-agnostic internal drivers, which can be used both for forensic analysis of root causes and as control measurements during product execution to track to continuous improvement objectives.

The Ergonometrics® Models

The Ergonometrics® framework builds on four process models, each analyzing a certain aspect of a software development process. Polaris updates all models in real-time, by analyzing the Ergonometric Graph® whenever code is pushed, and cards or pull requests are updated.

The Model for Work

Ergonometrics®  focuses on cards in the work tracking system that require code changes, i.e the work that actually passes through engineering. We call these specs, and focusing on these eliminates noise in the work tracking system related to non-development tasks. We call this the visible work in the system.

Conversely, we also measure invisible work, code changes that are not tracked by cards in the work tracking system.

Together they give a complete picture of where engineering capacity is being consumed at all times, and constitute what we call work in the model.

A key driver measurement here is traceability, the fraction of visible work in the system, and maximizing this is one of the key steps to improving efficacy of capacity allocation decisions.

The Model for Delays

A large part of the work of making software is thinking and collaboration between engineers. A commit represents an increment of work output - of anywhere from a few minutes to several hours or even days of actual work - by an engineer.

Given this variability,  we view commits primarily as evidence of progress towards implementation of cards.  The more interesting signal is the lack  of progress toward a card.

Ergonometrics®  therefore focuses on delays, and the most important driver measurement here is internal latency, the time between progress events, including commits and state change events in the card lifecycle.

Measuring internal latency in real time identifies delays due context switches and handoffs on the  team and makes them visible while work is in progress.

Almost all additional measurements such as lead time, cycle time, implementation time etc. can be viewed as composites of latency measurements, and we look at monitoring and reducing latency as the key mechanism to drive down non-essential delays in a software implementation process.

The Model for Cost

The fundamental driver measurement for cost in Ergonometrics® is called effort,   and it is defined as the  number of developer days needed to implement a card.  Like other measurements, this can be directly read off of the Ergonometric Graph, and Polaris can compute this in real time for all cards in the system at the commit granularity.

This is much more efficient than trying to measure effort manually using time tracking and other tools that are in use today. Polaris computes this directly by analyzing developer activity without any additional interventions.

Analyzing costs of development using effort measurements is a key forensic tool to improve alignment between business and engineering in allocating engineering capacity to maximize customer facing value.

It is one of the most powerful tools in the analytical toolbox that Ergonometrics® brings to the table.

There are three key metrics we derive from this driver measurement.

Cost of Wip

For work in progress, the key metric we track is the Cost of Wip:  the total effort of all cards that have entered implementation but have not yet been released to production.

The Cost of Wip metric shows the economic cost of carrying too much work in progress. It is well known that high Wip levels lead to high cycle times and overall reduced efficiency in software development, but it has traditionally been hard to quantify this cost directly. Polaris makes this easy.

In our advisory practice, we use this metric to guide teams to move from relying on utilization based allocation strategies to ones that optimize flow of work through the delivery pipeline.

Effort Throughput

This is defined as the fraction of available engineering capacity in a given period, that manifested as work that was actually released to production in that same period. It is used to shed light on process efficiency and analyze overall capacity allocation decisions. It is a very powerful counterpart to volume based throughput measurements in use today.

Feature Costs

Card level effort can be aggregated and analyzed along multiple dimensions such as features, card types, releases, epics etc. to understand development costs along business facing dimensions. This is useful for planning effort budgets for future work and generally understanding what development costs are in more granular detail than previous models allow for.

The Model for Quality

The quality model in Ergonometrics®  starts with conventional measurements of external quality which focuses on customer facing defects, but then extends it include measurements of internal quality, focusing on engineering practices that impact the quality of the code base as work is delivered.

Since the Ergonometric Graph has a detailed model of file and line level changes that go into implementing each card, it allows us to analyze how the code base evolves as cards are implemented.

This is a much richer source of insight than analyzing commit history at the commit granularity, as the business context represented by a card allows us to analyze  collections of logically related changes needed to implement cards that would not normally be revealed by static analysis of a code base, or even analyzing single commits in isolation.

Initial Quality

The first application are we are currently bringing out of the R&D phase is initial quality. We analyze the code changes made for a card and measure whether tests were written in engineering for a given card.

Code coverage, which measures what fraction of a code base is exercised by a test suite, and is the most prevalent metric for test quality. However it is backward looking, and typically not easily actionable since it is a property of the code base as a whole.

Ergonometrics®, together with Continuous Measurement,  enables test quality inspections at the card level while work is in progress, allowing teams to know whether tests are being written for specific cards, and whether untested code is being promoted to production for specific cards and pull requests. It is a powerful tool to bring testing discipline to a team and ensure that the policies around this are followed.

This is still an emerging area of applications for Ergonometrics®, and we will be announcing a number of novel features in Polaris  based on the quality model in the near future.

The Ergonometrics™ Models

The Ergonometrics™ framework build on four process models, each analyzing a certain aspect of a software development process. Polaris updates all models in real-time, by analyzing the Ergonometric Graph™ whenever code is pushed, and cards or pull requests are updated.

The Model for Work

Ergonometrics™  focuses on cards in the work tracking system that require code changes, i.e the work that actually passes through engineering. We call these specs, and focusing on these eliminates noise in the work tracking system related to non-development tasks. We call this the visible work in the system.

Conversely, we also measure invisible work, code changes that are not tracked by cards in the work tracking system.

Together they give a complete picture of where engineering capacity is being consumed at all times, and constitute what we call work in the model.

A key driver measurement here is traceability, the fraction of visible work in the system, and maximizing this is one of the key steps to improving efficacy of capacity allocation decisions.

The Model for Delays

A large part of the work of making software is thinking and collaboration between engineers. A commit represents an increment of work output - of anywhere from a few minutes to several hours or even days of actual work - by an engineer.

Given this variability,  we view commits primarily as evidence of progress towards implementation of cards.  The more interesting signal is the lack  of progress toward a card.

Ergonometrics™  therefore focuses on delays, and the most important driver measurement here is internal latency, the time between progress events, including commits and state change events in the card lifecycle.

Measuring internal latency in real time identifies delays due context switches and handoffs on the  team and makes them visible while work is in progress.

Almost all additional measurements such as lead time, cycle time, implementation time etc. can be viewed as composites of latency measurements, and we look at monitoring and reducing latency as the key mechanism to drive down non-essential delays in a software implementation process.

The Model for Cost

The fundamental driver measurement for cost in Ergonometrics is called effort,   and it is defined as the  number of developer days needed to implement a card.  Like other measurements, this can be directly read off of the Ergonometric Graph, and Polaris can compute this in real time for all cards in the system at the commit granularity.

This is much more efficient than trying to measure effort manually using time tracking and other tools that are in use today. Polaris computes this directly by analyzing developer activity without any additional interventions.

Analyzing costs of development using effort measurements is a key forensic tool to improve alignment between business and engineering in allocating engineering capacity to maximize customer facing value.

It is one of the most powerful tools in the analytical toolbox that Ergonometrics™ brings to the table.

There are three key metrics we derive from this driver measurement.

Cost of Wip

For work in progress, the key metric we track is the Cost of Wip:  the total effort of all cards that have entered implementation but have not yet been released to production.

The Cost of Wip metric shows the economic cost of carrying too much work in progress. It is well known that high Wip levels lead to high cycle times and overall reduced efficiency in software development, but it has traditionally been hard to quantify this cost directly. Polaris makes this easy.

In our advisory practice, we use this metric to guide teams to move from relying on utilization based allocation strategies to ones that optimize flow of work through the delivery pipeline.

Effort Throughput

This is defined as the fraction of available engineering capacity in a given period, that manifested as work that was actually released to production in that same period. It is used to shed light on process efficiency and analyze overall capacity allocation decisions. It is a very powerful counterpart to volume based throughput measurements in use today.

Feature Costs

Card level effort can be aggregated and analyzed along multiple dimensions such as features, card types, releases, epics etc. to understand development costs along business facing dimensions. This is useful for planning effort budgets for future work and generally understanding what development costs are in more granular detail than previous models allow for.

The Model for Quality

The quality model in Ergonometrics™  starts with conventional measurements of external quality which focuses on customer facing defects, but then extends it include measurements of internal quality, focusing on engineering practices that impact the quality of the code base as work is delivered.

Since the Ergonometric™ Graph has a detailed model of file and line level changes that go into implementing each card, it allows us to analyze how the code base evolves as cards are implemented.

This is a much richer source of insight than analyzing commit history at the commit granularity, as the business context represented by a card allows us to analyze  collections of logically related changes needed to implement cards that would not normally be revealed by static analysis of a code base, or even analyzing single commits in isolation.

Initial Quality

The first application are we are currently bringing out of the R&D phase is initial quality. We analyze the code changes made for a card and measure whether tests were written in engineering for a given card.

Code coverage, which measures what fraction of a code base is exercised by a test suite, and is the most prevalent metric for test quality. However it is backward looking, and typically not easily actionable since it is a property of the code base as a whole.

Ergonometrics™, together with Continuous Measurement,  enables test quality inspections at the card level while work is in progress, allowing teams to know whether tests are being written for specific cards, and whether untested code is being promoted to production for specific cards and pull requests. It is a powerful tool to bring testing discipline to a team and ensure that the policies around this are followed.

This is still an emerging area of applications for Ergonometrics™, and we will be announcing a number of novel features in Polaris  based on the quality model in the near future.