As an engineering director supporting multiple teams, it can be difficult to keep track of what’s really going on. By using metrics, you can take a more structured approach to finding the gaps.
But getting started can be easier said than done. What are the right metrics to look for? How will they impact your teams? What metrics are other directors using that they find helpful – and harmful?
As part of our series, ‘The engineering leader’s guide to data-driven leadership’, LeadDev brought together a group of engineering directors to discuss how they are currently using metrics, and where they have found success and failure with metrics in their teams.
The session kicked off with a group discussion about which metrics attendees are currently using across all their teams. Most agreed that they use metrics to measure their technical processes, including lead time, deployment frequency, mean time to resolve, change fail percentage, and wait time at different stages. One attendee said they use metrics to measure team health, while another said they are tracking running costs.
Attendees also discussed what metrics they think they should be using. A few were interested in using SPACE metrics alongside DORA metrics to measure data holistically across three groups: Individual, team, and organizational. Team health and time spent in meetings were also mentioned as potentially valuable things to measure.
How Code Climate use metrics
Next, Jimmy McGill, VP of Engineering at Code Climate, gave a presentation on how directors can use metrics to improve team performance and engineer satisfaction. He shared why metrics are important at the director level and outlined some best practices for getting started. Here are his key takeaways.
1. Why metrics matter
You should be aware of what’s going on in your teams already, but metrics can back up or disprove your hunches. Using metrics also means you’re less likely to overestimate the ability of your org (it’s easy to think you’re doing a better job than you are, especially if you’re only attending stand-ups with senior folks). And metrics help to benchmark industry standards, allowing you to measure yourself against other companies.
2. Introducing new metrics
When thinking about new ways to use metrics, address your engineers’ problems first; the things that improve delivery across the organization will often also remove day-to-day frustrations for your team. When you’re introducing a new type of metric, start with setting your intentions to the team, explaining why it’s important, and how you plan to use it.
3. Best practices for using metrics
Look holistically at a number of metrics across your org and identify those that are known to balance each other out so that you can get a more accurate picture. And always interpret metrics using your human knowledge of the team. For example, if a team’s cycle team increases dramatically, and you know they had problems on-call that week, you’ll know to overlook the metric.
4. Cycle time
If you have to focus on one metric, look at cycle time, the beating heart of the engineering org. It benefits the business (innovating products quickly wins customers). There’s a positive correlation between delivering quickly, and producing stable code. And developers with quicker cycle time experience more satisfaction.
Group discussion
After Jimmy’s presentation, the group came together again to discuss their own experiences of using metrics. First, they were asked to share examples of metrics that have triggered improvements in their teams. A few agreed that looking at cycle time can help to identify bottlenecks and limit WIP time. Folks also shared that measuring the number of commits with tests can lead to more automated testing; tracking the amount of ticket reopens can lead to better groomings; and monitoring the number of P1 alerts can lead to a reduction in alert fatigue.
Then folks shared examples of metrics that have triggered unwanted behaviors. One attendee said that using metrics for things that don’t have a lot of precision can negatively impact happiness and culture. For example, tracking lead times can indicate that completion dates are more concrete and predictable than they really are, which can lead to tense (and unhelpful) conversations when deadlines aren’t met. Folks also agreed that measuring code coverage and velocity can increase perfectionism and stress in engineers. And using metrics can mean teams are too focused on beating the numbers, sometimes at the expense of quality work.
Finally, attendees were asked what they would like to know from other engineering leaders related to metrics. There were a couple of questions around the technical process: How do you choose to measure code quality for your teams? How do you measure direct impact? There were questions related to the value of metrics: Do you believe there is a strong correlation between metrics and success? Can metrics kill innovation? And there was a question around reporting up: What do your non-technical business leaders expect to hear about engineering metrics?
Reflections
As the conversation came to a close, the group agreed that metrics are a challenge: Measure the wrong thing, and you can trigger some unwanted behaviors. But when used correctly, they can have a significant impact on your teams’ performance. It’s important to look at metrics comprehensively, not only those that measure your engineering org, but also those related to other departments, from product to customer service, as well as the business bottom line. As directors, you’re aiming to deliver improvements holistically across all of these areas – and that can start with measuring them.
