Many developers are skeptical of metrics. There’s a common misconception that they’re a top-down management tool to be used (and potentially misused) by managers.
The reality is, engineering metrics are most valuable when they’re used by the entire team. They’re a powerful tool that can help developers at all levels work together to understand and eliminate the team’s points of friction.
Teams that use metrics to pinpoint and remove bottlenecks and slowdowns are ultimately more efficient, more productive, and happier.
As developers reap the benefits of increased efficiency – fewer frustrating roadblocks and more deployments – they’ll be motivated to find more opportunities to optimize processes. This is called the Virtuous Circle of Software Delivery, and teams that can set this cycle in motion generally enjoy higher morale and less developer turnover.
Choosing your metrics
There are 50+ metrics you can track, and while an engineering analytics platform can help you make sense of them all, you’ll benefit from focusing on a few key metrics.
The following framework can help you determine which part of your process is most in need of your attention, so you can focus in on the corresponding metrics.
The development process can be broken into four main phases:
- Time to open. The time between a developer’s first commit and the opening of an associated Pull Request (PR). If a team isn’t pushing commits in the first place, it could be a sign that developers aren’t getting enough time to work. If developers are coding but not opening PRs, they could be confused about project specifications or might need a refresher on good coding hygiene.
- Time to first review. The period between when a PR is opened, and when it’s picked up for review. If things are lagging at this part of the process, it could mean that engineers are too focused on their own work and don’t have the time or the inclination to review their teammates’ code.
- If this is the source of your slowdown, you may want to look at your team’s review speed.[3]
- Time to approve. The period of time a PR spends in the code review process. If PRs get stuck here, it could signal a knowledge silo or a lack of alignment on the team.
- To dig deeper into roadblocks at this stage, it might be helpful to look at review cycles[4] or review distribution.[5]
- Time to deploy. The actual deployment process. Delays at this stage might arise from technical issues or a lack of confidence in new code.
- For issues in this part of the process, check performance metrics from your CI tool. Look at error rates, how long builds are taking, and how frequently builds are failing.
Different organizations will benefit from tracking different sets of metrics – it all depends on your team’s current state, and its goals.
For example, if you’re looking to increase collaboration and eliminate knowledge silos, you may want to start tracking review metrics. This will give you greater insight into how quickly developers are picking up PRs for review, and whether review responsibilities are well-distributed through your team.
If you’re currently scaling, it might be helpful to track your team’s rework levels. High levels of rework could indicate a need to improve onboarding processes or documentation in order to make it easier for new hires to get oriented in the codebase.
Making sense of trends in data
While data can point you towards a potential issue, it can’t tell you why something is falling short of expectations. Data isn’t diagnostic. When a particular metric is trending in a concerning direction, treat it as a starting point for investigation. Through stand-ups, retrospectives, and 1:1s, you can gather information and identify the source of the slowdown.
When our team went fully remote back in March, we noticed that rework was trending upward. Our VP of Engineering used this data as a starting point for conversation, and approached her team leads with her observations. Through their discussions, it became clear that developers were struggling due to frequent miscommunications and unclear specs.
The team was missing out on the context provided by being in an office together and overhearing conversations. When developers had questions, they would ask them in direct messages, rather than public channels of communication, and their teammates would miss out on the answers. With less context, developers were making assumptions about how their work would fit into the big picture, and those assumptions weren’t always correct. Code written based on incorrect assumptions was ultimately code that needed to be reworked.
Using data to empower your team
Once your team understands what’s slowing them down, they can work together to remove their blockers. In our case, we created new processes for communication and documentation that ensured everyone had the context they needed to do their job. Your solution may differ, but your process should be similar; involve your team members in designing solutions, test their impact, and revisit them if necessary. As your team works together to improve your processes, they’ll reap the benefits of those improvements, and be encouraged to keep finding new ways to improve.
[1]The number of times developers commit code in a given period.
[2] The number of lines of code that were added, changed, or removed in a given pull request.
[3] The time between when a Pull Request is opened and when a reviewer first submits feedback.
[4] The number of times a Pull Request goes back and forth between author and reviewer.
[5] The balance of reviews across all the reviewers on your team.
