Earlier this week we hosted a phenomenal webinar on measuring engineering organizations and the good and bad about using DORA.

‍

Here are the key highlights from our panelists:

Don Smith - Engineering leader at LendingTree, Managing 30-40 engineers, spread out across 8 dev teams, both onsite and remote. Don recently restructured the team which was previously structured by platform to a more functional/cross-platform structure. The change was needed due to the complexity of projects and the across platforms functionality that required a lot of coordination.

Jonathan Wanagel - Leading the 45 people engineering team at Axon. The team is split. Half in the US and half are in Vietnam. The team is structured by product so they are relatively autonomous. Jonathan’s prior experience was at Meta and Microsoft

Rob Fletcher - Currently COO and engineering leader at DevZero, where he manages a team of 20 engineers across the US, Canada, Europe and Latin America. Previously Rob was managing engineering teams at Uber and Meta. 

‍

Do you measure DORA? How frequently?

‍

Don was kind enough to share some of the DORA dashboards he is looking at. The dashboard is generated using Power BI (here’s a detailed explanation on how to create your DORA dashboard). He mainly pays attention to stability, mean time to recovery, deployment frequency. He reviews that with the team on a monthly basis and mostly looks for trends rather than hard numbers. “It is key to look not just at productivity which is how quickly you produce stuff, but rather efficiency, which takes quality into account”, Don commented.

Jonathan is more focused on operational efficiency at Axon. “We are paying attention to delivery metrics. How are we tracking against our roadmap? Hitting all of our roadmap goals is key. We are looking to improve our operational metrics: incidents and regressions. “. Jonathan is also focused on pre-morton vs. just postmortem. He calls them “insurance packages”. What is the risk of something happening, how impactful is it on the business and how do we prevent it. 

‍

What’s missing in DORA?

Everyone agreed that while metrics are great, developers are knowledge workers. Seniority is another factor to take into account. The work that senior developers do varies from junior developers, hence metrics should be used with context. Don shared his experience as a Staff Engineer, where he mostly did reviews and didn’t write much new code hence context is key. 

As Don said, “DORA is not enough but it’s enough for what they are designed for. DORA doesn’t really reveal how your team is doing. It’s easy to be over focused on one vs. another. I prefer to think about effectiveness vs. productivity. We try to balance efficiency and quality, cross training and risk mitigation.”

‍

How to measure impact?

Impact is business impact. Engineers have a lot of control over delivery productivity but less over impact productivity. As you’re more senior you make more impact productivity . Impact varies a lot depending on the business: more signups, more engagement. 

Jonathan said, “I encourage engineering managers to define the business model for your organization. Leaders should know how to balance delivery productivity and impact productivity.”

‍

What’s the role of tools in increasing productivity? How do you decide whether to buy or not? 

Don, Jonathan and Rob, take inputs from their Staff Engineers on tools they recommend. They experiment and use mostly judgment as opposed to trying to see how these tools impact metrics such as DORA. 

“We recently adopted Github Copilot. While we saw no noticeable impact on metrics, my developers liked it, our acceptance rate was around 80% and the general impression was that it is valuable”, said Don. 

How did Uber improve Productivity?Rob shared his experience at Uber, which during 2016-2021 went through a transformation in developer productivity and experience, breaking nomorepos, optimizing build times and introducing Devpod. Debo covered this more in depth here.

‍

Building and Scaling High Performing Technology Organizations

Don recommended "Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations" by Nicole Forsgren PhD, Jez Humble and Gene Kim.

It was his way of educating himself about DORA and it inspired him to introduce it at LendingTree.

‍