Measuring the unmeasurable - How AWS, Alexa, Tableau Software and Raygun monitor what matters

By Freyja Spaven | Posted Dec 19, 2019 | 6 min. (1086 words)

The Tech Leaders’ Tour is a series of events bringing tech leaders together to learn from each other about improving software quality and customer experience.

“3.7%. That’s how penetrated the cloud market is today. We’re talking a trillion-dollar market only penetrated by 3.7%. The world is up for grabs.”

Kicking off the Tech Leaders’ Tour in Seattle, Peter from AWS introduces our central theme direct from the Oscar Building at Amazon HQ.

“Customers are not going to want technology to be slower. They’re not going to want it less performant or less secure. That is the topic for today - to talk about how we can build great software and then make it run to customers’ expectations.”

Watch the full panel below:

The wide-ranging conversation drew on experience from some of the world’s highest-performing teams. From left to right, we had:

Moderating the panel was Cassidy Williams. Previously named as one of the 35 Women Under 35 Who Are Changing the Tech Industry, Cassidy offers her unique skills as a teacher to dig into how to open meaningful communication lines to customers. In the full panel video, we discover how our leaders balance culture, metrics, and customer feedback to drive innovation and to support millions of code releases every year.

Here are some of our favorite insights from the day.

1. Help your developers get as close as possible to customers

Similarly echoed in previous tech leaders discussions, first hand and real-world customer experience for development teams is a focus for all the panelists. The teams all receive these signals in different ways.

Doug of Alexa places value in receiving feedback from many different sources and actioning that feedback quickly. He says that the feedback form inside the Amazon app is invaluable, but he also leans on his team to be communicating in a way that helps everyone to be responsible for raising operational issues.

Rory from AWS likes to ask her customers the big questions to push developers out of their comfort zone and into innovation. Diana emphasized that dogfooding is a great way to connect the team with the product, while JD offered that having the right feedback mechanisms for staff and customers improves the quality of feedback we use to make product decisions.

The important sentiment was that although each team received feedback in different ways, all used the customer insights to fuel better engineering outputs.

2. When it comes to quality, measure the right things - ignore the rest

In terms of measuring the quality of engineering outputs, Doug explains that he likes to think of monitoring maturity as a hierarchy of needs; that you need to have the foundations in place in order to unlock the potential. “Tools like AWS Cloud Watch and Raygun help you move up the ladder. What you’re aiming for is self-actualization - where people aren’t getting paged, you’re growing, and you have product-market fit.”

Doug adds that at Alexa, they measure perceived latency to better understand how product updates impact customer experience, rather than latency or load time. He argues this could be a fairer measure of customer experience as “cache first” becomes best practice.

JD Trask adds that his advice for smaller companies is to wait until you have statistical significance in feedback before you change the product road map based on customer feedback. Looking for meaningful feedback should be high on the agenda - one of the reasons that Raygun removed NPS in favor of more intimate feedback methods, like Slack and feature usage tracking.

3. Respond to smoke signals, not fires

Another common theme on the panel was the importance of responding to issues proactively.

Staying in front of issues protects the team from constant fire-fighting while simultaneously protecting customers from bad experiences. “It’s all about the timeline.” Shares Rory. “If you’re always putting out customer fires, then your timeline is also wrong for how you are serving the customers. Most of the time, we don’t get to that point because we are customer-obsessed. But we need to collect that data earlier in the sales cycle.”

The panelists got in front of issues by empowering their teams with the autonomy to reduce the friction. If someone sees a smoke signal, they need to be enabled to either fix it or raise it.

4. Culture - Balancing relentless customer focus with team happiness

“At Tableau, we’re customer-focused, but we also have a value-driven culture.” Says Diane. Doug agrees that “the best teams are able to self-guide to a point where they can self regulate. Make sure people understand what going above and beyond looks like, but allow people to set their own boundaries.”

All panelists take culture very seriously and agree that giving team members autonomy was a key driver in building a better product. Embracing failure is closely linked to the “blameless postmortem” culture - and one that Rory advocates. “Failure at AWS is not a bad thing. People wait too long. I have a list of trigger words that must be escalated to me - that process is what makes people feel safe. People wait too long.”

Teams are also leaning on software for that feedback loop. At Raygun, employees are encouraged to use employee feedback software 15five to raise issues with their managers, while an internal tool called Connections checks on team health at Alexa.

Last thoughts

Seattle’s panel made it clear that the more you talk to customers, the better you can prioritize between feature building, bug fixes, company objectives, and employee happiness. Make sure to watch the full panel, and express your interest in the next event.

Get a rare glimpse at how companies like Alexa, Nike, Microsoft, AWS, Tableau Software, Raygun, The Standard, Xero, Vend, and PushPay monitor software quality with our Tech Leaders’ Tour panel.

Watch the recaps here:

  • Auckland panel - Xero, Vend, and PushPay on prioritizing user experience during the development workflow.
  • Wellington panel - Trade Me, Xero, Raygun, and Sharesies on maintaining an excellent standard of software quality.
  • Portland panel - Microsoft, Chef Software, Nike and the standard share tips on actionable monitoring.
  • Auckland panel 2020 - Xero, Vend, Lancom, and Tend on prioritizing customer experience in today’s digital climate.
  • Read the Tech Leaders’ Report and learn how other tech leaders measure software quality and user experience.