How much could software errors be costing your company?

By Laura Marwick | Posted Jul 9, 2023 | 15 min. (3013 words)

Errors are an inevitable part of building software. However, while you can’t eradicate them, you can definitely mitigate them and reduce their impact. And the reality is, if you don’t measure, track, or resolve errors, you’re ignoring a loss in revenue.

Once you understand the financial impact of software errors on your company, you can start to prioritize your biggest opportunity costs and catch issues early with methods like smart, early testing and proactive error reporting. Using a few industry averages, you can put a number to the unseen cost of software errors for your team, and start to plug hidden cash leaks like wasted developer time and lost customers.

In this post:

Reducing the cost of errors

Software errors and their impact

According to the Consortium for Information and Software Quality, poor software quality costs US companies upwards of $2.08 trillion annually. These losses include costs from operational failures, unsuccessful projects, and software errors in legacy systems.

Operational problems contributed the largest share, with $1.56 trillion in losses. However, CISQ notes that technical debt is taking center stage as one of the most insidious detractors from profitable software systems.

Let’s look at the obvious bills first: the downtime and PR agony of a big, public software failure. Here are a few recent cautionary tales that illustrate the impact of a catastrophic failure or breach.

  • The big Facebook outage in 2021 (remember that?) was reported to cost $65 million in advertising revenue, and (temporarily) tanked Mark Zuckerberg’s personal wealth to the tune of $6 billion.
  • Twitter has famously suffered an ongoing series of outages since Elon Musk’s extensive layoffs, during which period the stock price also plummeted to a 3rd of its former value. While this is no doubt related to other turbulence and negative press, the recurring outages play a major role.
  • The most far-reaching outages occur in the services that form the infrastructure of much of the internet, like Cloudflare and AWS. The economic impact hits not just the providers themselves, but their clients and in turn, their customers at the consumer level.
  • The infamous SolarWinds attack cost 18,000 clients an average of $12 million each. Impacted companies in the U.S. reported an average of a 14% impact on their annual revenue. SolarWinds itself had $40 million in upfront recovery costs, plus their stock plummeted from $25/share in 2020 to about $9.
  • In September 2022, hackers stole $160 million from crypto platform Wintermute. In March 2022, hackers stole $620 million in Ethereum cryptocurrency fromm play-to-earn game Axie Infinity. In June 2023, hackers stole $35 million in crypto from Atomic Wallet.
  • In fact, analysts say that hackers stole a combined total of $4 billion in crypto in 2022.

While crypto is clearly a major target, it’s far from the only one. According to Cybersecurity Ventures, cybercrime costs alone will reach $10.5 trillion annually by 2025, and the US will shoulder at least one-third of that cost.

The cost of software errors to your company

The cost of your ad platform going down, or a vulnerability letting a hacker raid your accounts, is pretty obvious. But it only accounts for a small chunk of the real cost of software errors. So what do these astronomical bills, up in the trillions, actually comprise of? How are software errors so expensive?

Well, there are the loud and public costs of major outages or breaches, and the silent, lingering ones. Software errors can cause a crisis, but equally, they can fly under the radar, eating away at your margins indefinitely, exposing your end-users to slow, buggy software. However, many businesses don’t have visibility into their software errors, so measuring them and their impact can be tricky.

Finding and fixing bugs is the single most significant expense in an application’s lifecycle. If your system completes its expected 25-year lifecycle, you’ll spend nearly 50 cents of every dollar isolating and fixing software errors.

And, of course, that’s only taking into account the time and money you’ll spend fixing defects. Then there are the missed opportunities, unhappy customers, and dissatisfied development teams. Brace yourself; let’s look at how much software errors are really costing you.

Effect of software errors on developer labor costs

The time your developers spend finding and fixing bugs is time that costs the business money, plus the opportunity costs of not building new features. As we just covered above, finding and fixing bugs is a big-ticket item in your development lifecycle. So exactly how much does it cost you?

On top of developer salary expenses, unaddressed software errors affect end users, and when your product doesn’t work as intended for your customers, support costs start to mount.

You should be aiming for this kind of reactive work (finding and fixing errors, support costs) to take no more than 20% of developer hours, allowing 80% on proactive work (building features and improving your products) rather than vice versa. If we assume that your team adheres to this 80/20 standard, based on a 40-hour work week, the average software developer spends 32 hours fixing errors and replicating issues each month. If you have 50 developers, the combined 1,600 hours of reactive work could potentially ramp up the cost of software errors to US$83,000 in lost time per month.

Then there’s the opportunity costs of time spent away from building regular features for customers. Reducing 1,600 hours by 50% to 800 hours would save your company US$41,500 every month in salary expenses, while increasing overall product and software quality.

barriers to bug fixing Image credit Undo

How do you cut back on the labor costs of errors? Unsurprisingly, the answer is automation. A huge component of the bug-fixing process is the time spent triaging and diagnosing issues. Tools like Crash Reporting will automate this process. Data from Raygun customers shows a 50-90% reduction in the time it takes for them to identify and resolve errors, freeing up development time to add value for customers.

Impact of software errors on customer retention and loyalty

There’s a wealth of data on how the speed and function of a website affect user behavior. Here are some of the most revealing numbers:

User behaviour

There’s also data uncovered by McKinsey that tells us that companies with higher developer velocity have 4–5 times faster revenue growth.

How does this play out in real-world impact? Well, let’s imagine a scenario involving a major fast-food delivery company.

Let’s say that their e-commerce site generates five million errors per month (not unusual in a big national company), between multiple engineering teams releasing new features daily and the complexity and scale of all their ordering and tracking systems.

If only one percent of those errors affect the order or checkout process, at $50 per order, the potential impact in lost revenue is $2.5 million per month. That’s without even factoring in all the flow-on effects of poor word-of-mouth and reputational damage. Even seemingly tiny error occurrences add up fast.

In a real-world example, iCook reduced their CLS (a measure of janky, buggy layout shifts while a page is loading) by 15% and saw a 10% increase in ad revenue.

Software errors and loss of investor trust

Boeing’s Starliner has suffered from problems related to software errors since 2018, with two failed attempts to launch. Boeing has reported losses totaling $595 million related to the project, and their stock price has suffered significantly.

Investors made their displeasure known when Slack outages showed up in their quarterly earnings, as they’d failed to meet the standards set out in their SLAs. The market responded with a chilly 14% drop in the stock valuation.

British Airways wiped about $200M off their stock price when they stranded hundreds of passengers in airports during a major systems outage.

And to cite just one of many Tesla reliability headlines, Tesla stock fell 5.7% after announcing a major patch to 362,000 of their self-driving vehicles. The company’s self-driving software has been plagued with widespread issues for years which have regularly chipped away at the stock price.

Opportunity loss and suspended growth

There are many startling statistics about the failure rate of software projects as a whole. Various sources have repeatedly found:

  • In terms of time and budget, only 10%-40% of technology projects are successful.
  • 47% of projects have problems with timescales and budgets, and result in low-quality products.

Poor software quality is a serious and widespread issue, and poor development practices can halt company growth, wasting huge amounts of time and money. While proactive error resolution plays an important role in the success or failure of a project, this also goes beyond just errors and into the design and architecture of code and DevOps concerns about methods of working.

Impact of errors on developer turnover

If you’re asking your development team to dedicate huge portions of their working life to the monotonous task of bug-fixing, you’re not helping their job satisfaction. We’ve established that errors shouldn’t occupy more than 20% of developer hours, but there are some nasty stats that show a different reality:

  • 26% of developers in a global survey said that they spend half their time fixing bugs
  • 10% in the Western U.S. said this occupies up to 75% of their time
  • 44% called bug fixing their biggest pain point
  • 55% said it kept them from building new features and functionality
  • 12% felt “resentful” about manual bug-fixing
  • 7% said it makes them want to quit.

This is especially confronting given that the tech sector has the highest staff attrition rates of any industry, and within this, software engineers have an even higher turnover rate of 21.7% (for context, the average across all industries is 10.9%). Plus, the business cost to hire a developer averages $50,000.

The upshot? It’s a very good investment to give your development team the right tools so that they can spend less time digging through log files and more time doing rewarding work. Plus, this means they’re building new features that strengthen your market offering, so it’s a win-win.

Preventing software errors is better than fixing them

The first thing is to realize that no software team is perfect, and a goal of zero bugs isn’t realistic. Perfect is the enemy of good, and success is just about tipping the balance in your favor.

As errors move through the software development lifecycle, the cost of addressing them increases. It’s always more cost-effective to fix them earlier (this is a big part of the case for shifting left). Undo’s survey respondents say the average amount of time needed to find and fix one failure in their backlog is 13 hours.

When you’re working in a development environment, you have more control and fewer variables. You only use one browser and tend to use the software only as intended. Plus, you probably already have a debugger on your machine. The major problem with bugs in production is the lack of supporting tools. Developers have no visibility and no way of debugging to figure out where a problem is coming from, unless they have a tool to shine a light on issues. Production is a much more hostile environment than development or UAT. Once issues are found in the front-end, they are more complex to diagnose and resolve. So, find and fix errors early and often. Jacek Materna, CTO at Assembla, agrees:

“One of the best ways we deal with bugs is empowering all of our individual developers to avoid bugs in the first place. We continuously deploy and use test-driven development, but one of our greatest assets is one of individuals and interactions: a rigorous code review process.”

Which leads us to better testing.

Better testing

While automated testing is making advances, it’s still underdeveloped, so manual testing is here to stay. Poor testing methodology costs a lot of money in wasted time and can result in a diminishing scale of return.

For example, if your testing process is too thorough and not efficient, your team will produce new features faster than you can test them, and you’ll end up with a backlog of features waiting to be deployed to production.

Many senior engineers write unit tests to check critical parts of the code. If you haven’t written tests previously and are working on a legacy codebase, then add unit tests whenever you find software errors. This approach builds up a unit test set over time and proves that the fixes are being made.

Finding and reproducing software errors is where most of the time is wasted (that reactive work we talked about earlier). A monitoring tool will do the heavy lifting for you, alerting your team to errors in real-time. There’s no need to dedicate so many hours to finding software errors when your team can know instantly what’s wrong and where. If you get the stack trace, you can solve the problem faster.

There’s also compelling data showing that the “highest-performing engineers are 1.5x more likely to have easy-to-use tools”.

It’s also a very good idea to conduct load testing and scale up capacity resources for any critical areas. Reliability success stories from Stripe, Nylas, and Shopify

Automating deployments

Once you have a fix ready, shipping it can be half the battle without automatic deployments. It’s easy for your developers to lose a whole day deploying a fix.

You can also use Deployment Tracking to see which releases and versions are generating errors and monitor whether newly shipped code has introduced any new issues or regressions of existing ones. This allows your developers to discover defects quickly and roll it back, or rapidly patch the error and push out a new build as a solution before end users (and sales) are affected.

This can shave hours off development time. Invest time in building new features and creating excellent user experiences, not fixing bugs.

Prioritize your bug fixes

What’s more important — a bug that occurs 500 times?

img 500 bug

Or 10,000 times?

Image showing 10,000 errors

Well, it depends. Which one is affecting your end-users more?

One of the most vital prioritization criteria is the customer impact of all outages or errors. Prioritizing software errors by the count of customers affected will allow you to allocate resources accordingly, reducing costs throughout the Software Development Lifecycle (SDLC).

Having tools and systems in place to help you intelligently prioritize issues allows for better allocation of resources and protects more end-users from a poor experience. If the goal is to reduce the cost of software errors, prioritize your bug fixes with the end-user in mind first.

This allows for better allocation of resources and protects end-users from having a poor experience. If the goal is to reduce the cost of software errors, prioritize your bug fixes with the end-user in mind first.

Encouraging a security-first approach

Security breaches are another huge contributor to lost revenue, often measured in billions of dollars. CISQ cites security as the biggest threat to the economic welfare of the software industry.

Are standards slipping, or are bad actors getting cleverer? Could breaches like SolarWinds’ 18,000 companies or Yahoo’s 3 billion account holders have been prevented?

While malicious attempts to intercept data are on the rise, there are many, many precautions that businesses can take to defend their cybersecurity. It’s crucial to think about security early on, while the project is still in the ideation phase. Security is not an afterthought in your development process. By prioritizing security as a culture, even if it slows you down, you’ll help save the costs of a breach and all the associated damage control. When your developers understand the patterns and nature of your product’s errors, it’s easier for them to spot discrepancies and keep your code safe.

Providing a healthy work environment where employees can speak up does wonders for preventing security breaches, especially for developers who are in the code base all day long. Unfortunately, software security is not on the curriculum in most universities. Therefore, it falls to the business to provide best practices around security. The more your development team knows about the health of your software, the more likely someone will notice when something is ‘off.’ Encourage and cultivate that in your company culture - it might just save you serious money.

Better workflows and tools can pay huge dividends

The full cost of software errors can be hard to pinpoint - often, it may be as much about avoiding catastrophic losses than producing obvious gains. Unchecked errors can tank careers and kill companies, and getting visibility into software errors is the first step to reducing the impact on your business. Focus on the mission-critical issues causing the most damage to your revenue in the form of buggy apps and lost customers.

One of the easiest ways to understand what vulnerabilities you have currently and make rapid progress is to start with a health check. Conduct a Cybersecurity Risk Assessment if you haven’t already and get a monitoring tool in place to get an immediate view of all the errors that are currently live in production.

If you don’t have any error monitoring in place, a monitoring tool will pay for itself pretty quick; it’s almost guaranteed that your dormant errors are having considerable business impact. You can take a free trial of Raygun for 14 days to get an initial view of the extent and severity of your errors, with full visibility and detailed diagnostics, plus the ability to measure and sort errors according to their impact on customers.

To get a health check and discover how Raygun can give your development team total visibility into errors, start your free 14-day Raygun trial.