A few years ago, we had a shadowing and feedback session with some of our users to validate a workflow dealing with lots of media content. This type of session is just one of many tools we use throughout product development, to make sure our technology is an enabler and accelerator for consumers and stakeholders.
In this session, we reviewed how they engage with a particular flow, and after a few minutes, we noticed a bug which propagated to the web frontend. It was of relatively low severity, but sufficient to produce excess cognitive load and confuse someone new to the system.
The user we were shadowing confirmed that it was something they’ve noticed on and off for months, but learned to ignore as it wasn’t intrusive or pressing enough to report.
We learned many years ago that the relationship our users have with the product is entirely different than the relationship we, the people building it, have with it. Understanding this early on is crucial to doing good product work.
When you build technology, it’s often easy to fall victim to the reasoning that users should report errors which propagate to the UI. But let’s face it, few of us ever report bugs with Facebook or gov.uk, right? Why would proprietary systems be any different in that respect for these users, especially when they’re busy with their own deliverables?
Any technology provider worth their salt knows how important it is to have healthy relationships with the users and the business, and to make sure that they have channels to report system issues. But we shouldn’t expect them to, because it’s something that technology should be able to deliver.
There are many tools out there today that can help with monitoring an application at all levels. It’s unnecessary and unreasonable to expect a busy user to look after our product. It simply isn’t their job, it’s ours.
Since that session with our users, we’ve made significant progress with error detection and reporting in production. On average, about 100 issues are automatically logged directly to that project’s GitHub every year. For comparison, the number of bugs reported by users over the same period is usually in the single digits.
The ability to automatically collect errors, especially the ones propagating to the frontend, had a massive impact on the overall quality of the system. At the same time, it informed us about the paths that our users take when using the product, which is always valuable intelligence.
But more importantly, if we wouldn’t collect and analyse failure data, we could easily get a false perspective about the standard of the product we’re putting out there. And knowing when and how you fail is always better than not knowing, no matter how bad it is.