Users: The Most Unhelpful Oracle

There's an old saying that I mistrust deeply: users don't know what they want. I think they do, and I think that products that are designed based on the unshakeable conviction that users don't know what they want routinely turn out to be terrible.

You don't see it because sometimes this frame of mind produces good results: specifically, it produces good results when the team that builds it wants the same things that users want. The bad results that it produces -- which far outnumber the good ones -- don't live long enough to be reviewed and discussed over and over and over again on Hacker News.

In my experience, users do know what they want. It's just that they can't always state it in a useful form -- which is completely understandable, given that it's really not their job to do that. They're a bit like an ancient oracle.

The Most Unhelpful Oracle Ever

The body of a program's users is like an oracle that possesses all the knowledge required to resolve your dilemmas. But nothing is free, especially in the world of oracles. This oracle allows you to do just two things: quietly observe how it's using your program or your device, and ask yes-or-no questions.

Oracle by Eugene Delacroix

Actual footage of an engineer questioning an user about the latest version of a program.

The green fumes emanating from the ground are very potent, so the oracle can't coherently answer with anything except yes or no. You can ask a question with a more complex answer, like "why do you need a LED to blink on error, it already says "Error" on the screen!" but all you'll get is the rant of a someone who's high as a kite.

This oracle is very honest and will answer questions in the most literal sense that you can imagine. If you remove every single button from a device's case and ask it if it looks friendlier now, the oracle will say yes. The fact that it won't be able to turn it on, let alone use it in any way, won't even cross the oracle's mind (because you didn't ask).

If you ask the oracle if the device is easier to use now, the oracle will also say yes. Using the device indeed requires no effort now. The fact that it requires no effort because there's nothing to do but stare at it is not the oracle's concern.

Of course, the oracle doesn't want a device that doesn't do anything, and they are perfectly aware of that. If you ask the oracle whether they like it or not, they'll say yes, because it looks pretty, even though they might otherwise think it's the most useless piece of junk ever built. If you ask the oracle whether they find it useful or not, they'll say no. However, if you ask the oracle whether they'd buy it, the oracle may say yes -- what, you don't buy useless things that you don't need and will never use? Of course, whether they will buy it or not is a different matter. If it's useless, they probably won't buy it unless they're on a spending spree.

The oracle knows exactly what it wants. But if you ask it what it wants, all you get is an endless, incomprehensible rant, because it's not a yes-or-no question, and the oracle's mind can't handle that right now.

Sometimes, the oracle will say things without being asked, things like "I would like to be able to see all users who have not logged in in six months". You have to remember that this is still someone who's high on whatever those green fumes are. They can't give you the whole context of this requirement, so you probably shouldn't just literally do what they ask you to. If you make a button that says "Find users who have not logged in in six months", that's probably not going to be precisely what they wanted (although they'll be happy about it).

Trying to figure out why that feature is needed in the first place may point you at the right way to implement it. But if you ask why they need it, you'll get a rant. Of course, the oracle knows exactly why they need that feature, but their speech neurons are in a green fog right now.

Sometimes, if you are very familiar with the subject you're asking about, and if you use your experience and intuition, you can ask a question with a more complex answer, and you will be able to figure out what the oracle means ("sometimes employee accounts have to remain in the system until all requests are processed because other systems don't check who the account belongs to and you end up paying the wrong person the wrong amount and if I go to Anna in HR she yells at me and tells me to fix the damn thing already but we have to fix everything else so anyway you have to manually remove these accounts after all requests are processed but you can't check every day so sometimes it's smart to just wait for long enough like three or four months or whatever but you still have to do it by hand"). But you can't really be sure.

It's up to you to figure out what's the best way to do it. Maybe you want to add a more advanced filtering dialog, with a few frequent choices upfront. Maybe you just want to add three buttons: "one month", "three months", "six months or more". But the oracle is right: they need that feature.

If that feature existed and you removed it because the oracle could not explain why they needed it, that's on you. Did you not hear the part about yes-or-no questions only?

If that feature existed and you removed it because you couldn't understand how it's useful, that's also on you. If you're building machines (or writing software) for surgeons, physicians, lawyers, finance professionals, HR professionals, or for anyone who is not an engineer, for that matter, whether something should be removed isn't really your call to make. You already don't understand almost anything about what those people are doing with the things you build. If you second-guess them about things that they do, you're almost guaranteed to be wrong.

As for observing the oracle at work: it's easy, but the logic of the oracle is very difficult to decipher. It may be the psychedelic effect of the green fumes emanating from the ground, who knows. Sometimes the oracle will do things that make no sense to you. However, since they -- unlike you, remember? -- possess all the knowledge required to do that work, if you watch long enough, you'll find that whatever the oracle is doing is actually incredibly efficient.

Consequently, if you change something in your program, simply observing the oracle for a few minutes won't be enough. You have to take measurements to find if you were right. For example, you have to time how long it takes to do something to see if your changes made it easier.

Also, precisely because the logic of the oracle is so inscrutable, it's usually hard to predict how a change in your program will affect its usefulness to the oracle. Things that obviously make your program easier to use, like a reworked "Open file..." dialog, will baffle the oracle. But if you ask why the oracle is baffled, or why the old one worked better... yep, not a yes-or-no question, you'll get nothing but a rant. Probably an angry one because the oracle really liked the old dialog and you've ruined it.

Work With Your Users

It's true that, especially in the consumer industry, a lot of users can't articulate what they want in technical terms. That doesn't mean they don't know it.

They may not be able to tell you that the interface's latency is too high, but they definitely know they want it to be "snappier", "faster", "quicker". That's not exactly the same thing as "more responsive, as in, with lower latency" but you didn't ask them how good they are with computers and how well they know the jargon back when you sold them the thing, did you? If you're not prepared to say no to non-technical users, then you can't be fussy about what feedback they give you, either.

They will definitely not be able to tell you what to do about it. Maybe the display driver is really slow. Maybe you compiled Mesa with the wrong flags. Maybe running an Electron application on a device with software rendering and 64 MB of RAM wasn't a great idea after all. If that was so easy to figure out, you and me wouldn't have jobs anymore.

It's also true that, when users take a stab at designing something the way they want it, their first attempt is so specific to their preferences that it's useless. But that's true of design decisions made by the development team as well. Nobody gets it right the first time (and the ones who insist they do usually get it the most wrong of them all).

It's difficult for many of us, but users are the final and universal source of truth in our field. They are the ultimate oracle of your product's success. They are the ultimate judge of every design decision: the battery controller and battery that you use and, thus, the battery life, the time it takes to charge it, and how precise you are in your estimate about how much juice it's got left. Or the steps required to carry out a procedure, or the size and shape of a button. If the engineering team thinks it's good, then it may be good. But if the users think it's bad, then it's definitely bad.

Us engineers may have sixty years of academic research spanning back to Fitts' law and a hundred detailed blog posts, whitepapers and opinion pieces from fifty experts in IC design, compiler design, operating systems and UX to back up our opinions. Once the users have spoken, it's no longer a question about whether we're wrong, it's only a question of why we're wrong.

Ultimately, hardware is built and software is written for its users. They are, effectively, the ones who know best. It's a bit presumptuous to say that they don't know what they want, don't you think?