Why Your Satisfaction Surveys Are Lying to You

When our organization first adopted Net Promoter Score (NPS), it felt like we had finally found a shared language for measuring how we were doing. Every team could use the same question. Every chart could be rolled up into a single number. We could compare our internal teams to each other and to organizations within and outside our vertical. For a group that had been wrestling with fragmented, team-specific metrics, NPS looked like great clarity.

For eight years, the numbers rewarded that choice. Our scores never dropped below 90. In some years they climbed into the high 90s. Detractors were rare. The dashboards were beautiful. Our comparisons to other organizations left us feeling good.

The only problem was that I also had a front‑row seat to our work—and the NPS numbers did not always match the reality I was seeing.

No organization is perfect, and I saw occasional hiccups. I saw tickets that bounced around longer than they should have and knew where we were understaffed or stretched thin. Occasional difficult conversations with customers or uncomfortable emails demonstrated that their experience of us was not uniformly glowing. Over time, the subtle disconnect between what I was occasionally seeing and what the NPS spreadsheet reported became impossible to ignore.

Like any good data person, I started by assuming the problem was me, or my NPS math—even though the math of NPS is straightforward.

I went back through the NPS spreadsheet, checked the formulas, and re-ran the calculations. I made sure we were applying the rules correctly: promoters as 9s and 10s, detractors as 0 through 6, passives excluded from the final score. The arithmetic held up perfectly. The charts were accurately representing what the survey data said.

The more I verified the math, the less comfortable I felt. The numbers were right. Something else was wrong.

That discomfort sent me down a path I probably should have walked much earlier: trying to understand what satisfaction surveys actually measure in the kinds of environments we work in—and just as importantly, what they quietly ignore.

When “how satisfied are you?” tells only a half-truth

When we first adopted NPS, I treated survey responses as a reasonably direct line to reality. A high score meant we were doing well. A low score meant something was broken. The assumptions under that view are simple and appealing:

• People respond thoughtfully and honestly.
• The people who respond are a good cross-section of everyone we serve.
• The way we ask the question does not significantly distort the answers.

However, if you spend any time reading the research on self-report surveys, those assumptions do not survive very long.

Researchers studying satisfaction in healthcare and other public-service environments have shown how dramatically results can swing based on survey design alone. Changes in who gets surveyed, when they are contacted, how questions are framed, or how anonymity is handled can produce double-digit shifts in reported satisfaction—with no real change in the underlying care being delivered.

Research on response bias, rating inflation, and constrained markets reveals that scores can lie in a quieter way: by appearing to be more complete and more objective than they really are. Certain groups are systematically more likely to respond. People who feel “generally fine” often lean toward polite, middle-of-the-road answers. And the most frustrated customers may be least likely to participate at all. In small professional communities, those same studies and practitioner reports point to the quiet pressure not to be the person whose harsh rating damages a relationship.

These investigations helped me separate what I directly observed from what the literature says is likely happening underneath. I cannot prove those dynamics were present in our own survey responses—but once I understood them, I could no longer read our numbers as a simple, neutral sample of “what everyone thinks.” Our consistently high satisfaction score started looking less shiny.

When customers cannot easily leave, scores mean something different

The gap between our NPS scores and my lived experience pushed me to look more closely at our vertical. In most cases, our customers do not pick us from a list of competing providers. They work with us because of structure: regional arrangements, funding models, or statewide service designs. We are not a streaming service someone can cancel next month if they are annoyed. We are part of the system they live in.

Researchers who study satisfaction in captive or quasi-monopoly environments have been pointing out this pattern for years. When customers lack easy alternatives, their answers to satisfaction questions tend to reflect tolerance more than enthusiasm. They still rate things as “good” or “very good,” but those categories quietly slide toward “good enough that I can live with it.”

In markets with lots of choice, a low satisfaction score can be a leading indicator of churn. In constrained markets, the same number can quietly coexist with deep, long-standing frustration. The survey instrument did not change. The surrounding context did.

Reading that work started to explain what I had been hearing from some customers. There were side conversations with customers who told us, very plainly, where they were struggling—but formal ratings from their organization remained high. There were operational issues that generated a great deal of internal work and frustration but barely moved our metrics. And there were the moments when people would say, “We love working with your team,” then describe a process that, by any reasonable standard, was much harder than it needed to be.

Their survey responses were not dishonest. They were answering a slightly different question—“Is this workable?”—while we were interpreting the results as “Is this excellent?”

Once you see that, the question shifts. Instead of asking, “How do we make this number go up?” you start asking, “What would we need to look at if we wanted a truer picture of what it is like to work with us?”

When the metric becomes the object of the exercise

Critiques of NPS—and other single-value metrics—also point out they can create internal behavior changes. When a single number is at the center of your story, people will naturally start organizing their behavior around it.

Accounts from customer-experience practitioners describe sharp internal changes. Frontline staff asking customers to “please give me a 10 or it will hurt my review.” Organizations quietly selecting who gets surveyed and when. Teams learning how to time invitations so they fell right after a positive interaction. The more weight placed on a single number, the more incentive there is to protect it and the less confident an organization could be that it was still telling the truth.

A Timely Example

About a week after publishing this article to the site (about 6 weeks after the site was created), I received an email from WordPress, the company responsible for both the technology on which this site operates and the servers providing its web hosting.

Their question?

It was, of course, their version of the Net Promoter Score! And they could not have provided a better example of asking the NPS in a way that is weighted toward a positive outcome! This is a screenshot of the email as viewed from my iPhone:

Measure the work people have to do, not just how they feel about you

One of the most useful ideas I encountered in my search for survey improvement was evaluating customer effort.

Research and practitioner work on Customer Effort Score (CES) suggest that effort is often a better predictor of future behavior than broad satisfaction alone. When something is consistently hard, people will avoid it, seek workarounds, or disengage, even if they like the people involved.

Satisfaction asks, “How did you feel about this interaction?” Effort asks something closer to, “How hard did you have to work to get what you came for?”

In our world, that might look like:

• A process that requires multiple emails and phone calls to complete a simple change.
• A process that takes far longer than customers reasonably expect.
• A support path where customers have to tell their story to three different people.
• A “solution” that technically works but creates extra steps for staff every day.

The more I listened to signals from customers, the more I realized that we were exclusively focused on customers’ feelings about their interactions with us. What was not being evaluated was the hard work of navigating complex forms and processes, repeating information, waiting for handoffs and approvals, and coordinating across teams.

You can start small. Ask one question in the places you already interact with customers — “How easy or difficult was it to accomplish what you needed today?” Wherever you see answers leaning toward “difficult,” you have found a concrete place to simplify the experience.

Look at what actually gets fixed

Another helpful metric emphasized in my research was problem resolution. Research on service quality and customer experience suggests that satisfaction scores really start to make sense only when you pair them with what actually gets fixed. When satisfaction is high but resolution rates are low, you are probably seeing customers who like the people they work with but still are not consistently getting what they came for. When resolution is high but satisfaction lags, the problem is usually somewhere else—communication, expectation-setting, or the broader relationship. Looking at those two signals together is far more honest than a single blended number, and it gives you a much sharper sense of what needs to change next.

From an operational perspective, tracking resolution is not complicated. You can look at:

• Whether issues are resolved on the first attempt or require multiple passes.
• How often tickets are reopened after being marked complete.
• How long it takes to reach an actual solution instead of a temporary patch.

You can see these patterns when positive satisfaction ratings show up even after issues bounce through multiple handoffs or are reopened later. Customers are expressing gratitude for the effort and professionalism of individual staff but are still living with processes that were harder than they needed to be.

Ask who would leave if they could

In constrained environments, one of the most uncomfortable—but clarifying—questions to ask is: If our customers had an easy alternative, how many of them would already be gone?

You will not find that answer in your NPS dashboard. You might see it in:

• Teams quietly building parallel capabilities in-house to support customers.
• Customer purchases from alternative providers in the areas where they still have choice.
• Side‑bar comments from employees (and, ideally, leaders) that amount to, “This works, but it could be much better”

Research on customer switching in services—for example, Susan Keaveney’s 1995 work on why people leave service providers—convinced me those patterns were early risk signals that should outweigh any satisfaction score.

Watch what people do when you are not asking questions

The last shift in my thinking came from stepping back from all formal instruments—NPS, CSAT, CES—and watching behavior more closely.

There is the official path and the path people actually take when something matters. You can learn a lot by quietly observing which inboxes and phone lines people bypass because they do not believe they will get timely help. You can usually spot the staff members who become informal ‘fixers’ who are called directly, regardless of official workflow. And you can usually see places where workarounds, shadow systems, or parallel tools have emerged because the sanctioned path is too slow or too rigid.

Researchers studying online ratings and behavioral data have shown how often there is a disconnect between what people say and what they do. Ratings can drift upward over time due to social pressure and selection effects, while actual behavior tells a more nuanced story.

I saw a similar divide in our own environment. The survey responses suggested smooth, positive experiences. Actual behavior—who people contacted, what or who they avoided, which processes they tried to circumvent—started to reveal where friction really lived.

What to keep, what to change about your surveys

NPS did help us in important ways. It gave us a shared starting point when we had none. It forced us to talk about customer experience in a more systematic way. It surfaced patterns we might otherwise have missed.

What changed for me was the role I believe a metric like NPS should play.

Instead of treating it as a primary indicator of organizational health, I now see it as one input among many—a rough, sentiment-level signal that needs to sit beside:

• Measures of effort, so we can see how hard people have to work to get value from us.
• Measures of resolution, so we know whether we are actually solving the problems brought to us.
• Behavioral signals, so we can see what people do when they are not being prompted for feedback.
• Context about market structure, so we remember that “very satisfied” in a captive environment does not mean the same thing it does where customers can easily walk away.

Research that takes a more balanced view of NPS—acknowledging both its usefulness and its limits—has been valuable here. The problem is not that NPS exists. The problem is what happens when we ask it to do more work than it can do, or when we base too many decisions on a number that is both fragile and easy to game.

Worth questioning

If you already rely on satisfaction surveys or NPS, none of this requires you to abandon them. It does invite a different set of questions.

You might ask yourself:

• What stories do our current scores allow us to keep believing about ourselves?
• Whose experience never shows up in our dashboards at all because they do not respond, they are not surveyed, or they worry about consequences?
• Where do we see consistent friction, workarounds, or quiet frustration that our metrics barely register?
• How much of our energy goes into protecting or explaining the number instead of understanding what it is trying—and failing—to tell us?

Those questions do not require a new platform or a new vendor. They require curiosity and a willingness to let your own lived experience carry as much weight as the survey results. That means getting over the story your metrics tell and examining exactly what story your organization is writing.

Can we think different?

If you set your satisfaction metrics to the side for a moment, what else could you look at?

You might start by noticing where customers or colleagues have to work hardest to get something simple done. You could track whether the issues they bring you are actually resolved, or just closed. You could pay attention to who they call when they really need help, and which official paths they quietly avoid.

You could also experiment, in a limited way, with effort and resolution questions—adding them to one workflow, one program, or one service area—simply to see what you learn.

The goal is not to produce a more sophisticated dashboard. The goal is to see your work more clearly.

For me, that has meant treating satisfaction scores as a starting point rather than a verdict, letting research challenge my assumptions about what those numbers mean in constrained environments, and paying closer attention to the gap between how we look on paper and how it actually feels to work with us.

If your own numbers feel a little too good to be true, that gap may already be talking to you. The real question is whether you are willing to listen—and what you might change if you did.

---