The XY Problem (2014)

The most amusing variant of the XY problem is when you want a seemingly ridiculous Y, and then, when people inevitably ask why, you tell them about your entirely-novel X, which they still don’t understand the use for, so you have to explain the entirely-novel W that motivates X, and so on... and by the time you’re done, you’ve explained your whole business model, written a ten-page journal paper, and completely killed the conversation—because they don’t feel they have the prerequisite knowledge to talk about A-through-X, so you just get a “good luck with that, buddy.”

Even though Y was a concrete problem that they do have domain experience in solving!

I call this “going up the rabbit hole.”

——

Example (deliberately shortened from its real depth):

• I’m trying to invent a novel algorithm for inter-procedural data-flow analysis, because none of the existing ones work for me.

• Why? Because I’m trying to reverse a stack machine bytecode with only indirect jumps and no reified CALL/RET opcodes, into a structured inter-procedural control-flow-graph.

• Why? Because I want to do program-slicing to recover HLL representations of key-value-store writes that the program does.

• Why? Because I want to use them as hints for my runtime execution tracer for said bytecode, so it can emit better traces.

• Why? Because I want to put those traces into an OLAP database and I want the data in the DB to be typed.

• Why? Because these traces are the only canonical representation of the state of the system, and our business model is allowing auditors to figure out what’s going on with the system.

It's pretty lame that the examples on the page show the answerer not just answering but also talking down to the questioner. XY is a problem that happens a lot, but disparaging people who are trying to learn is not the right response.

Here's how that conversation should have gone:

---

Q: How can I echo the last three characters in a filename?

A: If they're in a variable: echo ${foo: -3}

A: However, if what you really want is to get the file extension, it's probably better to handle extensions of any length. Try this instead: ${foo##*.}

A: That expression says to find the longest prefix of `foo` that matches the given glob pattern, and remove it.

Q: Oh wow! That is actually what I wanted, but it hadn't occurred to me that such powerful pattern-matching would be built into the shell. I already had a solution involving piping through sed, but it was pretty slow and ugly. I noticed that all the files I'm operating on have three-letter extensions and I figured there'd be an easier way to take a slice of a string, since that's a common thing to have built-in. But your answer is the best of both worlds. TIL!

---

There's really no need to admonish the questioner for starting with the wrong question. That won't help. It will only make them afraid to ask questions at all.

I agree with the rest of the commenters here that the XY problem problem is a way bigger one. Example:

"How do you implement a new system call in the Linux kernel?"

"Ah, well, a system call is overkill and leads to compatibility issues, you should try to see whether you cannot solve your problem some other way..."

"My problem is I want to understand how system calls are implemented thus I want to go through all the steps of making a new one so that I don't miss something"

Or, a more recent one from Reddit:

"When will we have desktop computers capable of replicating the feat of Google with AlphaGo?"

"Ah, well, we already have distributed computing trained NNs, also there is some research that it may not require such power..."

"I'm not interested in Go, I'm interested in the power of future computers!"

Seriously, people second-guessing you are annoying.

Real conversation I've had:

"Does Django's query builder use prepared statements by default?"

"What are you really trying to do?"

"I'm trying to find out if Django's query builder uses prepared statements by default..."

Not everything is an XY problem. If somebody found that question on SO via Google, they would be pretty annoyed to see it XY'd instead of answered.

OTOH I find it quite annoying, when asking about Y, to be bombarded with questions like 'why would you even do that' etc.

Eg,

Me: "What's the best trackball in 2019?"

Crowd: "Do you have RSI? You probably need a vertical mouse."

etc

Unfortunately, the assumption that "someone who's asking for help with something doesn't really know what they want or need " breeds horrible culture.

This is evident by the examples in the article and the ones in the cited sources. If someone is asking for help, don't try to shame them into explaining what "they really want". What they want is help with a morsel of a problem they ask, nothing more. If it's not the right path to a solution, they will decide, not you.

Most comments here are about how annoying people asking about X are.

We get it, sometimes you are asking about X simply because you need to know about X and you know what you are doing.

But often, especially when dealing with more junior engineers, I've found this simple to keep in mind and helpful. It's also a good proof of experience to know when to ask about X and try to make sure you understand the context of the question before answering it.

OTOH, wanting to do y for other reasons, finding an xy SO post on Google, and not being able to get the answer because it went off topic about rather or not y is the solution to X is frustrating.

Bonus points, ask y again because the other question answered z only and get it closed as a dupe of the other question.

I'm one of the people who tend to ask why. Sorry to everyone on this thread who are frustrated by it.

From my point of view, I'm really not trying to undermine what you're doing, it's a genuine attempt at getting you the best solution.

Here are some examples:

>How do I use apt on Fedora?

>How do I unroll this bash loop?

>How do I multiply this number by 1024 with sed?

I could have a lot of fun answering each question exactly as stated and if you say "I know it's the wrong tool for the job but I'm doing it for fun" that's exactly what I'll do.

However, it turned out that:

>They just wanted to install stuff thinking apt was universal. The better solution was using yum

>Their loop was slow and they had read that unrolling was an optimization. The better solution was reducing the number of subshells spawned

>They just wanted a result, not to implement an integer multiplication state machine in sed. The better solution was using perl/awk

I know some people are jerks about it, and that sometimes you want answers to weird questions. When people do appear to be going down the wrong path though, it would be a disservice to help them do that without letting them know that there's probably a better way.

Sorry for any false positives.

One thing I hate is when the XY problem gets encoded into a knowledge base:

1. It's common for people trying to do Y to ask about X, which is a bad way to accomplish Y. 2. I'm trying to accomplish Z, for which X is a perfectly good option. 3. Every answer on stack overflow for "How to X" is people assuming the XY problem and answering "here's how you do Y instead".

The question, "what are you trying to do?" is super useful to remember to ask. I work at a company that's trying to modernize and move to devops for dev environments. Because the infrastructure is not nailed down, this means that every single team needs to have at least one person who understands the devops.

The process of learning k8s and grokking the security concerns in the course of a few sprints is not pleasant. This "XY Problem" crops up all the time, but the devops team is absolutely slammed all the time so many times basic queries go unanswered. Everybody has to help everybody else.

I don't think anyone wants to go back to the days of rigid development infrastructure so we're kind of stuck with devops until the ecosystem can mature.

My great hope is for serverless to take off and so programmers can further specialize. I'm getting tired of learning new languages and stacks every few months or years. Just hand me a biz requirement and let me implement it any way I like.

Conversely, there is the XY fallacy: when one [frequently condescendingly] assumes they know better than the person asking, errantly classifies it as an XY problem, and fails to answer the original question.

Depends. This is a matter of prudential judgement. The proper attitude is contextually determined and the linked page lists some qualifications, though still takes too much of a narrow, absolutist attitude.

For example, if someone random poses a problem on IRC, the solution to the "real" problem to be solved is NOT the concern or responsibility of the people in the channel. Unless there's a reason to ask for more context, just answer the damn question. Blindly asking the user what they're REALLY trying to solve is taking an intrusive and patronizing attitude. However, if asking will provide helpful context, then that's a valid reason to ask. If the problem as posed seems onerous or contrived, you COULD courteously ask the other person what they're ultimately trying to solve in the spirit of offering to be helpful if the other person so desires, but no one is hiring you do solve the "real" problem for them, so mind your own business.

If a coworker approaches you with a question, similar principles apply, but if you're working on the same project and in the same context, it may be natural to ask about what the larger problem is in a spirit of collaboration, esp. when the questions posed by your coworker smell funny. After all, you both share concerns about a project, though again, this does not give you license to automatically wrest the problem from your co-worker. His ticket, his problem. Your role is advisory.

So, common sense. Don't be one of those doltish know-it-all jerks who like to run the show. Mind the proper boundaries of concern.

Is it just me or is the XY Problem really poorly named? It's unlikely that anyone could hear "XY Problem" for the first time and have any idea what it means.

I often have the problem that I want exactly X and thousands of Internet know-it-alls tell me that I want Y.

Sometimes they are right. Often they are wrong.

Stop making assumptions, people!

Here's the thing, if you ask an unusual question, one of two scenarios is often playing out:

In case 1, you are a beginner. You have an XY question. But you've never heard of an XY question. You also probably couldn't identify your question as an XY question if you had heard of it. Trying to educate people about XY problem is pointless, because beginners don't see it or understand it or have the ability to apply it.

In case 2, you are advanced enough to avoid asking XY questions. You are instead stuck on some esoteric question which you can't resolve by exhausting all your skills. So you post a question on the internet. But you are only asking a question because it is a really hard question. Chances are that most other people cannot answer the question either. So, the only responses you get are people suggesting that you have an XY problem.

I have exactly the opposite problem. I ask, very honestly, for X, and then people incorrectly assume that what I really want is Y.

Just give me X or shut up!

If a programmer whom I know and respect in general asks me how she can do X, I tend to start with an answer about X.

When a name I've never seen before on a -users mailing list asks me how to do X -- and X is bizarre -- I assume we have the XY problem.

We have the XY problem because the Internet is full of newbies at any given point, and we've wasted enough attention-span having sixteen-part conversations about how to extract the IP address from ifconfig output* that we'd really like to know if we can skip all that and answer with "Use a dynamic DNS client, don't write your own."

*Don't. Use ip a

...

Use awk.

...

There are an infinite number of guides to awk, but mostly you need to know how it splits strings into fields and prints the ones you specify. How are you planning on handling machines with multiple IPs, by the way?

...

Stop right there. Get a dynamic DNS client.

I just want to point out that the example is really poor behavior on the part of the teacher: Shouting in all-caps to ASK FOR WHAT YOU REALLY WANT is more likely to result in the newbie not asking at all next time.

If this is your work environment you need to make it okay to ask questions, and to dialogue about a problem without shaming the learner. (Really that should be civil behavior in any environment.)

While it's an important phenomenon to be aware of, its name fails to carry any semantic meaning. Should be named something like "falsely known solution" or "context stripping" or something

Lazy social-signaling drives the over-application of the "XY Problem".

In a private medium, Questioner and Answerer can delve into the wider context and arrive at a well-informed solution.

In public however, any line-of-inquiry is corrupted by its presentation. In the specific context of public Q&A, this means that the motivation of the average answerer is often just to _present_ themselves as knowledgeable without actually being helpful.

In most difficult questions I've seen posted on SO, most answerers have neither an understanding of the context nor the basis with which to compose the "right" answer. Compounding this ignorance is that the Answerer fears losing face and also that he or she is not not personally or deeply invested in the question itself. Thus, a clueless Answerer who accidentally or foolishly interjects in the thread, will eventually default to a simple strategy: negotiating the question.

Less "digging" and more "dumbing", the question is simplified or distorted beyond its original intent so the Answerer can "answer" or rather discount it. They thereby "resolve" their stake in the question posed: giving them (or at least not hurting) the esteem they crave in the larger group. At the very least, they'll have said something and appeared, in their minds, to be just as smart or smarter than the questioner.

StackOverflow's basic design weights against this kind of child's play, since only the Questioner can award the Answerer the checkmark. But I think the more popular Questions still suffer from the transients desperately seeking their ego-fix.

It makes me think that whole software projects (and popular ones at that) have fallen prey to the XY problem. For instance, are containers really necessary, or are they shoring up a lack of robust dependency management allowing different software programs to use different versions of the same application simultaneously.

I remember in the early days of phone apps, you'd find thousands of different apps that try to solve a trivial problem. It inspired me to make this video: The alarm clock problem https://youtu.be/ebQAM5ADfYQ

Can someone explain the rationale of setting up a domain for just one blog post?

As well as seeming like quite a lot of effort, the fact domains expire means that content probably won't past 2 years unless you fancy keeping paying for the handful of visitors it'll get.

Basically lifted straight from https://mywiki.wooledge.org/XyProblem. GreyCat's wiki is the best Bash resource in existence, but the first example is pretty typical for the IRC channel. There's really no reason to berate people. For example, lots of people have never or very rarely been exposed to anything other than three letter extensions (including most DOS/Windows users), and it's entirely possible that they were trying to do something which only involved three letter extensions. Not every script needs to be a completely generic solution.

Give people context and explain your end goals.

It's quite funny, the number of replies in this thread saying: "No, I really do need Y because I'm special!"

Just remember where you are. The XY problem is a far larger problem in a far larger population than HN.

The problem comes in many guises. A similar problem arises when someone says "Y" doesn't work. After going on a long trek you find out "Y" is just fine - they had some other problem "X", had done some diagnosis and decided it was caused by "Y" being broken.

After many years of being dragged down this rabbit hole my first question is now "what makes you think Y isn't working". If the answer is "because X is doing ...", you've just dogged another hours wasted time in another rabbit hole.

This gets reposted now, a day (ish) after I saw someone comment "is this an instance of the XY problem?"

I suspect that people are mining data from ("paying attention to") certain posted keywords and then re-posting.

How might I disprove the null hypothesis that people are just repackaging highly voted comments? How might I convince myself that I am wrong or right? Please show your work or cite sources.

The conflict between describing X vs Y is also about a few things:

- the size of signal: asking a pointed (Y) question saves typing/reading/etc. in some situations - saving the person time and showing that you came to them after some thought (rather than with the original problem)

Asking Y can quickly become about X even if you begin by asking Y.

e.g.

Y: where do I find the foo.bar config value

X: I need to implement feature to _k_

I often encounter this problem in the form of newbies asking how to dynamically create variables inside a loop:

"foo1 on the first iteration, foo2 on the second, etc."

As much as I am a fun of weird little endeavors like programmatically modifying the local scope, my answer is usually along the lines of "...arrays."

The problem most often is asking X because I want exactly X, but people assume I want Y!

This presumes that the asker is intentionally concealing information. I think it's better to assume that they just don't know that some details of their problem might be extremely relevant...

Thread from 2015: https://news.ycombinator.com/item?id=10023882

“I want to make more money.”

Yes, but what do you really want?

I know solution to hiding OS from nmap. Use exotic OS, like ReactOS, XTS-400, Genode, etc.

I made this slide about XY at a previous job: https://i.imgur.com/hH97U9b.png

The inverse also happens:

You: I want to find a solution to X.

Response: you probably don't want to solve X, but rather W. Here's a solution for W, ...

You: sigh

> n00b doesn't actually want the last 3 chracters in a filename, he wants the file extensions, so why ask for the last 3 characters?

n00b doesn't actually want the file extensions either, he wants to <insert purpose of the program he's working>.

n00b doesn't actually want <insert purpose of the program he's working> either, he wants not to get fired from his job.

n00b doesn't actually want not to get fired from his job either, he wants to acquire money.

n00b doesn't actually want not to acquire money either, he wants to avoid starvation.

Looks like the answer to "How can I echo the last three characters in a filename?" starts with "Buy some seeds".

There's a money side to this that's also important.

Had a friend call me a few months back with a technical problem. "How do I fix Y on my server?"

Now I knew that server wasn't bringing in a lot of money, and the Y he wanted to fix could be accomplished the same way with say, 100 bucks of purchasing something else. But I figured it was better if he figured this out himself.

So I asked him what X he was using it for.

That was it. We never got past that, mainly because he was dead-set on spending a week or two of effort and tens of thousands of dollars on something that was (pretty much) trivial.

So after circling the barn a few times on the phones, I gave up. Told him that yes, based on what he was saying he had a fix for Y that was fine. It was just going to take a long time and cost a lot of money.

"But there has to be a quicker, cheaper way!"

Argghhhh.

Yes, there is. But if you can't figure out that you're solving the business problem first, the technical problem second? There might not be one. Technology exists for a reason, and technical problems exist because business needs are not being met. Fix the business problem. It'll drive the tech decisions.

It was quite a frustrating call. For both of us. The X of the XY problem might be critical. Or not. People who ask dumb questions get dumb answers.