Every time I entered the same prompt, I got a different answer, and I had no idea what was actually true
Early in my AI use, I was writing a white paper and decided to use AI for the research I would normally do on Google. I typed a research question and read the answer. I entered the same prompt again and got something different. I tried a third time. Different again. The weirdest results, every single time. I was sitting there with three versions of what the tool was telling me and no way to determine what was actually true. I decided to fall back to Google. No AI output went into that white paper. I went back to work, but the confusion stayed with me.
That experience did not make me an AI skeptic. It made me someone who did not understand what the tool was actually for.
The question is not whether AI is good, it is whether your task is the right shape for it
I was one of them as well during my initial days. Someone who assumed that because AI could answer questions, it worked the same way a search engine worked. That assumption is the root of every AI task selection mistake I have seen, and I made it myself first.
The white paper failure was not a prompting-craft mistake. I was not typing three keywords and pressing enter. I was asking a real research question repeatedly and receiving real-sounding answers that contradicted each other. That is a different kind of failure. The real question this post answers is when to use AI and when to reach for something else.
AI predicts the next probable word in a sequence based on patterns in its training data. It does not retrieve indexed facts from the real world. So how do you decide, before you start, if a task is right for AI?
Usain Bolt would not win a Formula 1 race, and that is not a criticism of Usain Bolt
The fastest human alive is not the wrong athlete. He is the wrong tool class for a motorised race. Applying him to that race does not reveal his weakness. It reveals the selector's mistake about which tasks belong to which tools.
AI is calibrated for pattern-matching and generative transformation. Inside that frontier, it works at a level that is hard to match by hand. Outside it, performance drops in ways that do not track human intuitions about task difficulty. A language model can draft a meeting summary with precision and fail a straightforward factual check in the same session. That is not a bad prompt. That is the shape of the tool.
I want to be pragmatic and not AI maximalistic. Not "AI is good" or "AI is bad." The question is always fit.
The research confirms the same paradox I experienced: exceptional on some tasks, surprisingly bad on others
The productivity case for AI is real when the task fits. A study on GitHub Copilot found that The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. That is a real number from a controlled experiment. But it came on a coding task that is by definition repeatable and process-known. The developer knows what a correct program looks like. That is where AI works best for repeatable structured tasks: the human can catch a deviation because the expected output is already understood.
The harder question is why AI at work produces wildly different results on tasks that feel similar in difficulty. A NeurIPS 2023 research paper named this directly: Transformer large language models (LLMs) have sparked admiration for their exceptional performance on tasks that demand intricate multi-step reasoning. Yet, these models simultaneously show failures on surprisingly trivial problems. Great at complex reasoning, stumped by trivial problems. The paradox is what makes AI task-fitness unintuitive. Difficulty as humans perceive it does not map to difficulty as AI processes it.
Ethan Mollick, professor of management at Wharton, gave that paradox a name in an essay on AI's shape: the weird ability of AI to do some work incredibly well and other work incredibly badly in ways that didn't map very well to our human intuition of the difficulty of the task. The mismatch is not random noise. It does not follow human intuition, which is why the filter needs to be explicit, not instinctive.
Why does AI give different answers every time on factual questions? Because consistency is not what it is designed for. In a separate essay on prompting, Mollick is direct about this: it is very hard to reach 100% consistency with LLMs. That is not a flaw to be patched in the next release. It is structural. Tasks that require the exact same correct answer on demand are a poor fit for the tool today.
The honest objection is that AI's factual abilities are improving fast, and that objection deserves a real answer
Retrieval-augmented generation, grounding, and web-search integrations are closing the factual-retrieval gap. That is true and I acknowledge it. The frontier moves.
But the filter is not about where AI will be. It is about the task in front of me today, with the tool available today. Even with retrieval-augmented tools, I cannot verify the output without independent research unless I already have the domain knowledge to catch a wrong answer. If I already have that knowledge, I did not need AI to find the fact. If I do not have it, I am relying on a model that the research shows struggles with the very consistency that factual retrieval requires.
"Improving" does not equal "verified." The AI vs search engine gap narrows with grounding tools, but the verification problem does not disappear. This post is not written against a future version of the technology. It is written about the decision I face before I open the chat window.
The filter I use now is two questions I ask before I open the chat window
The decision filter for using AI at work comes down to two questions.
First: is the process repeatable and known, and could I catch a deviation if the output went wrong? When I draft a structured email type, summarise a meeting transcript I attended, or reformat a requirements table, I know what good output looks like. I can spot a gap. The formulation that stays with me is this: "you know the process and there is no deviation from that process." That is the condition where the filter confirms the task belongs in the repeatable category AI handles well.
Second: am I looking for a specific factual answer that must be anchored to the real world? A statistic, a named attribution, a citation. If yes, I open a browser. The filter resolves the everyday question of when to use AI vs Google for research-style tasks: Google retrieves indexed facts; AI generates probable text. Those two mechanisms are not interchangeable on factual queries.
If question one is yes and question two is no, I reach for AI. I stay behind the wheel.
The trap is using the filter to open the chat window and then forgetting to stay in front of the output
Task selection is necessary. It is not enough on its own.
A well-matched task - repeatable process, known outcome shape - can still produce output with embedded confident errors. The filter selects when to start. It does not waive the obligation to read every line before that output leaves my screen.
Picture a business analyst who runs the two-question test correctly. The task fits. She opens the chat window, gets a well-structured draft, and sends it without reading through. The draft contains a figure the model produced with full confidence. That figure is plausible, wrong, and now it is in a client document. Task selection was correct. Output review was skipped. The mistake still shipped.
The filter and the read-through are two parts of the same discipline. One tells you when to open the window; the other keeps you from closing your eyes once the output arrives.
The filter does not make AI smaller, it makes your use of it sharper
The two questions do not subtract from what AI can do. They reduce the time you spend confused about what it just gave you.
Once I understood that the white paper failure was a fit problem and not a quality problem, I stopped drifting away from AI and started keeping a working list of tasks where it earns its place. The question I carry into every session is not "should I use AI?" It is the narrower one: is this specific task the right shape for it?
That one shift - from a general verdict about the tool to a task-by-task test - is a ray that leads your way.
