AI Isn't Magic: A Practical Framework for Deciding What to Automate

Cover design: Vishvajeet Shukla

We get some version of "should we add AI to this?" on nearly every new engagement now. It's the right question to ask — and most of the time, it gets answered by vibes instead of a framework. We built an entire page around this exact question (/do-i-need-ai), and this post is the reasoning behind it.

Start with the failure mode, not the technology

The wrong starting question is "can AI do this?" — the answer to that is almost always yes, for almost anything. The right starting question is: what happens when it's wrong? A task where a wrong answer is mildly annoying and instantly correctable is a good AI candidate. A task where a wrong answer is silent, costly, and hard to catch is not — no matter how good the model.

Three questions before writing a line of ML code

1. Is this task repetitive at real volume? If a human does it twenty times a week, that's not enough volume to justify a real pipeline — a checklist or a simple rule-based tool will beat AI on cost and reliability. The volume has to be high enough that the maintenance cost of an AI system is smaller than the cost of the manual work it replaces.
2. Does the task have a "good enough" answer, not just a "correct" one? Classification, summarization, and drafting all tolerate a range of acceptable outputs. Exact computation — payroll math, tax calculation, financial reconciliation — doesn't, and shouldn't be handed to a model no matter how tempting the demo looks.
3. Is there a fallback when the model is uncertain? Every AI system we've shipped that actually held up in production has an explicit low-confidence path — escalate to a human, ask a clarifying question, or refuse to answer rather than guess. Systems without that path degrade silently, and silent degradation is how trust in an AI feature gets destroyed in one bad week.

What "not worth automating" actually looks like

Some of the most useful conversations we have with prospective clients end in "you don't need this yet." A one-off internal report that runs monthly doesn't need a fine-tuned pipeline. A support inbox getting a handful of genuinely varied questions a day doesn't need RAG — it needs a well-written FAQ and a real human, because the volume doesn't clear the bar where automation maintenance pays for itself.

AI earns its maintenance cost when three things line up: real volume, a task that tolerates "good enough," and a safe fallback for the cases it gets wrong. When those three don't line up, the honest answer is a simpler tool — or no tool at all.

The interesting question was never "can we build this with AI." It's always "should this specific task be automated at all, and if so, with what."

That's the exact framework behind our own Do I Need AI assessment — worth five minutes before committing engineering time either direction.