Four hard problems in enterprise AI: the pilot succeeds, the rollout stalls
Almost every manufacturer that has run an AI pilot has seen the same scene: heads nod all round during the demo, and three months later nobody opens it. The problem rarely lies with the model — it stalls on four things that should be faced early and are the easiest to skip.
What you'll take away
- The four hurdles in order: scattered data, no domain knowledge, not embedded, no business case — and the order cannot be reversed
- AI stitches contradictory metric definitions into one complete-sounding story — more dangerous than failing to answer
- A general-purpose model knows language, not your business: 8D, operation codes and customer-specific requirements all have to be taught — and it must be able to cite its source
- To judge whether AI has truly landed, ask one question: switched off for a week, would anyone come asking you to bring it back?
- If any one of the three accounts — security, accountability, ROI — cannot be made to add up, the project clears neither approval nor acceptance
Problem 1: scattered data — you cannot cook a meal without rice
At a monthly quality review, the quality manager reported an incoming defect rate of 2.3% for the previous month. The purchasing manager objected on the spot: by his definition it was 4.1%. Ten minutes of argument later they found both were right — one counted by batch, the other by purchase value; one took QMS inspection results, the other ERP return records. The data was all in the systems; it had simply never been aligned.
That argument costs ten minutes in a human meeting. In front of an AI it takes a more troublesome form: it will not say “your definitions disagree” — it stitches the two numbers into a complete, smooth-reading conclusion, say a middle value wrapped in a line about quality being “stable and improving on the whole”. What management sees is a confident answer, not an unfinished argument.A confident wrong answer does far more damage than an honest “I don't know”, because it gets used as a basis and carried on — until a customer complaint digs it up.
So for AI, “scattered data” is not a minor flaw you patch as you go — it is an entry requirement. Manufacturing data is typically spread across ERP, MES, QMS, WMS, Excel and paper forms: one material with several codes, one customer with several short names, one metric with several formulas, one document with a different status in each system. However capable the model, it cannot derive a trustworthy output from contradictory input — it has no extra source of information and can only polish whatever you give it into the thing that most resembles an answer.
Self-test on the spot:Without opening a system or asking anyone, can you say which document last month's incoming defect rate came from, by which definition, and covering which organizations? If answering that takes three people in a room reconciling definitions, then putting AI live only speeds the argument up — it will present all three answers at once, each in a confident tone.
The pragmatic route is to do two unglamorous things first: a single source of truth for master data, and a metric dictionary (who defines it, when it takes effect, and a change trail). Then let AI consume only frozen definitions. There is no shortcut here and no model that can skip it for you. Projects that skip it usually come back to do it half a year later, by which time some of the budget and the patience are gone.
Problem 2: no domain knowledge — the general model answers the wrong question
Paste the story of a customer complaint into a general model and ask it to analyse the cause. What comes back is usually a handsome generic template: confirm the problem statement, analyse possible causes, then define corrective actions with follow-up verification. The process is right; it is simply unusable — because what you actually need to know is that this is the second case this month on the same operation, that the previous corrective action is still in its verification period and not closed, and that the customer raised the same special requirement three months ago. All three facts sit in your systems; the model knows none of them.
A model holds the patterns of language, not your business constraints. It does not know what your plant's operation codes mean, how deep a root cause must go in an 8D to count as closed, what the customer-specific requirements (CSR) register says, or what IATF16949 expects, in time and evidence, of a closed corrective loop. None of this is in public corpora; it can only come from the company's own material — policy documents, SOPs, historical records, defect code dictionaries, CSR registers and past audit findings.
How that knowledge is injected is the crux. Pouring all of it into the context usually yields something more dangerous —an answer that can quote the jargon but cannot point to where it came from. It reads more like an insider than a generic template, so it is harder to challenge on the spot and harder to correct afterwards. Acceptable domain injection must be searchable and citable: the answer can name which version of which document and which batch of records it rests on, so the reader can follow the trail and check it themselves.
One judgement principle applies neatly here: the asymmetry of error costs. If AI drafts a production material list and gets it wrong, you fix it once; if AI makes a quality judgement and gets it wrong, a batch may already have been released. So the steadier division of labour in domain scenarios is this — AI compresses three hours of filing into three minutes and hands you candidates and clues; judgement and signature stay with people. It does not conclude for you; it just means you leaf through fewer boxes of paper before you do.
Self-test on the spot:Take a real internal term or defect code and ask it. See whether it answers correctly or invents a plausible-sounding explanation. That single question separates “knows the domain” from “sounds like it knows the domain”. If it cannot get even this right, it will not help in a review meeting — it will only add one more paragraph for the meeting to correct.
Problem 3: not embedded — the agent stays in a chat box
During the pilot you built a Q&A bot and a document summariser, and the feedback was decent. After two months of demos, back-end visit records fell to single digits. The reason is usually not that it works badly, but that it is not on anyone's work path — to use it you must first remember there is a chat box, then switch to it, then phrase the question well. Any one of those three steps is enough to send people back to the old way.
Enterprise software has always been pushed along by process: a to-do left undone goes overdue, a document not reviewed stops the flow, a production report not filled blocks the month-end close. A feature hanging off the process does not need anyone to remember it; a feature that only waits in a chat box is always filed under “later”.
The test can be very simple:If this AI feature were switched off for a week, would anyone come asking you to bring it back?If nobody does, it has not entered the process. That question is more accurate than any usage report, because it asks about dependence, not interest.
What actually lands is embedded in an existing step rather than given a new entrance. A few concrete manufacturing examples: when a nonconforming material review form opens, pull in similar historical NCRs along with the disposition taken and the final outcome; before incoming inspection is entered, bring up the supplier's 90-day defect trend and any open 8D; when a production report is submitted, flag values outside the historical range with possible causes; when a change order is submitted, list the in-flight changes linked to it, so that fixing one place does not miss three others. They share one trait — the user does not have to “go and ask AI”; AI appears at the step they were already working on, offering checkable candidates and clues rather than a verdict on their behalf.
While we are at it, draw the boundary: an agent is an acceleration layer and replaces none of the engines — organization, permissions, forms, process, reporting or integration. It can push to-dos and raise prompts, and within that person's own permissions it can create records and complete reviews through the business API. It does not approve for someone else, does not change master data, does not bypass row-level permissions, and does not write the database directly. Thinking of it as just another business module is clearer — it has defined inputs, defined boundaries and a named owner, rather than being a window that answers anything.
Self-test on the spot:List your three most frequent business actions and ask whether AI appears in any of them. If it only comes to mind at the fourth — “I don't know who to ask” — then it has not landed; it is just one more optional chat entrance.
Problem 4: the numbers do not add up — three accounts block approval
The first two hurdles are cleared, the third is done, and it still stalls in the meeting room. Not a technical problem — nobody can make the three accounts add up on the spot.
Security account— can the data leave the company? Which fields may go to an external model, which must stay on the intranet, and what record is kept of calls to external capability. This is the question most easily left to the end, and if the answer turns out to be “it cannot leave”, a solution built on an external model has to be scrapped and rebuilt. Asking a month earlier often saves the whole schedule.
Accountability account— if an AI suggestion leads to a wrong call, who is accountable? If the answer is “the AI suggested it”, the project should not go live, because that means nobody is. An acceptable design: AI supplies clues only before a person decides, the decision and the signature stay human, and every generation and adoption is logged, so you can answer who decided what, when, and on which knowledge. The log is not for assigning blame; it is what makes people willing to use it.
ROI account— this is the account most easily fudged. Writing the benefit as “30% efficiency gain” cannot be evidenced, so the acceptance meeting is bound to argue — and the argument is rarely lost because the benefit is too small, but because it cannot be stated clearly. What passes is tying benefits to four evidential categories: labour hours, scrap and rework, audit and compliance cost, and delivery and cash cycle — each with its own before-and-after method of proof. A benefit you cannot evidence, however attractive, does not belong in the approval pack.
| Account | Question to answer on the spot | Consequence of getting it wrong |
|---|---|---|
| Security account | Which data may leave, and where the call log is kept | The solution is scrapped mid-way, costing both schedule and trust |
| Accountability account | Can an error be traced to a person, a knowledge version, a data batch | When something goes wrong nobody dares use it; the feature ships and sits idle |
| ROI account | How each benefit is evidenced | Acceptance becomes a debate and the next budget is not approved |
If any one of the three accounts cannot be made to add up, the project will not clear approval; force it through and it will not clear acceptance. They also share a trait: none of them shows up in a demo, so they are the easiest to skip — and the easiest to have come back all at once at project close.
Self-test (security):List your five most sensitive fields and check whether the current design states clearly whether each may leave the country, who approves it, and where the record is kept.
Self-test (accountability):Suppose a wrong AI suggestion caused a batch of misjudgements. Can you locate, within two hours, which knowledge version and which data batch caused it? If not, the logging has not been designed yet.
Self-test (ROI):Go through the expected benefits one by one and ask “how is this evidenced”. Strike out every line you cannot answer for right away — better to strike it now than have someone strike it for you at acceptance.
Get the order wrong and the money is wasted: the four hurdles have a sequence
The four hurdles are not four topics to pick from at will; they form a sequence of dependencies: without unified data and definitions you feed in contradictory input; without traceable knowledge, the more domain-savvy the answer the more dangerous it is; without process embedding, however accurate it is nobody uses it; and without the three accounts, the first three can be done perfectly and the project still will not clear approval. Skip any link and you come back to it later.
The most common way to crash is exactly to reverse the order:buy the model and the platform first, then go back and patch the data and definitions.The budget goes on the visible part, the invisible part blocks it, and half a year later the project is shelved as “AI is not mature enough”. The real reason is usually not that AI is immature but that the order was wrong — and the cost of that mistake is finally charged to AI, making the next approval harder.
| Hurdle | Typical symptoms | Do only after clearing | Pragmatic move if not cleared |
|---|---|---|---|
| Data and definitions | One metric, three numbers, no one can say where they come from | Let AI consume frozen definitions | First make master data unique and keep a metric register; let AI only assist entry and validation |
| Domain knowledge | Handsome answers that are useless; the terminology does not match | Let AI assist quality and delivery judgement | Build traceable knowledge sources first; start with retrieval and citation, do not rush into generation |
| Process embedding | Impressive in the demo, never opened afterwards | Put AI before the decision points | Start with prompts and handling within that person's own permissions; do not auto-approve or write the database directly. |
| The three accounts | Stumped at approval, arguing at acceptance | Expand the scenario scope and the organizational pilot | Start with the scenario with the smallest account — usually internal assistance |
One more pragmatic line belongs in the project charter:If the second hurdle is not cleared, better to delay business analytics by six months than to burn management's trust with one wrong answer.Trust is a one-shot resource. The lesson bought with one wrong answer costs far more than waiting three extra months — and it spends not only this project's credit but the credit line for the next budget request.
With the order right, you still do not have to wait for everything to be ready. The more workable route is to let AI run in low-cost scenarios first (assisted entry, similar-case retrieval, field validation) and use real usage records to expose the messiest parts of the data, which in turn drives definition governance. Two lines pushing each other along gets closer to landing than waiting for a perfect starting point.
JEDP puts this in the product structure rather than leaving it to the proposal deck: AI is an acceleration layer on top of the existing engines; design-time generation takes effect only after human confirmation; runtime queries consume only frozen definitions; creating records and completing reviews goes only through the business API, stays within that person's permissions, and is fully logged; and permissions inherit the user's own scope without widening visible data. Use those four to check a vendor directly — if they cannot say clearly which data a questioner can see, or how a wrong answer would be traced, they have not walked the four hurdles yet.
- JEDP Platform Capability: Dual Intelligent Agents
- Where AI should enter enterprise systems: keep design-time and runtime agents separate
- Information Technology Selection Scoring Sheet (Self-Test Without Providing Contact Information)
- Platform and Dual-Agent Sandbox Trial
Want to health-check your own plant against the four hurdles?The Resource Center has matching online self-tests and document templates, or you can book a demo in a real environment.
FAQ
Quick answers on this topic—still stuck?Contact
Which of the four hurdles is the hardest?
Data and definitions. It never appears in a demo and has no technical shine, yet it decides whether the next three steps are possible. Look back at where most projects got stuck and it is this one — and the later you patch it, the more it costs.
Why does it get messier after AI goes in?
Two common cases. First, definitions are not unified and AI is let straight to conclusions, so it stitches contradictory numbers into a complete story; second, AI's visibility exceeds the user's own and it answers questions about data that should not be seen. The first damages judgement, the second is a leak; both must be blocked at design time.
Our data is already messy — does that mean we cannot use AI at all?
You can, but narrow it to low-cost scenarios: assisted entry, similar-case retrieval, document field validation — uses where a person reviews the result anyway. Running AI there first actually exposes the messiest parts of the data fastest and becomes the entry point for governance, rather than waiting until governance is finished before daring to use it.
Can we deploy a private model first and take the rest slowly?
We would not advise reversing the order. The model is the easiest of the four hurdles to obtain and the easiest to replace. Only with trusted data and traceable knowledge does model capability have somewhere to work; otherwise most of the criticism after go-live is not about the model, yet all of it lands on the model.
How do we tell whether AI is genuinely in use or just looks good in the pilot?
Two hard indicators: whether anyone asks you to restore the feature after a week offline, and whether usage is concentrated on ordinary days after the demo period rather than at reporting milestones. Add one soft signal — users start complaining “why is there no AI here”, which is when it has truly entered the workflow.