A useful mental model of how these systems work, and what that mechanism implies about where you can safely put one.
Prediction, not retrieval
A large language model is trained on an enormous volume of text and learns, statistically, which token tends to follow which. Ask it a question and it does not look up an answer; it produces a continuation that is likely given everything it has seen.
That single fact explains most of the behaviour that surprises people. The model is equally fluent whether or not it has the information, because fluency and accuracy are produced by the same mechanism. There is no internal signal that says "I do not know this" — the confident tone is not evidence of anything.
Why transformers changed the ceiling
The architecture underneath is a transformer, and its contribution is context. Through a mechanism called self-attention, the model weighs how every part of the input relates to every other part, so the end of a paragraph can inform how the beginning is read.
This is why these systems tolerate input that traditional software cannot. A conventional program accepts a fixed syntax; a language model will take a half-finished sentence from a maintenance technician, in the technician's own phrasing, and do something reasonable with it. For an operations interface, that tolerance is the whole value.
Fluency and accuracy come out of the same mechanism, which is why one is such a poor guide to the other.
The three limits worth designing around
The knowledge is frozen and generic. Training ended on a date, and it never included your equipment, your part numbers, or your procedures. A model asked about your plant will answer about plants in general, in the same confident register.
Wrong answers are well-formed. When the model has no basis for an answer it does not fail loudly. It produces something plausible. In a maintenance or compliance context, a plausible wrong answer is more expensive than an obvious one, because it survives review.
Input is not private by default. Whatever an operator pastes in — a drawing, a customer specification, an incident report — goes to whoever runs the model, under whatever terms apply. This is a procurement and data-governance question before it is a technical one, and it should be settled before a pilot, not after.
What this implies for deployment
None of the above argues against using these models. It argues for putting them where a wrong answer is cheap and visible: drafting, summarising, searching, translating between one team's vocabulary and another's. Keep them advisory, keep a person on the decision, and make the interface show what the answer was based on.
Where the answer must be right, the model needs grounding in sources you control, and the output needs to cite them. That is a different piece of engineering, and it is the subject of a separate note.
Written by the software delivery team. Published articles carry a named author once attribution is confirmed.
