Same Prompt,
Different Answer
Why LLM determinism is a legal problem, what Azure OpenAI's seed and system_fingerprint actually do about it, why AWS Bedrock has nothing comparable for text models - and an honest opinion on what it means to build on something that's fundamentally guessing the next word.
↓ scroll to read · click a section on the left to jump
Monday: "low risk." Friday: "flag for review."
A law firm feeds the same indemnification clause into the same model, through the same prompt, twice in one week. Monday it comes back "standard, low risk." Friday - same clause, same prompt, same everything the associate can see - it comes back "unusual scope, recommend review." Nobody changed the contract. Nobody changed the prompt.
This is not a hypothetical edge case. It's the default behavior of a chat completion API unless you go out of your way to fight it. And "go out of your way" turns out to mean something very specific, with real limits, that most teams shipping AI into regulated workflows haven't looked at closely. That's this article.
A precise vocabulary for the problem (determinism vs. reproducibility vs. consistency - they're not the same thing), what Azure OpenAI's seed parameter actually guarantees (less than you'd hope), what AWS Bedrock offers instead (less than that), and a concrete audit-logging playbook you can use regardless of which vendor you're on.
Non-determinism isn't a bug report
Three separate things conspire to make the same input produce different output, stacked on top of each other:
- Sampling. Above temperature zero, the model draws from a probability distribution over the next token instead of always taking the top one. That's the whole point of temperature - it's not a bug to route around, it's the feature that makes output feel less robotic.
- Floating-point non-associativity. On GPUs, matrix multiplication across many parallel cores can sum floating-point numbers in a different order between runs, batch sizes, or hardware - and floating-point addition isn't associative, so (a+b)+c can differ by a few bits from a+(b+c). Tiny numerical drift, amplified token by token.
- Silent backend changes. The provider updates infrastructure, quantization, or routing behind a stable-looking model name, and your prompt now runs on a subtly different system than it did last week.
$ ask("summarize clause 4.2") # → run 1
"Standard indemnification, market terms."
$ ask("summarize clause 4.2") # → run 2, same day
"Fairly broad indemnification scope,
worth a second look."
# nothing you did was wrong. this is default behavior.
Even setting temperature to 0 doesn't fully fix this - it removes the sampling variance but not the floating-point and infrastructure variance. There is no dial that takes you to "always identical," on any major provider, today.
"It said something different last time" is not a sentence a court wants to hear
For a lot of use cases, output drift is a shrug - a chatbot phrasing a greeting two different ways doesn't matter. It stops being a shrug the moment the output feeds a decision someone has to defend later: a contract risk flag, a compliance classification, a discovery-relevance call, a drafting suggestion that ends up in a filed document.
The problem isn't that AI-assisted legal work is inherently reckless. It's that legal and compliance workflows run on an assumption most software quietly promises and LLMs quietly don't: same input, same output, or at least an explanation for why not. A calculator that gave you a different sum for 2+2 depending on server load would be malpractice-adjacent in any regulated context. An LLM that does the rough equivalent, today, by default, is treated as normal - because until recently there was no alternative.
This isn't about whether AI belongs in legal workflows - plenty of firms already use it well, with a human signing off on every output. It's that "a human reviews everything" and "the system is auditable" are different safeguards, and only one of them explains, after the fact, why the same clause got two different risk labels in the same week.
Determinism, reproducibility, consistency - pick the one you actually need
Bit-identical, always
Same input → the exact same output, every time, forever. This is the standard a calculator meets. No major LLM API meets it today, and - see the Azure section next - no vendor claims to.
Can detect + explain drift
Output may vary, but you can tell when it changed and roughly why - a version pin, a fingerprint, a logged config. This is the realistic target for production systems today.
Semantically stable
The wording changes, the substance doesn't - "standard indemnification" phrased two ways is fine; "standard" flipping to "unusual" is not. Often what people actually mean when they say "I need it to be deterministic."
Chasing bit-identical determinism (not currently achievable, and arguably not even what you want - you'd be freezing in every quirk of today's model forever) when what your compliance team actually needs is reproducibility: a paper trail that shows what ran, when, and whether it changed.
A best-effort dial, and a way to check if it moved
Azure OpenAI (mirroring OpenAI's own Chat Completions API) exposes two things that, used together, get you from "pure chaos" to "reproducibility, with a receipt":
- seed - an integer you pass in. Per Microsoft's own docs: "our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result." Note the hedging - "best effort," not "guaranteed."
- system_fingerprint - a string that comes back on every response, identifying the backend configuration that produced it. It changes when the provider updates infrastructure. You don't prevent drift with it - you detect it.
response = client.chat.completions.create(
model="gpt-4o",
seed=42, # pin the sampling
temperature=0,
messages=[{"role": "user",
"content": "Flag risk in clause 4.2: ..."}]
)
# log this next to the output - every time:
audit_log.write(seed=42,
fingerprint=response.system_fingerprint,
output=response.choices[0].message.content)
The seed is the part everyone notices. The fingerprint is the part that actually makes this defensible: if a clause gets re-analyzed six months later and the answer changed, you can pull up two system_fingerprint values, see they differ, and say precisely why - "the backend changed on this date" - instead of shrugging.
Read the fine print before you rely on it
To Microsoft's credit, their own documentation doesn't oversell this. Direct from the reproducible-output guide:
the seed parameter and system_fingerprint are the same across API calls
it's currently not uncommon to still observe a degree of variability
in responses. Identical API calls with larger max_tokens values will
generally result in less deterministic responses even when the seed
parameter is set."
- Same seed + same fingerprint can still drift - you've narrowed the variance, not eliminated it.
- Longer generations drift more - a one-word classification is far more stable than a five-paragraph memo.
- Every sampling parameter has to match, not just seed - temperature, max_tokens, top_p, all of it, every call.
Shipping a compliance workflow that quietly assumes "I set the seed, so it's locked in now" - and finding out the hard way, in front of a client or a regulator, that "best effort" meant exactly that.
Turn the temperature down and good luck
Here's what AWS Bedrock's Converse API - the interface for Claude, Llama, Titan, and every other text foundation model on Bedrock - offers for reproducibility: nothing. No seed parameter for text generation exists in the current documentation. AWS's own inference-parameters guidance for reducing randomness amounts to "lower temperature and top_p" - the exact same lever every API has had since before seed existed anywhere.
There is a seed parameter in the Bedrock docs - for the Stability Diffusion image models. It reproduces a picture of a cat. It has nothing to do with reproducing a paragraph about contract risk.
Imagine hiring an analyst who, when asked "will you give the same answer if I ask you this again," says "I'll try to think about it the same way, but I can't promise, and also I won't tell you if my brain gets a firmware update overnight." That's roughly the deal Bedrock's text APIs offer today. If you're building something a regulator might ask to see the receipts for, "trust me" isn't a receipt.
This isn't uniquely AWS's failing
It would be cheap to make this an AWS pile-on and stop there, so it's worth being precise about who else is in the same boat: as of this writing, Anthropic's own first-party Claude API also has no publicly documented seed parameter. Temperature is the lever Anthropic's own guidance points you to, same as Bedrock. Even at temperature 0 (greedy decoding), Anthropic's own developer community reports the output isn't guaranteed bit-identical across calls.
So the honest framing isn't "Azure solved this and everyone else is behind." It's: one vendor (Azure/OpenAI) shipped a best-effort knob and a drift-detection field; most of the rest of the industry, including model providers' own first-party APIs, haven't shipped either yet. That's an immature-tooling problem across the field, not a single company's oversight - which is exactly why it's worth writing about now, while the norms are still being set.
Picking a cloud vendor by this one axis alone, or assuming a competitor's stack quietly has this solved. Ask every vendor the same question - "what do you give me to detect and explain output drift?" - and compare real answers, not marketing pages.
You can't buy determinism. You can buy auditability.
Full request + response + fingerprint
Store the exact prompt, all sampling params, the seed if set, the system_fingerprint if available, and the raw output - every call that feeds a decision. This is the actual audit trail, not "we used AI."
Never deploy against "latest"
Pin an exact model snapshot or version ARN instead of a rolling alias. "The model changed under me" should be a deliberate upgrade you chose, logged with a date - not a surprise you discover in a review.
Run N, flag disagreement
For anything high-stakes, run the same input 3–5 times and compare. If the outputs disagree in substance (not just phrasing), that's a signal to route to a human - treat high variance itself as a risk flag, not noise to average away.
Treat "the model gave a different answer" as an expected production event, not a bug ticket - the same way you'd treat a flaky network call. Build the logging and the variance checks assuming drift will happen, because on every vendor covered here, it will.
"It's just guessing the next word" - yes, and?
Mechanically, an LLM is sampling from a probability distribution over the next token. Calling that "guessing" isn't an insult or a gotcha - it's a correct, literal description of the algorithm. People sometimes say it like it's a debunking. It isn't. What's actually remarkable is that a system built entirely on "predict the statistically likely next piece of text" produces answers that are right often enough to be useful for real work, across an enormous range of tasks nobody explicitly programmed it for.
Here's where I land, for what it's worth: "good most of the time" and "trustworthy on an irreversible, high-stakes call" are different properties, and conflating them is where most of the overconfidence in this space comes from. A model that's right 97% of the time is an extraordinary tool for drafting, summarizing, and first-pass triage - and a genuinely bad sole decision-maker for the 3% of calls where being wrong has real consequences, precisely because you can't tell which bucket any single answer falls into just by looking at it. Confidence and correctness aren't the same signal, and current models don't reliably tell you which one you're getting.
The people who say "it's just autocomplete, ignore it" and the people who say "it reasons like a person, trust it" are both, in my view, being lazy in the same way - both are refusing to sit with the actually uncomfortable fact, which is that this is a genuinely new category of tool: probabilistically excellent, individually unverifiable, in ways that don't map cleanly onto how we've evaluated software or people before. Determinism tooling like seed and system_fingerprint doesn't fix that underlying fact. It just makes the fact auditable - which, for anything you'd have to explain to a judge, a regulator, or a client, might be the whole ballgame.
None of this is a reason to avoid LLMs in serious work - plenty of what's in this article assumes you're already shipping them. It's a reason to be precise about what you're actually trusting: not "the model," but "the model, plus the logging, plus the human check, plus the honesty about where the 3% lives." Skip any one of those and you haven't removed the risk, you've just stopped being able to see it.
Further reading
Everything factual above about specific vendor behavior traces back to these - worth reading directly rather than taking my word for it, especially before making an architecture decision on top of any of it.
- Microsoft Learn - How to generate reproducible output with Azure OpenAISource for the seed / system_fingerprint mechanics and the "determinism isn't guaranteed" caveat quoted above.
- OpenAI Cookbook - Reproducible outputs with the seed parameterThe underlying mechanism Azure OpenAI mirrors from OpenAI's own Chat Completions API.
- AWS Bedrock - Inference request parameters and response fieldsConfirms no seed parameter for Bedrock's text foundation models; temperature/top_p are the documented levers.
- AWS Bedrock - Stability.ai Diffusion model parametersWhere Bedrock's only documented "seed" parameter actually lives - image generation, not text.
Field notes: Anupam Kumar · Backend & Generative AI Engineer. Vendor docs cited as of July 2026 - APIs move fast, re-check before you rely on any specific claim above. v1.0