The Prompt That Worked in Isolation and Died in Production: Eval-Driven Prompt Engineering for a Real-Time Interview Copilot

InterviewMate (interviewmate.tech) is a real-time interview copilot I built and run in production: it listens to a live interview, transcribes it as it happens, and streams back answer suggestions grounded in the candidate’s own experience — in under a second. The pipeline is Deepgram streaming STT into a FastAPI WebSocket Q&A pipeline, RAG retrieval over my own past answers, and a Claude streaming response. I’ll run the working system live.
Then I’ll show the part that actually mattered: the eval harness behind the system prompt. It started as a bug — my copilot answered the viral “car wash” question correctly (“drive”) while the same model, called with no prompt around it, said “walk,” and I couldn’t explain which prompt layer was responsible. So I turned the production prompt into a controlled experiment: 6 prompt conditions, 20 runs each, showing STAR-style structured reasoning beating context injection by 2.83x — and a follow-up where that same winning prompt collapsed to 0–30% once I dropped it into the real 60-line production prompt. Code, eval logs, and the pipeline architecture will all be on screen.