All insights

Evaluating an AI assistant before trusting it

A method for measuring the quality, cost and reliability of a model’s answers before choosing it.

A plausible answer is not a correct one

A personal project needed an AI assistant to help players complete video-game trophies: building a route, listing every required item and answering follow-up questions about a save. A fluent answer can still lead a player into an irreversible mistake. Before choosing a model, I needed to measure answers rather than judge them by impression.

Separate what can be checked from what must be judged

Structure, required fields, duplicates and expected item sets are verified by exact programmatic checks. Correctness, source support and usefulness go to a separate evaluator using a weighted rubric. A dimension that was not graded stays ungraded: it never receives full credit by default.

Some errors outweigh any average

An instruction that locks a player out of a trophy, invented progress or a fabricated source disqualifies a configuration, whatever its average score. Quality, cost and latency are reported separately, so a cheap, fast mistake cannot offset weak advice. An unknown price remains unknown, never zero.

Make every run reproducible and honest

Each run records the model, prompts, dataset, prices and code revision, and resuming refuses changed inputs. The harness is covered by 249 offline tests. The current 18-case suite is a development set: results remain preliminary until the evaluator is calibrated against human reviews, and no model has been selected yet.