The basics

1. Score

Every scorer returns a Score with three fields:

2. Scorers

A scorer compares a model's response with the expected answer. The two string scorers ignore case and leading or trailing whitespace.

The string scorers need no API key, so you can try them right away:

from evalstack.scorer import ContainsScorer, ExactMatchScorer

reply = "The capital of France is Paris."

print(ExactMatchScorer().score(reply, "Paris"))
# Score(value=0.0, passed=False, reason='No exact match')

print(ContainsScorer().score(reply, "paris"))
# Score(value=1.0, passed=True, reason="Response contains 'paris'")

3. Running evals

EvalRunner sends the prompt to a model, scores the reply, and returns an EvalResult. Its default scorer is ContainsScorer, because models usually answer in full sentences.

from evalstack.core import EvalRunner

runner = EvalRunner(model="claude-opus-5-5")
result = runner.run("What is the capital of France?", expected="Paris")

# Example values; the model's exact wording will vary:
result.passed               # True
result.score                # 1.0
result.response             # "The capital of France is Paris."
result.metadata["reason"]   # "Response contains 'Paris'"

metadata also records the model, provider, scorer and latency_s (seconds the model took).

To run many cases, pass a list to batch. Every case is checked for prompt and expected before any model call, so a typo doesn't cost you half a run:

results = runner.batch([
    {"prompt": "Capital of France?", "expected": "Paris"},
    {"prompt": "Capital of Japan?", "expected": "Tokyo"},
])
print(sum(r.passed for r in results), "of", len(results), "passed")

4. The LLM judge

String matching can't tell that "It's Paris, the French capital" means the same as "Paris". LLMJudgeScorer asks a model to grade the meaning from 0.0 to 1.0 and passes at pass_threshold (0.7 by default):

from evalstack.scorer import LLMJudgeScorer

judge = LLMJudgeScorer(model="claude-opus-5-5", pass_threshold=0.8)
runner = EvalRunner(model="claude-opus-5-5", scorer=judge)

The judge makes one extra model call per case, so it costs more than the string scorers. Use it when wording varies but meaning matters.

5. Models and providers

The core package has no required dependencies. Install the SDK for the provider you use:

pip install -e ".[anthropic]"   # Claude models (default)
pip install -e ".[openai]"      # OpenAI models, with provider="openai"

Anything that calls a model also accepts generate: any function that takes a prompt and returns text. Use it for another provider, a local model, or a fake in tests. This example runs with no API key:

from evalstack.core import EvalRunner

def fake_model(prompt):
    return "The capital of France is Paris."

runner = EvalRunner(model="fake", generate=fake_model)
result = runner.run("What is the capital of France?", expected="Paris")
print(result.passed, result.metadata["reason"])
# True Response contains 'Paris'

6. When something goes wrong

A failing eval and a broken eval should never look the same, so EvalStack raises an error instead of guessing a score:

To try scorers by hand, run the local playground from a clone of the repo: python playground/server.py.