Tests

Tests tell you whether your context works. Each test is a question with a known correct answer. pda test sends every question to Polaris, runs the reference SQL, and compares the data Polaris returned with the expected data. Run the tests after every change to RULES.md or the table docs.

Write a test

Create one YAML file per test under tests/:

tests/ot_approved_drugs_bladder_cancer.yml

name: ot_approved_drugs_bladder_cancer
database: epigenomics
prompt: Which drugs are approved for bladder cancer?
sql: |
    SELECT DISTINCT k.drug_name
    FROM tcga_project_disease_map m
    JOIN ot_known_drugs k ON k.disease_id = m.disease_id
    WHERE m.project_id = 'TCGA-BLCA'
      AND k.indication_max_stage = 'APPROVAL'
    ORDER BY k.drug_name

Key	Required	What it is
name	No	The test’s name. Defaults to the file name.
prompt	Yes	The question, written the way a real user would ask it.
sql	No	The reference SQL that returns the correct answer.
database	No	Which database connection the reference SQL runs on.
assertions	No	Checks on which tools the agent called. See below.

Polaris reads every .yml and .yaml file under tests/, including subfolders, except tests/outputs/. If one file cannot be read, the whole run fails.

Tips for good tests

• Write prompts like real questions. Do not name tables, columns or methods. “How’s KIRC survival by stage?” tests more than “Group tcga_patients by ajcc_pathologic_stage”.
• Name output columns by meaning and unit. Use names such as mutation_rate_percentage_0_100 or diagnosis_date_yyyy_mm_dd, so it is clear how to read each value.
• Cover each key metric at least once, then add tests for date ranges, missing values and ambiguous wording.

Assertions

An assertion checks that the agent called a tool, optionally with certain arguments:

assertions:
    - type: tool_call
      tool: load_skill
      args:
          name: survival-analysis
      min_count: 1

args matches when the call includes those arguments. min_count defaults to 1.

Run tests

pda test sends the questions to a running Polaris server, which answers them with the project it serves. Start a local server first, from the project folder:

pda chat &
pda test

To test a deployed Polaris, point BACKEND_URL at it:

BACKEND_URL=https://epigenomics.polaris.aws.grail.com pda test -u you@example.com

Note

The server answers with its own copy of the project context, not your local files. When you test a deployed Polaris, deploy your changes first.

Common options:

pda test -m bedrock:us.anthropic.claude-sonnet-5-5   # pick the model
pda test -t 4                                         # run 4 tests at a time
pda test -k 3                                         # run each test 3 times
pda test -s tcga_,ot_approved_drugs_bladder_cancer    # run only some tests

See the CLI reference for every flag. Defaults come from the test block of pda_config.yaml.

How a test passes

A test passes when the data Polaris returned matches the reference data:

• It has the same columns as the expected result.
• It has the same number of rows.
• Numbers match after rounding to decimals places (default 2), within the tolerances rtol and atol.
• Every assertion holds.

With -k above 1, each test runs several times. The report shows pass@k (passed at least once) and pass^k (passed every time). pass^k is the better measure of reliability.

Each run also records tokens, cost, time and the number of tool calls.

Read the results

Results are saved to tests/outputs/results_<timestamp>.json. To browse them in a web page, run:

pda test server

For each failure, decide whether the test is wrong or the context is. Then make the smallest context change that fixes it, and run the tests again. See Context engineering.

Save a chat as a test

People who can manage tests see Save as test in an answer’s More actions menu. It turns a good question and answer into a test.

Hide the answers

Keep tests/ in .pdaignore, so the agent cannot read the expected SQL.