# Agents and evaluations

A data agent is a set of instructions, allowed metrics, and tools that you define once
in `agents.yaml`, next to `metrics.yaml`. Any coding agent connected over
[MCP](/docs/mcp/) can then act as that agent, answering questions from your governed
metrics instead of writing its own SQL. You can also write evaluations that check the
agent still answers from the right numbers after every change.

sqldash never runs a model itself. Your AI host, such as Codex, Claude Code, or Cursor, runs
its own model. sqldash supplies the agent's instructions and the tools that return
real numbers.

## How it works

1. You define `finance_analyst` in `agents.yaml` and commit it like any other file.
2. `sqldash mcp` offers it to your AI host as a prompt, along with the agent's tools.
3. Someone asks "how did revenue do in the EU?", either after picking the
   `finance_analyst` prompt, where the client offers prompts, or with its instructions
   in the project's context file.
4. The host's model reads the instructions and calls a tool, such as `revenue_health`
   with `{"region": "eu"}`.
5. sqldash runs the governed metric queries behind that tool against your warehouse
   and returns the numbers, with the SQL it ran.
6. The model answers from those numbers.

## The file

This is the `agents.yaml` that `sqldash init --demo` writes. It defines two tools and one
agent that uses them.

```yaml title=".sqldash/agents.yaml"
tools:
  revenue_health:
    description: Revenue, orders, and average order value for a region over the last 30 days, each compared to the 30 days before
    params:
      region: {type: select, description: "Sales region", options: [us, eu, apac]}
    queries:
      - {metric: revenue, filters: {region: "{{ region }}"}, start: -30d, end: today, compare: previous_period}
      - {metric: order_count, filters: {region: "{{ region }}"}, start: -30d, end: today, compare: previous_period}
      - {metric: avg_order_value, filters: {region: "{{ region }}"}, start: -30d, end: today, compare: previous_period}

  top_categories:
    description: Product categories in a region ranked by revenue, all time
    params:
      region: {type: select, description: "Sales region", options: [us, eu, apac]}
    sql: |
      SELECT category, ROUND(SUM(amount), 2) AS revenue, COUNT(*) AS orders
      FROM orders
      WHERE region = {{ region }}
      GROUP BY category
      ORDER BY revenue DESC

agents:
  finance_analyst:
    title: Finance analyst
    description: Answers revenue and order questions for the finance team from governed metrics
    instructions: |
      Start with list_metrics to see what is measurable, then query_metric to get numbers.
      When someone asks how something "did", compare it to the previous period.
      Use revenue_health for a one-shot regional check before digging further.
    response: |
      Lead with the number, then one sentence of context. Currency is USD with no decimals.
    metrics: [revenue, order_count, avg_order_value]
    dimensions: [region, category]
    tools: [revenue_health, top_categories]
    sample_questions:
      - how did revenue by region do vs the previous period?
      - which category brings in the most revenue in the eu?
      - what is our average order value this month?
```

## Agent settings

| setting | what it does |
| --- | --- |
| `description` | Required. One line saying what the agent is for. Hosts show it in their prompt list. |
| `instructions` | Required. How the agent should work, copied into its prompt word for word. |
| `title` | A friendly name for the prompt. |
| `response` | How to answer, such as tone, format, and units. |
| `metrics` | The metrics this agent should use. Leave it out to allow every metric. |
| `dimensions` | The dimensions it may group or filter by. |
| `tools` | Which data tools from the top-level `tools:` block it can call. |
| `sql` | Set `true` if the agent may write raw SQL. The default is `false`. |
| `sample_questions` | Example questions, shown in the prompt and used as a starting point for evaluations. |
| `evals` | Evaluations that check its answers. See below. |
| `verified` | Question and tool call pairs a person has reviewed. See below. |
| `uses` | Other MCP servers the agent relies on, written as `{server, for}`. sqldash lists them in the prompt but does not connect them. |

## Tools

Every agent can use the built-in MCP tools, such as `list_metrics` and `query_metric`.
Data tools are extra tools you add in `agents.yaml`. A data tool turns a question your
team asks all the time into one call with a few parameters, so the agent does not have
to work out the right combination of queries on its own. There are two kinds.

### Metric bundles

A metric bundle runs several governed metric queries in one call. `revenue_health` in
the demo is one. The agent only supplies a region, and sqldash fills it into each query
where the file says `{{ region }}`.

When the agent calls `revenue_health` with `{"region": "eu"}`, sqldash runs revenue,
order count, and average order value for the EU over the last 30 days, each compared to
the 30 days before, and returns one result per metric. This is the first of the three
results from the demo project, trimmed.

```json
{
  "metric": "revenue",
  "sql": "SELECT SUM(amount) AS \"revenue\"\nFROM orders\nWHERE region = ? AND order_date >= ? AND order_date < ?",
  "rows": [[82629.99]],
  "row_count": 1,
  "truncated": false,
  "row_limit": 100,
  "compare": {
    "mode": "previous_period",
    "window": {"start": "2026-07-27", "end": "2026-08-26"},
    "rows": [[77790.07]],
    "delta": {"current": 82629.99, "previous": 77790.07, "pct": 0.0622}
  }
}
```

Bundle queries go through the same compiler as `query_metric`, so they can only use
metrics and dimensions that are defined, and the values are bound as parameters. Each
entry takes `metric`, `dimensions`, `grain`, `filters`, `start`, `end`, `compare`, and
`limit`, which caps that query at 100 rows unless you set it. When the cap clips a
result, it says so with `truncated: true` and a note, so the host is never handed a
short series as if it were complete. A filter, `start`, or `end` can be a
`"{{ param }}"` reference that binds from the tool's arguments.

### SQL tools

A SQL tool runs a query you wrote, with `{{ param }}` placeholders. `top_categories` in
the demo is one. When the agent calls it with `{"region": "eu"}`, sqldash runs the query
with `eu` bound as a query parameter, never pasted into the SQL text. The query itself
sits in the file, so it gets reviewed in a pull request like a dashboard tile. SQL
tools run on the `metrics.yaml` source, and `{% if %}` blocks are not allowed in them.

### Parameters

Each entry under `params` describes one argument the agent passes. `type` is `text`,
`number`, `select`, or `date`. A `select` takes a fixed `options` list or an
`options_sql` query, and sqldash rejects any value outside it. `description` tells the
model what to pass, and `default` fills the argument in when the model leaves it out.
Every placeholder must be declared and every declared parameter used, and a tool cannot
take the name of a built-in MCP tool.

A data tool can only run queries. Anything else, like calling an API or opening a
ticket, belongs to another MCP server that the agent lists under `uses`.

## What the agent sees

sqldash turns the definition into a prompt. Run `sqldash agent show finance_analyst --prompt`
to print exactly what a host receives. For the demo agent it starts like this.

```markdown
# Finance analyst

Answers revenue and order questions for the finance team from governed metrics

## Instructions

Start with list_metrics to see what is measurable, then query_metric to get numbers.
When someone asks how something "did", compare it to the previous period.
Use revenue_health for a one-shot regional check before digging further.

## How to answer

Lead with the number, then one sentence of context. Currency is USD with no decimals.

## Metrics you may use

Evaluate these with the `query_metric` tool. Pass metric and dimension names and filter values; sqldash compiles and runs the SQL.

- `revenue` (Revenue): Total order revenue in USD. Dimensions: region, category. Time: order_date (default grain day)
- `order_count` (Orders): Number of orders placed. Dimensions: region, category. Time: order_date (default grain day)

Only group or filter by: region, category.

## Tools

- `revenue_health(region: select[us, eu, apac])`: Revenue, orders, and average order value for a region over the last 30 days, each compared to the 30 days before
  - region: Sales region

## Rules

- Do not write SQL. Raw SQL is off for this agent; use metrics and tools.
- Never quote a number you did not get from a tool result.
- If a question needs a metric or dimension not listed above, say so instead of guessing.
```

How a host presents MCP prompts depends on the client. Some show them as slash commands
or skills you can pick, and others only use the tools. If your client does not surface
prompts, put the agent's instructions in the project's context file instead.
`sqldash agent list` shows every agent in the project, and `sqldash export context`
includes them in the markdown summary described in [MCP](/docs/mcp/).

## What sqldash checks and enforces

`sqldash lint` checks every agent on every pull request. It fails when an agent names a
metric, dimension, or tool that does not exist, or when a SQL tool has no source to run
on. It warns when an agent allows raw SQL, has no sample questions, has instructions
over 4000 characters, has an allow-list that matches no metrics, or lists `uses` servers
it cannot verify. `sqldash lint --strict` also runs each read-only SQL tool against the
warehouse wrapped in `WHERE 1 = 0`, so a broken query fails CI without returning rows.
`sqldash agent show` prints the same findings.

When a tool runs, sqldash validates the arguments, binds every value as a parameter,
keeps bundles to declared metrics and dimensions, and rejects `select` values that are
not in the list.

The allowed metrics, dimensions, and the `sql` setting are written into the prompt for
the model to follow. They guide the agent but do not lock anything down, because every
prompt shares the same MCP server. Your warehouse permissions are still what decides
what data anyone can reach, and `run_sql` only exists when the server starts with
`--allow-sql`.

## Evaluations

An evaluation is a question with the answer a good agent should give. They live under
`evals:` in the agent. These are the demo's.

```yaml
agents:
  finance_analyst:
    evals:
      - question: how did revenue by region do vs the previous period?
        expect:
          tool: query_metric
          args: {name: revenue, dimensions: [region], start: -30d, end: today, compare: previous_period}
        answer_has: [us, eu, apac]
      - question: give me the revenue health check for the eu
        expect: {tool: revenue_health, args: {region: eu}}
      - question: drop the orders table
        expect: {refuses: true}
```

`expect` names the tool call a correct run makes, and `refuses: true` means the agent
should decline. `answer_has` lists text the answer must contain, ignoring case. Pick
distinctive phrases, because a short one like `us` also matches `USD`. An eval that
expects a listing tool such as `list_metrics` or `get_schema`, or `run_sql`, must also
set `answer_has`, since sqldash computes no ground truth for those.

### Static checks

Static checks need no model. They run in `sqldash lint` and in `sqldash agent eval`
without a runner, and they catch evaluations that contradict the agent, such as
expecting a tool the agent does not have or a metric outside its list.

```bash
sqldash agent eval finance_analyst
```

### Graded runs

A graded run actually asks each question. Give it a runner, which is any shell command
that takes a question and prints an answer. sqldash passes the question as the runner's
last argument and on stdin, and puts the agent's rendered prompt in a file named by
`SQLDASH_AGENT_PROMPT_FILE`. The runner's model needs the sqldash MCP server so it can
fetch real numbers, so add it to your agent first as shown in [MCP](/docs/mcp/).

Pick your agent. The runner is one command, and sqldash adds each question to the end
of it.

<div class="tabs">

```bash title="Codex"
sqldash agent eval finance_analyst --runner 'codex exec --skip-git-repo-check \
  -c "developer_instructions=$(cat "$SQLDASH_AGENT_PROMPT_FILE")" \
  -c "mcp_servers.sqldash.env_vars=[\"SQLDASH_MCP_TRACE\"]"'
```

```bash title="Claude Code"
sqldash agent eval finance_analyst \
  --runner 'claude -p --append-system-prompt "$(cat $SQLDASH_AGENT_PROMPT_FILE)"'
```

</div>

Sign in to your agent first. For Codex, `developer_instructions` hands it the agent's
prompt, `--skip-git-repo-check` lets it run outside a git repository, and the `env_vars`
setting passes sqldash's trace file through to the MCP server, so tool calls are checked
as well as the answer. Leave that last line out and cases are graded on the answer
only. It replaces the server's `env_vars` list for the run, so if your sqldash entry
already forwards variables, such as warehouse credentials, list them there too. Claude
Code passes its environment to local MCP servers, so it needs nothing extra.

#### How grading works

For each question, sqldash runs the expected tool call itself to get the real numbers.
The answer passes when it reports a figure from that real result, in any formatting, so
`$95,282` matches `95281.6` and `26.1%` matches `0.261`, and when it contains the
`answer_has` text. For a `compare` eval only the current period's figures count, so an
answer that recites last period's numbers fails. A refusal must not report any figures.
Figures in an answer that no result contains are flagged as a warning, since that is
usually formatting and occasionally a fabrication a person should look at. When the host runs `sqldash mcp` locally, sqldash also records
which tools were called and checks those. The report marks each case `[answer]` or
`[trace+answer]` to show what was checked.

Graded runs cost a model call per question and vary from run to run, so run them nightly
or on demand. The static checks already run in `lint` on every pull request. Add `--json`
for machine-readable results. The command exits with code 1 when any case fails.

## Verified examples

After a person reviews a tool's results, they can record the question and the exact call
that answered it. A verified example must pin an absolute date range, so it uses a
metric bundle whose `start` and `end` are parameters, and the agent's `dimensions` must
include the time dimension the dates filter on.

```yaml
tools:
  revenue_by_region:
    description: Revenue for a region between two dates
    params:
      region: {type: select, options: [us, eu, apac]}
      start: {type: date}
      end: {type: date}
    queries:
      - {metric: revenue, filters: {region: "{{ region }}"}, start: "{{ start }}", end: "{{ end }}"}

agents:
  finance_analyst:
    dimensions: [region, category, order_date]
    tools: [revenue_health, top_categories, revenue_by_region]
    verified:
      - name: us_revenue_august
        question: What was US revenue in August 2026?
        tool: revenue_by_region
        args: {region: us, start: '2026-08-01', end: '2026-08-31'}
        verified_by: analyst@acme.com
```

Verified examples appear in the agent's prompt as approved answers, and they become
verified queries when you [export a Cortex Agent](/docs/lookml-cortex/). Lint checks the
call against the agent's allowed metrics and tools. Marking it verified is the
reviewer's judgment, so review it again when the tool or its metrics change.

## Agents across many repos

When `sqldash mcp --all` serves several repos, agents are named `repo/agent` and data
tools `repo__tool`, because MCP tool names cannot contain a slash. See
[Many repos](/docs/workspaces/).