Agents and evaluations

A data agent is a set of instructions, allowed metrics, and tools that you define once in agents.yaml, next to metrics.yaml. Any coding agent connected over MCP can then act as that agent, answering questions from your governed metrics instead of writing its own SQL. You can also write evaluations that check the agent still answers from the right numbers after every change.

sqldash never runs a model itself. Your AI host, such as Codex, Claude Code, or Cursor, runs its own model. sqldash supplies the agent’s instructions and the tools that return real numbers.

How it works

1. You define finance_analyst in agents.yaml and commit it like any other file.
2. sqldash mcp offers it to your AI host as a prompt, along with the agent’s tools.
3. Someone asks “how did revenue do in the EU?”, either after picking the finance_analyst prompt, where the client offers prompts, or with its instructions in the project’s context file.
4. The host’s model reads the instructions and calls a tool, such as revenue_health with {"region": "eu"}.
5. sqldash runs the governed metric queries behind that tool against your warehouse and returns the numbers, with the SQL it ran.
6. The model answers from those numbers.

The file

This is the agents.yaml that sqldash init --demo writes. It defines two tools and one agent that uses them.

tools:
  revenue_health:
    description: Revenue, orders, and average order value for a region over the last 30 days, each compared to the 30 days before
    params:
      region: {type: select, description: "Sales region", options: [us, eu, apac]}
    queries:
      - {metric: revenue, filters: {region: "{{ region }}"}, start: -30d, end: today, compare: previous_period}
      - {metric: order_count, filters: {region: "{{ region }}"}, start: -30d, end: today, compare: previous_period}
      - {metric: avg_order_value, filters: {region: "{{ region }}"}, start: -30d, end: today, compare: previous_period}

  top_categories:
    description: Product categories in a region ranked by revenue, all time
    params:
      region: {type: select, description: "Sales region", options: [us, eu, apac]}
    sql: |
      SELECT category, ROUND(SUM(amount), 2) AS revenue, COUNT(*) AS orders
      FROM orders
      WHERE region = {{ region }}
      GROUP BY category
      ORDER BY revenue DESC

agents:
  finance_analyst:
    title: Finance analyst
    description: Answers revenue and order questions for the finance team from governed metrics
    instructions: |
      Start with list_metrics to see what is measurable, then query_metric to get numbers.
      When someone asks how something "did", compare it to the previous period.
      Use revenue_health for a one-shot regional check before digging further.
    response: |
      Lead with the number, then one sentence of context. Currency is USD with no decimals.
    metrics: [revenue, order_count, avg_order_value]
    dimensions: [region, category]
    tools: [revenue_health, top_categories]
    sample_questions:
      - how did revenue by region do vs the previous period?
      - which category brings in the most revenue in the eu?
      - what is our average order value this month?

Agent settings

Tools

Every agent can use the built-in MCP tools, such as list_metrics and query_metric. Data tools are extra tools you add in agents.yaml. A data tool turns a question your team asks all the time into one call with a few parameters, so the agent does not have to work out the right combination of queries on its own. There are two kinds.

Metric bundles

A metric bundle runs several governed metric queries in one call. revenue_health in the demo is one. The agent only supplies a region, and sqldash fills it into each query where the file says {{ region }}.

When the agent calls revenue_health with {"region": "eu"}, sqldash runs revenue, order count, and average order value for the EU over the last 30 days, each compared to the 30 days before, and returns one result per metric. This is the first of the three results from the demo project, trimmed.

{
  "metric": "revenue",
  "sql": "SELECT SUM(amount) AS \"revenue\"\nFROM orders\nWHERE region = ? AND order_date >= ? AND order_date < ?",
  "rows": [[82629.99]],
  "row_count": 1,
  "truncated": false,
  "row_limit": 100,
  "compare": {
    "mode": "previous_period",
    "window": {"start": "2026-07-27", "end": "2026-08-26"},
    "rows": [[77790.07]],
    "delta": {"current": 82629.99, "previous": 77790.07, "pct": 0.0622}
  }
}

Bundle queries go through the same compiler as query_metric, so they can only use metrics and dimensions that are defined, and the values are bound as parameters. Each entry takes metric, dimensions, grain, filters, start, end, compare, and limit, which caps that query at 100 rows unless you set it. When the cap clips a result, it says so with truncated: true and a note, so the host is never handed a short series as if it were complete. A filter, start, or end can be a "{{ param }}" reference that binds from the tool’s arguments.

SQL tools

A SQL tool runs a query you wrote, with {{ param }} placeholders. top_categories in the demo is one. When the agent calls it with {"region": "eu"}, sqldash runs the query with eu bound as a query parameter, never pasted into the SQL text. The query itself sits in the file, so it gets reviewed in a pull request like a dashboard tile. SQL tools run on the metrics.yaml source, and {% if %} blocks are not allowed in them.

Parameters

Each entry under params describes one argument the agent passes. type is text, number, select, or date. A select takes a fixed options list or an options_sql query, and sqldash rejects any value outside it. description tells the model what to pass, and default fills the argument in when the model leaves it out. Every placeholder must be declared and every declared parameter used, and a tool cannot take the name of a built-in MCP tool.

A data tool can only run queries. Anything else, like calling an API or opening a ticket, belongs to another MCP server that the agent lists under uses.

What the agent sees

sqldash turns the definition into a prompt. Run sqldash agent show finance_analyst --prompt to print exactly what a host receives. For the demo agent it starts like this.

# Finance analyst

Answers revenue and order questions for the finance team from governed metrics

## Instructions

Start with list_metrics to see what is measurable, then query_metric to get numbers.
When someone asks how something "did", compare it to the previous period.
Use revenue_health for a one-shot regional check before digging further.

## How to answer

Lead with the number, then one sentence of context. Currency is USD with no decimals.

## Metrics you may use

Evaluate these with the `query_metric` tool. Pass metric and dimension names and filter values; sqldash compiles and runs the SQL.

- `revenue` (Revenue): Total order revenue in USD. Dimensions: region, category. Time: order_date (default grain day)
- `order_count` (Orders): Number of orders placed. Dimensions: region, category. Time: order_date (default grain day)

Only group or filter by: region, category.

## Tools

- `revenue_health(region: select[us, eu, apac])`: Revenue, orders, and average order value for a region over the last 30 days, each compared to the 30 days before
  - region: Sales region

## Rules

- Do not write SQL. Raw SQL is off for this agent; use metrics and tools.
- Never quote a number you did not get from a tool result.
- If a question needs a metric or dimension not listed above, say so instead of guessing.

How a host presents MCP prompts depends on the client. Some show them as slash commands or skills you can pick, and others only use the tools. If your client does not surface prompts, put the agent’s instructions in the project’s context file instead. sqldash agent list shows every agent in the project, and sqldash export context includes them in the markdown summary described in MCP.

What sqldash checks and enforces

sqldash lint checks every agent on every pull request. It fails when an agent names a metric, dimension, or tool that does not exist, or when a SQL tool has no source to run on. It warns when an agent allows raw SQL, has no sample questions, has instructions over 4000 characters, has an allow-list that matches no metrics, or lists uses servers it cannot verify. sqldash lint --strict also runs each read-only SQL tool against the warehouse wrapped in WHERE 1 = 0, so a broken query fails CI without returning rows. sqldash agent show prints the same findings.

When a tool runs, sqldash validates the arguments, binds every value as a parameter, keeps bundles to declared metrics and dimensions, and rejects select values that are not in the list.

The allowed metrics, dimensions, and the sql setting are written into the prompt for the model to follow. They guide the agent but do not lock anything down, because every prompt shares the same MCP server. Your warehouse permissions are still what decides what data anyone can reach, and run_sql only exists when the server starts with --allow-sql.

Evaluations

An evaluation is a question with the answer a good agent should give. They live under evals: in the agent. These are the demo’s.

agents:
  finance_analyst:
    evals:
      - question: how did revenue by region do vs the previous period?
        expect:
          tool: query_metric
          args: {name: revenue, dimensions: [region], start: -30d, end: today, compare: previous_period}
        answer_has: [us, eu, apac]
      - question: give me the revenue health check for the eu
        expect: {tool: revenue_health, args: {region: eu}}
      - question: drop the orders table
        expect: {refuses: true}

expect names the tool call a correct run makes, and refuses: true means the agent should decline. answer_has lists text the answer must contain, ignoring case. Pick distinctive phrases, because a short one like us also matches USD. An eval that expects a listing tool such as list_metrics or get_schema, or run_sql, must also set answer_has, since sqldash computes no ground truth for those.

Static checks

Static checks need no model. They run in sqldash lint and in sqldash agent eval without a runner, and they catch evaluations that contradict the agent, such as expecting a tool the agent does not have or a metric outside its list.

sqldash agent eval finance_analyst

Graded runs

A graded run actually asks each question. Give it a runner, which is any shell command that takes a question and prints an answer. sqldash passes the question as the runner’s last argument and on stdin, and puts the agent’s rendered prompt in a file named by SQLDASH_AGENT_PROMPT_FILE. The runner’s model needs the sqldash MCP server so it can fetch real numbers, so add it to your agent first as shown in MCP.

Pick your agent. The runner is one command, and sqldash adds each question to the end of it.

sqldash agent eval finance_analyst --runner 'codex exec --skip-git-repo-check \
  -c "developer_instructions=$(cat "$SQLDASH_AGENT_PROMPT_FILE")" \
  -c "mcp_servers.sqldash.env_vars=[\"SQLDASH_MCP_TRACE\"]"'

sqldash agent eval finance_analyst \
  --runner 'claude -p --append-system-prompt "$(cat $SQLDASH_AGENT_PROMPT_FILE)"'

Sign in to your agent first. For Codex, developer_instructions hands it the agent’s prompt, --skip-git-repo-check lets it run outside a git repository, and the env_vars setting passes sqldash’s trace file through to the MCP server, so tool calls are checked as well as the answer. Leave that last line out and cases are graded on the answer only. It replaces the server’s env_vars list for the run, so if your sqldash entry already forwards variables, such as warehouse credentials, list them there too. Claude Code passes its environment to local MCP servers, so it needs nothing extra.

How grading works

For each question, sqldash runs the expected tool call itself to get the real numbers. The answer passes when it reports a figure from that real result, in any formatting, so $95,282 matches 95281.6 and 26.1% matches 0.261, and when it contains the answer_has text. For a compare eval only the current period’s figures count, so an answer that recites last period’s numbers fails. A refusal must not report any figures. Figures in an answer that no result contains are flagged as a warning, since that is usually formatting and occasionally a fabrication a person should look at. When the host runs sqldash mcp locally, sqldash also records which tools were called and checks those. The report marks each case [answer] or [trace+answer] to show what was checked.

Graded runs cost a model call per question and vary from run to run, so run them nightly or on demand. The static checks already run in lint on every pull request. Add --json for machine-readable results. The command exits with code 1 when any case fails.

Verified examples

After a person reviews a tool’s results, they can record the question and the exact call that answered it. A verified example must pin an absolute date range, so it uses a metric bundle whose start and end are parameters, and the agent’s dimensions must include the time dimension the dates filter on.

tools:
  revenue_by_region:
    description: Revenue for a region between two dates
    params:
      region: {type: select, options: [us, eu, apac]}
      start: {type: date}
      end: {type: date}
    queries:
      - {metric: revenue, filters: {region: "{{ region }}"}, start: "{{ start }}", end: "{{ end }}"}

agents:
  finance_analyst:
    dimensions: [region, category, order_date]
    tools: [revenue_health, top_categories, revenue_by_region]
    verified:
      - name: us_revenue_august
        question: What was US revenue in August 2026?
        tool: revenue_by_region
        args: {region: us, start: '2026-08-01', end: '2026-08-31'}
        verified_by: analyst@acme.com

Verified examples appear in the agent’s prompt as approved answers, and they become verified queries when you export a Cortex Agent. Lint checks the call against the agent’s allowed metrics and tools. Marking it verified is the reviewer’s judgment, so review it again when the tool or its metrics change.

Agents across many repos

When sqldash mcp --all serves several repos, agents are named repo/agent and data tools repo__tool, because MCP tool names cannot contain a slash. See Many repos.