Hooks

Hooks allow you to run custom logic at specific points in the evaluation pipeline.

Hook Types

Pre-Execution Hooks

Run before the AI agent executes. Use for environment setup, file preparation, and secret injection. Learn more →

Post-Evaluation Hooks

Run after evaluation completes. Use for result export, notifications, and custom analysis. Learn more →

Pipeline Flow

┌─────────────────────┐
│  Workspace Setup    │
└──────────┬──────────┘
           ▼
┌─────────────────────┐
│ Pre-Execution Hooks │◄── Setup, inject secrets, modify files
└──────────┬──────────┘
           ▼
┌─────────────────────┐
│   Agent Execution   │
└──────────┬──────────┘
           ▼
┌─────────────────────┐
│     Evaluators      │
└──────────┬──────────┘
           ▼
┌─────────────────────┐
│Post-Evaluation Hooks│◄── Export, notify, analyze
└──────────┬──────────┘
           ▼
┌─────────────────────┐
│      Results        │
└─────────────────────┘

Key Differences

Aspect	Pre-Execution	Post-Evaluation
When	Before agent runs	After evaluators complete
Purpose	Setup workspace	Export/notify
Execution	Sequential (one at a time)	Parallel (all at once)
Failure behavior	Stops evaluation	Never fails main evaluation
Access	Can modify workspace	Read-only access to results

Quick Example

pre_execution:
  - name: script
    config:
      command: bash
      args:
        - "-c"
        - |
          echo "Setting up environment..."
          mkdir -p ${WORKSPACE_DIR}/config
          echo '{"apiKey": "test"}' > ${WORKSPACE_DIR}/config/secrets.json

post_evaluation:
  - name: webhook
    config:
      url: ${SLACK_WEBHOOK_URL}
      method: POST

Available Hooks

Pre-Execution

Hook	Description
`script`	Run shell scripts for custom setup

Post-Evaluation

Hook	Description
`database`	Export results to JSON/JSONL files
`webhook`	POST results to HTTP endpoints
`script`	Run custom analysis scripts

Environment Variables

All hooks have access to these variables:

Variable	Description
`WORKSPACE_DIR`	Workspace directory path
`REPO_DIR`	Repository directory
`ARTIFACTS_DIR`	Artifacts directory
`TEST_CASE_NAME`	Test case name
`REPO_URL`	Repository URL
`BRANCH`	Branch being tested

Post-evaluation hooks also have:

Variable	Description
`RESULTS_PATH`	Path to results.json