Pre-Execution Hooks

Pre-execution hooks run after workspace setup but before the AI agent executes. Use them to prepare the environment for the agent.

Use Cases

Environment variable injection
Search and replace in source code
Code generation
File setup and templating
Mock data creation
Dependency installation

Script Hook

The script hook runs shell commands:

pre_execution:
  - name: script
    config:
      command: bash
      args:
        - "-c"
        - |
          echo "Setting up environment..."
          mkdir -p ${WORKSPACE_DIR}/config

Configuration

pre_execution:
  - name: script
    config:
      command: bash              # Command to run
      args:                      # Arguments array
        - "-c"
        - "echo 'Hello'"
      timeout_ms: 30000          # Timeout in milliseconds
      env:                       # Additional environment variables
        MY_VAR: "value"

command (required)

The shell or executable to run:

command: bash
# or
command: node
# or
command: python

args

Arguments passed to the command:

args:
  - "-c"
  - |
    echo "Multi-line script"
    mkdir -p ./temp

timeout_ms

Maximum execution time in milliseconds:

timeout_ms: 60000  # 1 minute

Default: 30000 (30 seconds)

env

Additional environment variables:

env:
  API_KEY: "${SECRET_API_KEY}"
  DEBUG: "true"

Environment Variables

Pre-execution hooks have access to:

Variable	Description
`WORKSPACE_DIR`	Full path to workspace
`REPO_DIR`	Path to cloned repository
`ARTIFACTS_DIR`	Path for output artifacts
`TEST_CASE_NAME`	Name from suite config
`REPO_URL`	Repository URL
`BRANCH`	Branch being tested

Examples

Create Configuration File

pre_execution:
  - name: script
    config:
      command: bash
      args:
        - "-c"
        - |
          mkdir -p ${WORKSPACE_DIR}/config
          cat > ${WORKSPACE_DIR}/config/auth.json << EOF
          {
            "jwtSecret": "test-secret-key",
            "tokenExpiry": "24h"
          }
          EOF

Install Dependencies

pre_execution:
  - name: script
    config:
      command: bash
      args:
        - "-c"
        - |
          cd ${REPO_DIR}
          npm install
      timeout_ms: 120000  # 2 minutes for npm install

Generate Mock Data

pre_execution:
  - name: script
    config:
      command: node
      args:
        - "-e"
        - |
          const fs = require('fs');
          const data = Array.from({length: 100}, (_, i) => ({
            id: i + 1,
            name: `User ${i + 1}`
          }));
          fs.writeFileSync(
            process.env.REPO_DIR + '/fixtures/users.json',
            JSON.stringify(data, null, 2)
          );

Search and Replace

pre_execution:
  - name: script
    config:
      command: bash
      args:
        - "-c"
        - |
          cd ${REPO_DIR}
          # Replace placeholder with actual value
          find . -type f -name "*.ts" -exec \
            sed -i 's/API_PLACEHOLDER/${API_URL}/g' {} \;
      env:
        API_URL: "https://api.example.com"

Multiple Pre-Execution Hooks

Hooks run sequentially in order:

pre_execution:
  # First: Install dependencies
  - name: script
    config:
      command: bash
      args: ["-c", "cd ${REPO_DIR} && npm install"]
      timeout_ms: 120000

  # Second: Generate config
  - name: script
    config:
      command: bash
      args: ["-c", "echo '{\"env\": \"test\"}' > ${REPO_DIR}/config.json"]

  # Third: Seed database
  - name: script
    config:
      command: bash
      args: ["-c", "cd ${REPO_DIR} && npm run db:seed"]

Error Handling

Failure Behavior

If a pre-execution hook fails:

Execution stops immediately
The agent is NOT executed
Evaluation is marked as failed
Error details are logged

Best Practices

pre_execution:
  - name: script
    config:
      command: bash
      args:
        - "-c"
        - |
          set -e  # Exit on first error

          # Validate prerequisites
          if [ ! -f "${REPO_DIR}/package.json" ]; then
            echo "Error: package.json not found"
            exit 1
          fi

          # Proceed with setup
          cd ${REPO_DIR}
          npm install

Security Considerations

Use environment variables for secrets, never hardcode
Limit permissions in scripts
Validate inputs before using in commands
Set appropriate timeouts to prevent hanging