Configuration

youBencha uses YAML or JSON configuration files to define evaluation suites. This guide covers all configuration options.

Configuration Files

Suite YAML

The main configuration file that defines repositories, agents, and evaluators. Suite YAML Reference →

Prompt Files

External prompt templates for better organization and reusability. Prompt Files Guide →

Quick Start

Create a configuration file using yb init:

yb init

Or create suite.yaml manually:

suite.yaml

# Repository to evaluate
repo: https://github.com/example/repo.git
branch: main

# AI Agent Configuration
agent:
  type: copilot-cli
  config:
    prompt: "Add error handling to all API endpoints"

# Evaluation Criteria
evaluators:
  - name: git-diff
  - name: agentic-judge
    config:
      type: copilot-cli
      assertions:
        error_handling: "Proper error handling is implemented. Score 1 if yes, 0 if no."

Format Support

youBencha supports both YAML and JSON formats with automatic detection based on file extension.

Extension	Format
.yaml, .yml	YAML
.json	JSON

Required Fields

Every suite configuration must include:

Field	Description
repo	Repository URL (HTTP/HTTPS only)
agent.type	Agent adapter type (e.g., copilot-cli)
evaluators	Array of evaluator configurations

Optional Fields

Field	Description	Default
name	Test case name	Auto-generated
description	Test case description	None
branch	Git branch	Repository default
commit	Specific commit SHA	None
expected_source	Reference source type	None
expected	Reference branch/commit	None
timeout	Operation timeout (ms)	300000
workspace_dir	Custom workspace directory	.youbencha-workspace
pre_execution	Pre-execution hooks	None
post_evaluation	Post-evaluation hooks	None

Validation

Always validate your configuration before running:

yb validate -c suite.yaml

For verbose output:

yb validate -c suite.yaml -v

Next Steps

• Suite YAML Reference - Complete field documentation
• Prompt Files - External prompt management
• Evaluators - Configure evaluation criteria