yb init

Create a starter configuration file with helpful comments and sensible defaults.

Synopsis

yb init [options]

Description

The init command creates a suite.yaml file in your current directory with a fully-commented template. This is the recommended way to start a new youBencha evaluation project.

The generated template includes:

• Repository configuration with placeholders
• Agent setup with example prompts
• Common evaluator configurations
• Helpful comments explaining each field

Options

Option	Short	Type	Default	Description
--force	-f	flag	false	Overwrite existing suite.yaml if present
--template	-t	string	default	Template to use (default, minimal, advanced)
--output	-o	string	suite.yaml	Output file path
--help	-h	flag	-	Show help message

Examples

Basic Initialization

Create a new suite configuration in the current directory:

Terminal

yb init

Tip

This is the most common usage. The generated file includes comments explaining each option.

Force Overwrite

Replace an existing configuration file:

Terminal

yb init --force

Caution

The --force flag completely replaces your existing configuration. Back up your file first if needed.

Minimal Template

Create a minimal configuration without comments:

Terminal

yb init --template minimal

Custom Output Path

Create configuration in a specific location:

Terminal

yb init --output configs/my-evaluation.yaml

Combined Options

Create a minimal config, overwriting if exists:

Terminal

yb init -f -t minimal -o test-suite.yaml

Generated File Structure

Default Template

suite.yaml

# youBencha Evaluation Suite Configuration
# See https://youbencha.dev/docs/configuration for full documentation

# Repository to evaluate
repo: https://github.com/youbencha/hello-world.git
branch: main

# AI Agent Configuration
agent:
  type: copilot-cli
  config:
    prompt: "Add a comment explaining what this repo is about"

# Evaluation Criteria
evaluators:
  - name: git-diff
    # No additional config needed for basic diff analysis

  - name: agentic-judge
    config:
      type: copilot-cli
      assertions:
        task_completed: "The task was completed successfully. Score 1 if yes, 0 if no."

Minimal Template

suite.yaml

repo: https://github.com/owner/repo.git
branch: main
agent:
  type: copilot-cli
  config:
    prompt: "Your prompt here"
evaluators:
  - name: git-diff

Advanced Template

suite.yaml

# Advanced youBencha Configuration

repo: https://github.com/owner/repo.git
branch: main
expected_source: branch
expected: feature/reference-implementation

agent:
  type: copilot-cli
  config:
    prompt_file: ./prompts/task.md
    model: gpt-4

hooks:
  pre_execution:
    - ./scripts/setup.sh
  post_evaluation:
    - ./scripts/cleanup.sh

evaluators:
  - name: git-diff
    config:
      assertions:
        max_files_changed: 10
        max_lines_added: 500

  - name: expected-diff
    config:
      threshold: 0.85

  - name: agentic-judge
    config:
      type: copilot-cli
      evaluator_file: ./evaluators/quality-check.md

Exit Codes

Code	Meaning
0	Configuration file created successfully
1	File already exists (use --force to overwrite)
2	Write permission denied

Workflow

1. Run yb init to create the configuration file
2. Edit suite.yaml to configure your evaluation
3. Run yb validate -c suite.yaml to check your configuration
4. Run yb run -c suite.yaml to execute the evaluation

Related Commands

• yb run - Execute an evaluation suite
• yb validate - Validate configuration
• Configuration Reference - Complete configuration options