yb validate

Validate a suite configuration file without running the evaluation.

Synopsis

yb validate -c <config-file> [options]

Description

The validate command performs comprehensive validation of your suite configuration. It checks everything that would cause a run to fail, without actually executing the evaluation.

Validation Checks

• YAML/JSON syntax - Ensures valid file format
• Schema compliance - Validates against youBencha schema
• File references - Checks that prompt files exist
• Evaluator configurations - Validates evaluator settings
• Security constraints - Blocks localhost/internal repos
• Agent availability - Verifies agent type is supported

Tip

Always validate before running evaluations to catch errors early and save time.

Options

Option	Short	Type	Default	Description
--config	-c	string	Required	Path to suite configuration file
--verbose	-v	flag	false	Show detailed validation information
--strict		flag	false	Fail on warnings (not just errors)
--check-network		flag	false	Verify repository URL is accessible
--help	-h	flag	-	Show help message

Examples

Basic Validation

Quickly check if configuration is valid:

Terminal

yb validate -c suite.yaml

Output on success:

✅ Configuration is valid

Verbose Mode

Get detailed validation information:

Terminal

yb validate -c suite.yaml -v

Verbose Success

Output

Validating: suite.yaml

✅ YAML syntax: valid
✅ Schema: valid
✅ Repository URL: valid (https://github.com/...)
✅ Agent type: copilot-cli (supported)
✅ Evaluators: 2 configured
   - git-diff: valid
   - agentic-judge: valid
✅ Prompt file: ./prompts/task.md (exists)

Configuration is valid and ready to run.

Verbose with Warnings

Output

Validating: suite.yaml

✅ YAML syntax: valid
✅ Schema: valid
✅ Repository URL: valid
✅ Agent type: copilot-cli
⚠️ Warning: No evaluators configured
⚠️ Warning: prompt_file not found, using inline prompt

Configuration is valid with 2 warnings.

Strict Mode

Fail on warnings as well as errors:

Terminal

yb validate -c suite.yaml --strict

Network Verification

Also verify the repository URL is accessible:

Terminal

yb validate -c suite.yaml --check-network

Note

Network verification requires internet access and may slow down validation.

Validate Before Running

Combine validation and run in a single command:

Terminal

yb validate -c suite.yaml && yb run -c suite.yaml

Validation Error Examples

Syntax Errors

Terminal

$ yb validate -c suite.yaml

❌ Validation failed

Errors:
  - Line 5: Invalid YAML syntax - missing colon
  - Line 12: Unexpected indentation

Fix: Check YAML syntax, ensure proper indentation (2 spaces).

Schema Errors

Terminal

$ yb validate -c suite.yaml

❌ Validation failed

Errors:
  - Missing required field: 'repo'
  - Invalid agent type 'unknown'. Supported: copilot-cli
  - Field 'evaluators' must be an array

Fix: Add missing required fields, use valid values.

File Errors

Terminal

$ yb validate -c suite.yaml

❌ Validation failed

Errors:
  - Prompt file not found: ./prompts/task.md
  - Evaluator file not found: ./evaluators/custom.md

Fix: Create missing files or update paths.

Security Errors

Terminal

$ yb validate -c suite.yaml

❌ Validation failed

Security violations:
  - Repository URL contains localhost (blocked)
  - Internal IP detected in repo URL: 192.168.1.1

Fix: Use public repository URLs only.

What Gets Validated

Repository Settings

Check	Description	Example
URL format	Valid HTTP/HTTPS URL	https://github.com/org/repo.git
No localhost	Blocks local URLs	❌ http://localhost/repo
No internal IPs	Blocks private networks	❌ http://192.168.1.1/repo
Branch format	Valid Git branch name	main, feature/test

Agent Configuration

Check	Description	Example
Supported type	Known agent type	copilot-cli
Prompt present	Has prompt or prompt_file	Required
Model valid	Valid model name (if specified)	gpt-4, gpt-3.5-turbo

Evaluators

Check	Description	Example
Valid names	Known evaluator names	git-diff, agentic-judge
Required config	Config present when needed	agentic-judge needs assertions
File references	Referenced files exist	evaluator_file: ./eval.md

Security

Check	Description
SSRF prevention	No localhost or internal network URLs
URL schemes	Only HTTP/HTTPS allowed
Path traversal	No ../ in file paths

Exit Codes

Code	Meaning
0	Configuration is valid
1	Validation errors found
2	Configuration file not found

Best Practices

1. Always validate before running - Catch errors early
2. Use verbose mode when debugging - Get detailed feedback
3. Use strict mode in CI/CD - Treat warnings as errors
4. Validate after editing - Confirm changes are valid

Recommended Workflow

# Edit configuration
vim suite.yaml

# Validate changes
yb validate -c suite.yaml -v

# Run if valid
yb run -c suite.yaml

Related Commands

• yb run - Execute evaluations
• Configuration Reference - Complete configuration options
• Troubleshooting - Common validation issues