Troubleshooting

Solutions to common problems when using youBencha.

Installation Issues

youBencha Not Found

Error: yb: command not found

Solutions:

1. Install globally:

   npm install -g youbencha

2. Check npm global bin path:

   npm bin -g
   # Add to PATH if needed

3. Use npx if installed locally:

   npx yb --version

Node.js Version Error

Error: Node.js 20+ required

Solution:

# Check version
node --version

# Install Node.js 20+ using nvm
nvm install 20
nvm use 20

Configuration Errors

Invalid YAML Syntax

Error: YAML parse error at line 5

Common causes:

1. Tabs instead of spaces:

   # ❌ Wrong (tabs)
   agent:
     type: copilot-cli
   
   # ✅ Correct (spaces)
   agent:
     type: copilot-cli

2. Missing colons:

   # ❌ Wrong
   agent
     type: copilot-cli
   
   # ✅ Correct
   agent:
     type: copilot-cli

3. Improper quoting:

   # ❌ Wrong
   prompt: Add "quotes" here
   
   # ✅ Correct
   prompt: 'Add "quotes" here'

Invalid JSON Syntax

Error: Unexpected token in JSON

Common causes:

1. Trailing commas:

   // ❌ Wrong
   { "key": "value", }
   
   // ✅ Correct
   { "key": "value" }

2. Single quotes:

   // ❌ Wrong
   { 'key': 'value' }
   
   // ✅ Correct
   { "key": "value" }

Schema Validation Error

Error: Invalid evaluator type 'unknown-eval'

Solution:

List available evaluators:

yb list

Use valid evaluator names: git-diff, expected-diff, agentic-judge

File Not Found

Error: Prompt file not found: ./prompts/task.md

Solutions:

1. Check file path relative to suite.yaml
2. Create the missing file
3. Verify file permissions

Agent Errors

Copilot CLI Not Found

Error: Copilot CLI not installed

Solution:

npm install -g @githubnext/github-copilot-cli

GitHub Authentication Failed

Error: GitHub authentication required

Solution:

gh auth login
gh auth status  # Verify

Copilot Subscription Required

Error: GitHub Copilot subscription required

Solution:

1. Verify subscription at github.com/settings/copilot
2. Ensure token has copilot scope

Model Not Available

Error: Model 'xyz' not available

Solutions:

1. Check supported models in Model Selection
2. Verify subscription tier supports the model
3. Use a different model:

   agent:
     model: claude-sonnet-4.5

Evaluator Errors

expected-diff Missing Reference

Error: expected_source and expected required for expected-diff

Solution:

Add reference configuration:

expected_source: branch
expected: feature/completed

evaluators:
  - name: expected-diff

agentic-judge Assertion Error

Error: Assertion 'quality' produced invalid score

Solutions:

1. Make assertion text clearer:

   assertions:
     quality: "Code quality is high. Score exactly 0 or 1."

2. Check agent output in logs

Workspace Issues

Workspace Already Exists

Error: Workspace lock file exists

Solution:

Clean up stale workspace:

rm -rf .youbencha-workspace/run-*

Disk Space Full

Error: No space left on device

Solutions:

1. Clean old workspaces:

   rm -rf .youbencha-workspace/

2. Use custom workspace location:

   workspace_dir: /tmp/youbencha

Permission Denied

Error: Permission denied: .youbencha-workspace

Solution:

chmod -R u+rwx .youbencha-workspace/

Network Issues

Repository Clone Failed

Error: Failed to clone repository

Solutions:

1. Check URL is accessible:

   git ls-remote https://github.com/org/repo.git

2. Verify authentication:

   gh auth status

3. Check for typos in URL

Webhook Failed

Warning: Webhook POST failed: Connection refused

Solutions:

1. Verify webhook URL is correct
2. Check network connectivity
3. Ensure endpoint accepts POST requests

Timeout Issues

Agent Timeout

Error: Agent execution timed out after 300000ms

Solution:

Increase timeout:

timeout: 600000  # 10 minutes

Pre-Execution Timeout

Error: Pre-execution hook timed out

Solution:

pre_execution:
  - name: script
    config:
      timeout_ms: 120000  # 2 minutes

Debugging Steps

1. Validate Configuration

yb validate -c suite.yaml -v

2. Check Agent Output

cat .youbencha-workspace/run-*/artifacts/youbencha.log.json | jq

3. Review Diff

cat .youbencha-workspace/run-*/artifacts/git-diff.patch

4. Inspect Workspace

ls -la .youbencha-workspace/run-*/src-modified/

5. Check Results

cat .youbencha-workspace/run-*/artifacts/results.json | jq '.summary'

Getting Help

Check Documentation

• Configuration Reference
• CLI Reference
• Best Practices

Report Issues

Open an issue on GitHub:

• youbencha/youbencha-cli

Include:

• youBencha version (yb --version)
• Node.js version (node --version)
• Error message
• Suite configuration (remove secrets)
• Steps to reproduce