Copilot CLI Adapter

The Copilot CLI adapter integrates youBencha with GitHub Copilot CLI for AI-powered code generation and modification.

Prerequisites

GitHub Copilot CLI installed:

npm install -g @githubnext/github-copilot-cli

GitHub authentication configured:
Terminal window
```
gh auth login
```
Active GitHub Copilot subscription

Basic Configuration

agent:
  type: copilot-cli
  config:
    prompt: "Add error handling to all API endpoints"

Configuration Options

type (required)

Always set to copilot-cli for this adapter.

agent:
  type: copilot-cli

model

Specify the AI model to use:

agent:
  type: copilot-cli
  model: claude-sonnet-4.5

Supported models:

claude-sonnet-4.5, claude-sonnet-4, claude-haiku-4.5
gpt-5, gpt-5.1, gpt-5.1-codex-mini, gpt-5.1-codex
gemini-3-pro-preview

agent_name

Use a named agent from .github/agents/ directory:

agent:
  type: copilot-cli
  agent_name: my-custom-agent

When specified:

The .github/agents/ directory is copied to workspace
Agent is invoked with --agent <name> flag
Custom agent instructions are applied

config.prompt

Inline prompt for the agent:

agent:
  type: copilot-cli
  config:
    prompt: "Add JWT authentication to the login endpoint"

config.prompt_file

Load prompt from external file:

agent:
  type: copilot-cli
  config:
    prompt_file: ./prompts/add-auth.md

Named Agents

Named agents allow you to use custom agent configurations stored in your repository.

Directory Structure

your-repo/
└── .github/
    └── agents/
        ├── code-reviewer.agent.md
        ├── test-writer.agent.md
        └── refactorer.agent.md

Agent File Format

# Code Reviewer Agent

You are an expert code reviewer. Focus on:

- Code quality and maintainability
- Security vulnerabilities
- Performance issues
- Best practices

Be thorough but constructive.

Using Named Agents

agent:
  type: copilot-cli
  agent_name: code-reviewer
  config:
    prompt: "Review the authentication module"

Complete Example

name: auth-feature-eval
description: Evaluate authentication implementation

repo: https://github.com/example/api-server.git
branch: main

agent:
  type: copilot-cli
  model: claude-sonnet-4.5
  agent_name: security-focused
  config:
    prompt_file: ./prompts/add-jwt-auth.md

evaluators:
  - name: git-diff
    config:
      assertions:
        max_files_changed: 10

  - name: agentic-judge
    config:
      type: copilot-cli
      assertions:
        auth_complete: "JWT auth is fully implemented. Score 0-1."

Environment Variables

The adapter uses these environment variables:

Variable	Description
`GITHUB_TOKEN`	GitHub authentication token
`COPILOT_MODEL`	Default model (can be overridden)

Troubleshooting

Agent Not Found

Error: Copilot CLI not found

Solution: Install Copilot CLI:

npm install -g @githubnext/github-copilot-cli

Authentication Failed

Error: GitHub authentication required

Solution: Authenticate with GitHub:

gh auth login

Subscription Required

Error: GitHub Copilot subscription required

Solution: Ensure you have an active Copilot subscription at github.com/settings/copilot.

Model Not Available

Error: Model 'xyz' not available

Solution: Check supported models and your subscription tier. Some models require Copilot Enterprise.

Best Practices

Use named agents for consistent behavior across evaluations
Specify models explicitly for reproducible results
Use prompt files for complex instructions
Test authentication before running evaluations