git-diff Evaluator

The git-diff evaluator analyzes Git changes made by the AI agent and provides assertion-based pass/fail thresholds.

Basic Usage

evaluators:
  - name: git-diff

Without configuration, git-diff reports metrics without enforcing any limits.

Configuration

evaluators:
  - name: git-diff
    config:
      assertions:
        max_files_changed: 5        # Maximum files that can be modified
        max_lines_added: 100        # Maximum lines that can be added
        max_lines_removed: 50       # Maximum lines that can be removed
        max_total_changes: 150      # Maximum total changes (additions + deletions)
        min_change_entropy: 0.5     # Minimum entropy (enforce distributed changes)
        max_change_entropy: 2.0     # Maximum entropy (enforce focused changes)

Available Assertions

Assertion	Description	Example
`max_files_changed`	Maximum number of files the agent can modify	`5`
`max_lines_added`	Maximum lines that can be added	`100`
`max_lines_removed`	Maximum lines that can be removed	`50`
`max_total_changes`	Maximum total line changes (add + remove)	`150`
`min_change_entropy`	Minimum change distribution (prevents all changes in one file)	`0.5`
`max_change_entropy`	Maximum change distribution (prevents scattered changes)	`2.0`

Metrics Produced

The evaluator produces the following metrics:

Metric	Type	Description
`files_changed`	number	Count of modified files
`lines_added`	number	Total lines added
`lines_removed`	number	Total lines removed
`total_changes`	number	Sum of additions and deletions
`change_entropy`	number	Distribution of changes across files
`changed_files`	string[]	Array of file paths
`file_metrics`	object	Per-file change statistics

Understanding Entropy

Change entropy measures how distributed changes are across files:

Low entropy (< 1.0): Changes concentrated in few files
High entropy (> 2.0): Changes scattered across many files

Use entropy assertions to enforce:

# Require focused changes (single feature)
assertions:
  max_change_entropy: 1.5

# Require distributed changes (refactoring)
assertions:
  min_change_entropy: 1.0

Example Output

{
  "name": "git-diff",
  "status": "passed",
  "metrics": {
    "files_changed": 3,
    "lines_added": 45,
    "lines_removed": 12,
    "total_changes": 57,
    "change_entropy": 1.23,
    "changed_files": [
      "src/auth/login.ts",
      "src/auth/middleware.ts",
      "tests/auth.test.ts"
    ],
    "file_metrics": {
      "src/auth/login.ts": { "added": 20, "removed": 5 },
      "src/auth/middleware.ts": { "added": 15, "removed": 7 },
      "tests/auth.test.ts": { "added": 10, "removed": 0 }
    }
  },
  "assertions": {
    "max_files_changed": { "expected": 5, "actual": 3, "passed": true },
    "max_lines_added": { "expected": 100, "actual": 45, "passed": true }
  }
}

Use Cases

Scope Limiting

Prevent agents from making excessive changes:

evaluators:
  - name: git-diff
    config:
      assertions:
        max_files_changed: 3
        max_total_changes: 50

Single-File Tasks

Ensure changes are focused on one file:

evaluators:
  - name: git-diff
    config:
      assertions:
        max_files_changed: 1

Refactoring Tasks

Allow more files but limit per-file changes:

evaluators:
  - name: git-diff
    config:
      assertions:
        max_files_changed: 20
        max_lines_added: 500
        min_change_entropy: 1.5

Best Practices

Start permissive - Begin without assertions to understand typical change sizes
Review metrics - Use results to set appropriate thresholds
Combine with other evaluators - Use alongside agentic-judge for quality assessment
Document thresholds - Explain why limits exist for future reference