Slack Notifications

Keep your team informed by sending evaluation results to Slack channels.

Quick Setup

1. Create Slack Webhook

1. Go to api.slack.com/apps
2. Create a new app or select existing
3. Enable Incoming Webhooks
4. Add webhook to your channel
5. Copy the webhook URL

2. Configure youBencha

suite.yaml

post_evaluation:
  - name: webhook
    config:
      url: ${SLACK_WEBHOOK_URL}
      method: POST

3. Set Environment Variable

export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T00/B00/XXXX"
yb run -c suite.yaml

Basic Webhook

youBencha sends a JSON payload that Slack can display:

post_evaluation:
  - name: webhook
    config:
      url: ${SLACK_WEBHOOK_URL}
      method: POST
      headers:
        Content-Type: "application/json"

Custom Slack Messages

For more control, use a script hook:

post_evaluation:
  - name: script
    config:
      command: ./scripts/slack-notify.sh
      args: ["${RESULTS_PATH}"]
      env:
        SLACK_WEBHOOK: "${SLACK_WEBHOOK_URL}"

Basic Notification Script

scripts/slack-notify.sh

#!/bin/bash
set -e

RESULTS=$1
STATUS=$(jq -r '.summary.overall_status' "$RESULTS")
PASSED=$(jq -r '.summary.passed' "$RESULTS")
FAILED=$(jq -r '.summary.failed' "$RESULTS")
SUITE=$(jq -r '.test_case.name // "Unnamed Suite"' "$RESULTS")

# Set color based on status
if [ "$STATUS" = "passed" ]; then
  COLOR="good"
  EMOJI="✅"
else
  COLOR="danger"
  EMOJI="❌"
fi

curl -X POST "$SLACK_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d "{
    \"attachments\": [{
      \"color\": \"$COLOR\",
      \"title\": \"$EMOJI youBencha: $SUITE\",
      \"fields\": [
        {\"title\": \"Status\", \"value\": \"$STATUS\", \"short\": true},
        {\"title\": \"Passed\", \"value\": \"$PASSED\", \"short\": true},
        {\"title\": \"Failed\", \"value\": \"$FAILED\", \"short\": true}
      ],
      \"footer\": \"youBencha Evaluation\",
      \"ts\": $(date +%s)
    }]
  }"

Rich Notification Script

Include evaluator details:

scripts/slack-notify-rich.sh

#!/bin/bash
set -e

RESULTS=$1
STATUS=$(jq -r '.summary.overall_status' "$RESULTS")
SUITE=$(jq -r '.test_case.name // "Unnamed Suite"' "$RESULTS")
DURATION=$(jq -r '.summary.duration_ms // 0' "$RESULTS")
DURATION_SEC=$((DURATION / 1000))

# Build evaluator summary
EVALUATORS=$(jq -r '.evaluators[] | "• \(.name): \(.status)"' "$RESULTS" | tr '\n' '\n')

if [ "$STATUS" = "passed" ]; then
  COLOR="good"
  EMOJI="✅"
  TITLE="Evaluation Passed"
else
  COLOR="danger"
  EMOJI="❌"
  TITLE="Evaluation Failed"
fi

curl -X POST "$SLACK_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d "{
    \"blocks\": [
      {
        \"type\": \"header\",
        \"text\": {
          \"type\": \"plain_text\",
          \"text\": \"$EMOJI $TITLE\"
        }
      },
      {
        \"type\": \"section\",
        \"fields\": [
          {\"type\": \"mrkdwn\", \"text\": \"*Suite:*\n$SUITE\"},
          {\"type\": \"mrkdwn\", \"text\": \"*Duration:*\n${DURATION_SEC}s\"}
        ]
      },
      {
        \"type\": \"section\",
        \"text\": {
          \"type\": \"mrkdwn\",
          \"text\": \"*Evaluators:*\n$EVALUATORS\"
        }
      }
    ]
  }"

Conditional Notifications

Only on Failure

scripts/notify-on-failure.sh

#!/bin/bash
RESULTS=$1
STATUS=$(jq -r '.summary.overall_status' "$RESULTS")

# Only notify on failure
if [ "$STATUS" = "passed" ]; then
  echo "Evaluation passed, skipping notification"
  exit 0
fi

# Send failure notification
curl -X POST "$SLACK_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d "{
    \"text\": \"⚠️ youBencha evaluation failed!\",
    \"attachments\": [{
      \"color\": \"danger\",
      \"text\": \"Check the logs for details.\"
    }]
  }"

Different Channels

post_evaluation:
  # Always notify team channel
  - name: webhook
    config:
      url: ${SLACK_TEAM_CHANNEL}

  # Alert on-call on failures (via script)
  - name: script
    config:
      command: ./scripts/alert-oncall.sh
      args: ["${RESULTS_PATH}"]
      env:
        ONCALL_WEBHOOK: "${SLACK_ONCALL_CHANNEL}"

GitHub Actions Integration

.github/workflows/youbencha.yml

- name: Run Evaluation
  run: yb run -c suite.yaml
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

- name: Notify on Failure
  if: failure()
  run: |
    curl -X POST "$SLACK_WEBHOOK_URL" \
      -H "Content-Type: application/json" \
      -d '{"text": "❌ youBencha evaluation failed in CI"}'
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Message Templates

Minimal

{
  "text": "youBencha: Evaluation passed ✅"
}

With Details

{
  "attachments": [{
    "color": "good",
    "title": "youBencha Evaluation",
    "fields": [
      {"title": "Status", "value": "passed", "short": true},
      {"title": "Evaluators", "value": "3/3 passed", "short": true}
    ]
  }]
}

With Actions

{
  "blocks": [
    {
      "type": "section",
      "text": {"type": "mrkdwn", "text": "✅ *Evaluation Passed*"}
    },
    {
      "type": "actions",
      "elements": [
        {
          "type": "button",
          "text": {"type": "plain_text", "text": "View Report"},
          "url": "https://github.com/org/repo/actions"
        }
      ]
    }
  ]
}

Best Practices

1. Use environment variables for webhook URLs
2. Set appropriate retry for network issues
3. Rate limit notifications to avoid spam
4. Include context like suite name and duration
5. Use threading for related notifications

Related

• Post-Evaluation Hooks
• CI/CD Integration
• Configuration Reference