# Inspect reviewer agreement and actual disagreements

Illustrative planning brief; no automatic product import.

Synthetic planning case: Two reviewers independently apply the same toy acceptance rubric to five cases. They agree on A,B,C and disagree on D,E.

## Decision

Observed agreement is three of five, or 60%. Retain the two disputed cases and their criteria; the percentage alone cannot establish reviewer reliability or decide which verdict is correct.

## Owned work

- Preserve independent verdicts
  - Owner role: Review coordinator
  - Acceptance evidence: Initial labels were recorded before a consensus discussion.
- Locate the disputed criteria
  - Owner role: Rubric owner
  - Acceptance evidence: D and E include the evidence and rule each reviewer used.
- Calibrate the next review
  - Owner role: Acceptance lead
  - Acceptance evidence: A clarified rubric is tested on fresh cases without rewriting the initial disagreement log.

## Workflow

1. Preserve independent verdicts. Check: Initial labels were recorded before a consensus discussion.
2. Locate the disputed criteria. Check: D and E include the evidence and rule each reviewer used.
3. Calibrate the next review. Check: A clarified rubric is tested on fresh cases without rewriting the initial disagreement log.

## Judgment

A five-case synthetic log does not validate a rubric or reviewer competence. Report actual disagreements and context.


## Filled manual planning note

Observed agreement is three of five, or 60%. Retain the two disputed cases and their criteria; the percentage alone cannot establish reviewer reliability or decide which verdict is correct. Agreement count=3; total=5;3/5=60%. This is a descriptive exact-label match, not a chance-adjusted reliability coefficient. A five-case synthetic log does not validate a rubric or reviewer competence. Report actual disagreements and context.


## Workflow questions

### Does agreement prove both reviewers are right?

No. Two people can share the same mistaken interpretation.

### Can I average pass and hold into acceptance?

Not automatically. Resolve the governing criterion and required evidence.

## Product connection

Use the owned checks and downloaded brief to discuss this planning decision alongside your TeamBoostAI tasks. Confirm available fields, roles and account features separately. The example is manual; it does not calculate live analytics, create work or run an experiment in the product.

Confirm account availability before adopting this manual outline.

## Original worked case

Synthetic records, manual planning only. No account import or live analytics.

### Inspect the invented case records

Case | Reviewer 1 | Reviewer 2 | Agreement
--- | --- | --- | ---
A | Pass | Pass | Yes
B | Hold | Hold | Yes
C | Pass | Pass | Yes
D | Pass | Hold | No
E | Hold | Pass | No

### Reasoning

Agreement count=3; total=5;3/5=60%. This is a descriptive exact-label match, not a chance-adjusted reliability coefficient.

### Bounded result

Observed agreement is three of five, or 60%. Retain the two disputed cases and their criteria; the percentage alone cannot establish reviewer reliability or decide which verdict is correct.

### Distinct decision

The artifact examines reviewer-label agreement with inspectable case-level disagreement.

### Limits

A five-case synthetic log does not validate a rubric or reviewer competence. Report actual disagreements and context.

### Definitions and method context

- Original worked-case definitions — Definitions, policy choices, records and calculations are authored for this worksheet. No external standard, statistical validation or live product measurement is claimed.