Intercoder reliability for LLM annotations
Usage
intercoder_reliability(
annotations,
method = c("auto", "cohen", "krippendorff"),
label_levels = NULL,
sample_col = "sample_id",
model_col = "model_id",
label_col = "label"
)Arguments
- annotations
Output from [explore()] or a compatible data frame with at least `sample_id`, `model_id`, and `label` columns.
- method
One of `"auto"`, `"cohen"`, or `"krippendorff"`. The `"auto"` setting computes both pairwise Cohen's Kappa and Krippendorff's Alpha (when applicable).
- label_levels
Optional factor levels to enforce a consistent ordering in the resulting tables.
- sample_col
Column name that identifies samples when `annotations` is a user-provided data frame.
- model_col
Column name for the model identifier when using a custom `annotations` data frame.
- label_col
Column name containing model predictions when using a custom `annotations` data frame.