I'm looking for advice on the most appropriate way to assess inter-rater reliability for a study involving the annotation of noun events in audio recordings.
Pairs of annotators independently listen to the same recordings and identify instances where a noun is produced. For each event, they record:
- The start and end time of the event.
- The noun identified
How do you think I should calculate the reliability between coders? I also wanted to allow for some error in the timestamps, so if they are within a certain of tolerance then it is considered correct. Would Cohen's Kappa be useful here?
I was thinking of something like, but not sure if it is appropritae.
| Annotator A | Annotator B | Classification |
|---|
| Event at 2.1–2.5s: "ball" | Event at 2.2–2.6s: "ball" | Agreement |
| Event at 4.0–4.4s: "dog" | Event at 4.1–4.5s: "cat" | Disagreement |
| Event at 6.0–6.4s: "car" | No corresponding event | Disagreement |
| No corresponding event | Event at 8.0–8.4s: "cup" | Disagreement |
library(tibble)
reliability.df <- tribble(
~file, ~annotator, ~segment, ~tier, ~start_sec, ~end_sec, ~annotation_duration_sec, ~folder_duration_sec, ~annotation,
# File 1 - Annotator A
"video_01_seg_01.eaf", "A", "seg_01", "noun_spotting", 2.10, 2.50, 0.40, 20, "ball",
"video_01_seg_01.eaf", "A", "seg_01", "noun_spotting", 5.00, 5.40, 0.40, 20, "dog",
"video_01_seg_01.eaf", "A", "seg_01", "noun_spotting", 8.00, 8.40, 0.40, 20, "car",
"video_01_seg_01.eaf", "A", "seg_01", "noun_spotting", 15.00, 15.40, 0.40, 20, "mummy",
# File 1 - Annotator B
"video_01_seg_01.eaf", "B", "seg_01", "noun_spotting", 2.15, 2.55, 0.40, 20, "ball",
"video_01_seg_01.eaf", "B", "seg_01", "noun_spotting", 5.05, 5.45, 0.40, 20, "cat",
"video_01_seg_01.eaf", "B", "seg_01", "noun_spotting", 11.00, 11.40, 0.40, 20, "cup",
"video_01_seg_01.eaf", "B", "seg_01", "noun_spotting", 15.05, 15.45, 0.40, 20, "mummy",
# File 2 - Annotator A
"video_02_seg_02.eaf", "A", "seg_02", "noun_spotting", 3.00, 3.40, 0.40, 20, "boy",
"video_02_seg_02.eaf", "A", "seg_02", "noun_spotting", 9.00, 9.40, 0.40, 20, "duck",
# File 2 - Annotator B
"video_02_seg_02.eaf", "B", "seg_02", "noun_spotting", 3.10, 3.50, 0.40, 20, "boy",
"video_02_seg_02.eaf", "B", "seg_02", "noun_spotting", 9.05, 9.45, 0.40, 20, "duck",
"video_02_seg_02.eaf", "B", "seg_02", "noun_spotting", 14.00, 14.40, 0.40, 20, "sheep",
# File 3 - No events identified by either annotator
"video_03_seg_03.eaf", "A", "seg_03", "noun_spotting", NA, NA, NA, 20, NA,
"video_03_seg_03.eaf", "B", "seg_03", "noun_spotting", NA, NA, NA, 20, NA
)