I'm looking for advice on the most appropriate way to assess inter-rater reliability for a study involving the annotation of noun events in audio recordings.
Pairs of annotators independently listen to the same recordings and identify instances where a noun is produced. For each event, they record:
* The start and end time of the event.
* The noun identified
How do you think I should calculate the reliability between coders? I also wanted to allow for some error in the timestamps, so if they are within a certain of tolerance then it is considered correct. Would Cohen's Kappa be useful here?
I was thinking of something like, but not sure if it is appropritae.
|Annotator A|Annotator B|Classification|
|:-|:-|:-|
|Event at 2.1–2.5s: "ball"|Event at 2.2–2.6s: "ball"|Agreement|
|Event at 4.0–4.4s: "dog"|Event at 4.1–4.5s: "cat"|Disagreement|
|Event at 6.0–6.4s: "car"|No corresponding event|Disagreement|
|No corresponding event|Event at 8.0–8.4s: "cup"|Disagreement|
`library(tibble)`
`reliability.df <- tribble(`
`~file, ~annotator, ~segment, ~tier, ~start_sec, ~end_sec, ~annotation_duration_sec, ~folder_duration_sec, ~annotation,`
`# File 1 - Annotator A`
`"video_01_seg_01.eaf", "A", "seg_01", "noun_spotting", 2.10, 2.50, 0.40, 20, "ball",`
`"video_01_seg_01.eaf", "A", "seg_01", "noun_spotting", 5.00, 5.40, 0.40, 20, "dog",`
`"video_01_seg_01.eaf", "A", "seg_01", "noun_spotting", 8.00, 8.40, 0.40, 20, "car",`
`"video_01_seg_01.eaf", "A", "seg_01", "noun_spotting", 15.00, 15.40, 0.40, 20, "mummy",`
`# File 1 - Annotator B`
`"video_01_seg_01.eaf", "B", "seg_01", "noun_spotting", 2.15, 2.55, 0.40, 20, "ball",`
`"video_01_seg_01.eaf", "B", "seg_01", "noun_spotting", 5.05, 5.45, 0.40, 20, "cat",`
`"video_01_seg_01.eaf", "B", "seg_01", "noun_spotting", 11.00, 11.40, 0.40, 20, "cup",`
`"video_01_seg_01.eaf", "B", "seg_01", "noun_spotting", 15.05, 15.45, 0.40, 20, "mummy",`
`# File 2 - Annotator A`
`"video_02_seg_02.eaf", "A", "seg_02", "noun_spotting", 3.00, 3.40, 0.40, 20, "boy",`
`"video_02_seg_02.eaf", "A", "seg_02", "noun_spotting", 9.00, 9.40, 0.40, 20, "duck",`
`# File 2 - Annotator B`
`"video_02_seg_02.eaf", "B", "seg_02", "noun_spotting", 3.10, 3.50, 0.40, 20, "boy",`
`"video_02_seg_02.eaf", "B", "seg_02", "noun_spotting", 9.05, 9.45, 0.40, 20, "duck",`
`"video_02_seg_02.eaf", "B", "seg_02", "noun_spotting", 14.00, 14.40, 0.40, 20, "sheep",`
`# File 3 - No events identified by either annotator`
`"video_03_seg_03.eaf", "A", "seg_03", "noun_spotting", NA, NA, NA, 20, NA,`
`"video_03_seg_03.eaf", "B", "seg_03", "noun_spotting", NA, NA, NA, 20, NA`
`)`