Reliability of observational- and machine-based teat hygiene scoring methodologies

Removal of teat-end debris is one of the most critical steps in the premilking process. We aimed to estimate inter- and intra-rater reliability of an observation-based rating scale of dairy parlor worker teat-cleaning performance. A nonrandom sample of 8 experienced raters provided teat swab debris ratings scored on a 4-point ordinal visual scale for 175 teat swab images taken immediately after teat cleaning and before milking unit attachment. To overcome the uncertainty associated with visual inspection and observation-based rating scales, we assessed the reliability of an automated observer-independent method to assess teat-end debris using digital image processing and machine learning techniques to quantify the type and amount of debris material present on each teat swab image. Cohen's kappa coefficient (κ) was used to assess inter-rater score agreement on 175 teat swab images, and the intraclass correlation coefficient was used to assess both intra-rater score agreement and machine reliability. The reliability of debris scoring of teat swabs by raters was low (overall κ = 0.43), whereas the machine-based rating system demonstrated near-perfect reliability (Pearson r > 0.99). Our findings suggest that machine-based rating systems of worker performance are much more reliable than observational-based methods when evaluating premilking teat cleanliness. Teat swab image analysis technology can be further developed for training and quality control purposes to enable more efficient, reliable, and independent feedback on worker milking performance. As automated technologies are becoming more popular on dairy farms, machine-based teat cleanliness scoring could also be incorporated into automated milking systems.