An Empirical Approach for Validation of Inter-Rater Reliability of Identified Candidate Aspects

Authors

  • JS Aravindan Department of Computer Science, Bharathiar University, Coimbatore, India
  • K Vivekanandan BSMED, Bharathiar University, Coimbatore, India

Keywords:

Kappa Score, Aspects, Relevance, Reliability

Abstract

The development of technologies for observing the relevancy of identified candidate aspects of the products and validating the reliability of filed experts is a challenging process. If the identified candidate aspects is taken into consideration for decision making process, the technology has to meet certain feasibility criteria like acceptance of aspects by product experts (Judges). Questionnaires with extracted aspects were distributed to Judges and the candidate aspects are identified. This paper describes the content validity evaluation of identified candidate aspects. The validity of the instrument was evaluated by two field experts who were asked to score each aspect on a 4-point Likert scale (1. Not Relevant, 2. Item need some revision, 3. Relevant but need minor revision, 4. Very Relevant) on relevance of 5 product (Apex AD2600 Progressive-scan DVD player, Canon G3, Creative Labs Nomad Jukebox Zen Xtra 40GB, Nikon Coolpix 4300 and Nokia 6610) datasets and the Kappa Score is calculated.

References

M. Alwan, B. Turner, S. Kell, K. Penberthy, W. Cohn, R. Felder, “Development of Survey Instruments to Guide the Design of Health Status Monitoring Systems for the Elderly: Content Validity Evaluation”, In the Proceedings of the 2006 IEEE International Conference on Information & Communication Technologies (ICTTA 2006), Syria, pp.793-797, 2006.

V. Bhambri, “Data Mining as a Solution for Data Management in Banking Sector”, International Journal of Computer Sciences and Engineering, Vol.1, Issue.1, pp.20-25, 2013.

M. Hu, B. Liu, “Mining and summarizing customer reviews”, In the Proceedings of 2004 ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 2004), New York, pp.168-177, 2004.

J. Cohen, “A Coefficient of Agreement for Nominal Scales” Educational and Psychological Measurement, Vol.20, Issue.1, pp.37-46, 1960.

J. Keyton, T. King, NM. Mabachi, J. Manning, LL. Leonard, D. Schill, “Content analysis procedure book”, Lawrence KS: University of Kansas, Kansas, pp.22-37, 2004.

M. Shelley, K. Krippendorff, “Content Analysis: An Introduction to its Methodology”, Journal of the American Statistical Association, Vol.79, Issue.385, pp.240-144, 1984.

Downloads

Published

2025-11-11

How to Cite

[1]
J. Aravindan and K. Vivekanandan, “An Empirical Approach for Validation of Inter-Rater Reliability of Identified Candidate Aspects”, Int. J. Comp. Sci. Eng., vol. 5, no. 5, pp. 94–100, Nov. 2025.

Issue

Vol. 5 No. 5 (2017): IJCSE May Edition

Section

Research Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.