Comparative Statistical Analysis: Semantic Adequacy between CV and Job Offer across TF-IDF+Cosine and AI-Assisted ATS Approaches.
DOI:
https://doi.org/10.55204/trc.v6i2.e667Keywords:
ATS, TF-IDF, Semantic similarity, Comparative statistical analysisAbstract
Abstract:
This study presents a comparative statistical analysis of semantic adequacy between two text-processing approaches applied to Applicant Tracking System (ATS) processes. The comparison involves a classical pipeline based on TF-IDF and cosine similarity and an artificial intelligence-assisted pipeline using the OpenAI API. A convenience sample of résumés submitted for a specific job vacancy was used. Each CV was anonymized and evaluated using both methods under a paired within-résumé research design. The metrics compared were similarity_score, coverage, diversity, density, confidence, and rationale_relevance. The statistical analysis included the Shapiro–Wilk test to assess the normality of the differences (deltas), paired t-test or Wilcoxon signed-rank test, effect sizes, Holm correction, and Bland–Altman analysis and the Intraclass Correlation Coefficient (ICC) to assess agreement. The results reveal systematic differences between the two approaches: the classical method tends to “inflate” similarity and the overall score, whereas the artificial intelligence-assisted method tends to be more sensitive in capturing inherent concepts that are not explicitly represented in the text.
Downloads
References
Ajjam, M.-H., & Al-Raweshidy, H. S. (2026). [Artículo sobre matching CV–oferta con TF-IDF y cosine similarity]. Information Sciences, 724, 122728.
Alonso, R., Dessí, D., Meloni, A., et al. (2025). [Artículo sobre job–résumé matching y extracción de texto en CVs no textuales]. Big Data Research, 39, 100509. https://doi.org/10.1016/j.bdr.2025.100509
Bestgen, Y. (2014). Improving the validity of the automated essay scoring system under conditions of text complexity variation. Journal of Writing Research, 6(2), 123–143. https://doi.org/10.17239/jowr-2014.06.02.2
Bevara, R. V. K., Dronavalli, S. C., Karedla, S. P., Rupeshkumar, S., & Xiao, T. (2025). Resume2Vec: Transformer-based resume embeddings for improved semantic matching in applicant tracking systems. Electronics, 14(4), 794. https://doi.org/10.3390/electronics14040794
Bevara, R. V. K., Mannuru, N. R., Karedla, S. P., Lund, B., Xiao, T., Pasem, H., Dronavalli, S. C., & Rupeshkumar, S. (2025). Resume2Vec: Transforming applicant tracking systems with intelligent resume embeddings for precise candidate matching. Electronics, 14(4), 794. https://doi.org/10.3390/electronics14040794
Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307–310. https://doi.org/10.1016/S0140-6736(86)90837-8
Chafi, S., Kabil, M., & Kamouss, A. (2025). Optimizing automatic CV classification with contrastive and generative learning. Procedia Computer Science, 265, 342–349. https://doi.org/10.1016/j.procs.2025.07.190
Chafi, S., Kabil, M., & Kamouss, A. (2025). Optimizing automatic CV classification with contrastive and generative learning. Procedia Computer Science, 265, 342–349. https://doi.org/10.1016/j.procs.2025.07.190
Chong, K. W., Ng, K. W., & Fu, Y. H. (2026). Resume data extract and job recruitment chatbot features for AI-based resume screening & analytics. MethodsX, 16, 103775. https://doi.org/10.1016/j.mex.2025.103775
Deshmukh, A., Raut, A., & Dahake, S. (2025). Intelligent resume screening system using BERT and cosine similarity. International Journal of Artificial Intelligence, 14(4), 3404–3411. https://doi.org/10.11591/ijai.v14.i4.pp3404-3411
Deshmukh, A., Raut, A., & Dahake, S. (2026). Intelligent resume screening system using BERT and cosine similarity. International Journal of Artificial Intelligence, 15(1), 953–961. https://doi.org/10.11591/ijai.v15.i1.pp953-961
Deshmukh, A., Raut, A., & Deshmukh, V. (2026). Scalable resume screening using large language model Meta AI version 3. IAES International Journal of Artificial Intelligence (IJ-AI), 15(1), 953–961. https://doi.org/10.11591/ijai.v15.i1.pp953-961
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Liu, X. (2025). Deep learning-based intelligent resume-position matching system: Semantic understanding and recommendation of BERT model in massive recruitment data. In Proceedings of MLSC 2025. https://doi.org/10.1145/3778450.3778452
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46
Mendoza, L. E., & Ramírez, J. P. (2020). Inteligencia artificial aplicada a la gestión del talento humano: Retos y oportunidades. Revista Iberoamericana de Tecnología y Sociedad, 12(3), 45–60.
Mkrtchyan, A. (2025). Text-based approaches for exploring job matching techniques (BS in Data Science thesis). American University of Armenia, Zaven & Sonia Akian College of Science and Engineering.
Murel, J., & Kavlakoglu, E. (2024). ¿Qué es el modelado de temas?. IBM Research.
Myneni, M. B., & Quadhari, F. S. (2025). [Artículo sobre resume parsing/matching y uso de cosine similarity y embeddings]. IJECE, 12(9), 63–71.
Nawalny, M., Łepicki, M., Latkowski, T., et al. (2025/2026). Comparative evaluation of GPT-4o, GPT-OSS-120B and Llama-3.1-8B-Instruct language models in a reproducible CV-to-JSON extraction pipeline. Applied Sciences, 16, 217. https://doi.org/10.3390/app16010217
Oracle. (s. f.). What is an applicant tracking system? https://www.oracle.com/human-capital-management/recruiting/what-is-applicant-tracking-system/Oracle
Psico Smart. (2024). Incorporating artificial intelligence in candidate experience management. Retrieved from https://psicosmart.com/en/blogs/blog-incorporatingartificial-intelligence-in-candidate-experiencemanagement10305#:~:text=According%20to%20a%20study %20by,per%2Dhire%20by%2025%25.
Purcell, K. (2025, 14 de julio). 2025 Applicant Tracking System (ATS) Usage Report: Key Shifts and Strategies for Job Seekers. Jobscan. https://www.jobscan.co/blog/fortune-500-use-applicant-tracking-systems/
Salazar, E. (2023). Análisis del impacto de Inteligencias Artificiales en la experiencia de los usuarios universitarios: un enfoque en la minería de datos y el análisis de sentimiento. Caso de estudio: ChatGPT. Pontificia Universidad Católica del Ecuador.
Saliu, S. (2025). The algorithmic turn in talent acquisition: A critical analysis of AI-mediated recruitment technologies. Iconic Research and Engineering Journals, 8(9), 755–767.
Sangwan, B., Sangeeta, & Diksha. (2025). Intelligent Resume Analyzer. Documento presentado en la conferencia IRDO 2025.
Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3/4), 591–611.
Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25. https://doi.org/10.2307/2331554
Sun, H., Lin, H., Yan, H., Song, Y., Gao, X., & Yan, R. (2025). MockLLM: A multi-agent behavior collaboration framework for online job seeking and recruiting. In Proceedings of KDD ’25. https://doi.org/10.1145/3711896.3737051
Suraj, M., Aruna Kumari, K., & Binila B. Chandran. (2019). A descriptive study on applicant tracking system: Automation software for recruitment and selection. International Journal of Research and Analytical Reviews, 6(1), 223–230.
Tyagi, S., Anuj, Qian, W., Xie, J., & Andrews, R. (2024). Enhancing gender equity in resume job matching via debiasing-assisted deep generative model and gender-weighted sampling. International Journal of Information Management Data Insights, 4, 100283. https://doi.org/10.1016/j.jjimei.2024.100283
Vyshotravk, D. (2025). Smart application for recruiting based on natural language processing methods, transformer models and siamese neural network architecture. Volume 17(Issue 5), 156.
Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Cristian Sebastian Luna Perez, Juan Carlos Santillán Lima
This work is licensed under a Creative Commons Attribution 4.0 International License.
The authors retain the moral and patrimonial rights of their works. They only give to the magazine Tesla Revista Científica the right to the first publication of this. Since Tesla Revista Científica is an open access publication, readers can fully or partially reproduce its content as long as they properly credit the corresponding authors and the journal itself. Tesla Revista Científica undertakes not to make commercial use of the texts it receives and/or publishes.
Our journal is governed by the international policies SHERPA/RoMEO: Green journal: They allow the self-archiving of both the pre-print (draft of a paper) and the post-print (the version corrected and reviewed by peers) and even the final version ( layout as it will be published in the journal).
See also "Copyright and licences".
