Domain-Specific Hybrid BERT based System for Automatic Short Answer Grading

  • Jai Garg
  • Jatin Papreja
  • Kumar Apurva
  • Dr. Goonjan Jain

Abstract

Effective and efficient grading has been recognized as an important issue in any educational institution. In this study, a grading system involving BERT for Automatic Short Answer Grading (ASAG) is proposed. A BERT Regressor model is fine- tuned using a domain-specific ASAG dataset to achieve a baseline performance. In order to improve the final grading performance, an effective strategy is proposed involving careful integration of BERT Regressor model with Semantic Text Similarity. A set of experiments is conducted to test the performance of the proposed method. Two performance metrics namely: Pearson’s Correlation Coefficient and Root Mean Squared Error are used for evaluation purposes. The results obtained highlights the usefulness of proposed system for domain specific ASAG tasks in real life.

Keywords: Automatic Short Answer Grading (ASAG), Se- mantic Text Similarity, Key-Response Similarity, Bidirectional Encoder Representation from Transformers (BERT), Masked and Permuted Pre-training for Language Understanding (MPNet)

Downloads

Download data is not yet available.

References

[1] Y. Oksuz and E. Demir, “Comparison of open ended questions and multiple choice tests in terms of psychometric features and student performance,” Hacettepe Univ. J. Edu., vol. 34, no. 1, pp. 259–282, 2019, doi: 10.16986/HUJE.2018040550.
[2] M. Mohler, R. Bunescu, and R. Mihalcea, “Learning to grade short answer questions using semantic similarity measures and dependency graph alignments,” Proc. 49th Annu. Meeting Assoc. Comput. Linguis- tics, Hum. Lang. Technol., 2011, pp. 752– 762.
[3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2018, arXiv:1810.04805. [Online]. Available: http://arxiv.org/abs/1810.04805
[4] E. B. Page, “The imminence of grading essays by computers,” Phi Delta Kappan, vol. 47, no. 5, pp. 238–243, 1966.
[5] S. Burrows, I. Gurevych, and B. Stein, “The eras and trends of automatic short answer grading,” Int. J. Artif. Intell. Edu., vol. 25, no. 1, pp. 60–117, Mar. 2015, doi: 10.1007/s40593-014-0026-8.
[6] L. Ramachandran, J. Cheng, and P. Foltz, “Identifying patterns for short answer scoring using graph-based lexico-semantic text matching,” Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, 2015.
[7] M. A. Sultan, C. Salazar, and T. Sumner, “Fast and easy short answer grading with high accuracy,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics, Human Lang. Technol., 2016, pp. 1070–1075, doi: 10.18653/v1/N16-1123.
[8] W. Zichao, S. L. Andrew, E. W. Andrew, P. Grimaldi, and R. G. Baraniuk, “A meta-learning augmented bidirectional transformer model for automatic short answer grading,” in Proc. 12th Int. Conf. Educ. Data Mining (EDM), 2019, pp. 1–4.
[9] C. Sung, T. Dhamecha, S. Saha, T. Ma, V. Reddy, and R. Arora, “Pretraining BERT on domain resources for short answer grading,” in Proc. Conf. Empirical Methods Natural Lang. Process. 9th Int. Joint Conf. Natural Lang. Process. (EMNLP-IJCNLP), 2019, pp. 6073–6077, doi: 10.18653/v1/D19-1628.
Statistics
0 Views | 0 Downloads
How to Cite
Garg, J., Papreja, J., Apurva, K., & Jain, D. G. (2022). Domain-Specific Hybrid BERT based System for Automatic Short Answer Grading. Asian Journal For Convergence In Technology (AJCT) ISSN -2350-1146, 8(2), 39-44. https://doi.org/10.33130/AJCT.2022v08i02.09