Background

Correcting reports are a very time consuming task in courses that is challenged by limited resources. Historically, we have a lot of data on carefully corrected reports using rubric evaluation criterias.

This project will explore the use of large language models (LLMs) and in particular recent developments enhancing LLMs with multimodality (i.e., image comprehension) [2] to enable an automated report evaluation system.

The data for the project will be historically evaluated 02450 Introduction to Machine Learning and Data Mining reports containing in the order of 6 years of two semesters with each about 200 groups performing two reports. Report 1 contains 23 evaluation criterias on a likert scale from 0 to 4 whereas report 2 contains 17 evaluation criterias. Additionally, an overall evaluation of the report quality is also provided that is used to assess the students performance in the course.

Objective(s)

i) Accurately predict rubric ratings from pdf files of the students > reports.

ii) Accurately predict overall report assessment.

iii) Provide student feedback based on produced evaluations.

The project will use the embeddings [3] produced by a large lange model in conjunction with a multi-output deep (likely transformer based) ordinal regression framework [1] to predict both the overall score as well as the individual rubric assessment criterias.

Requirements

Need to have:

  • Basic understanding of Deep Learning and Machine Learning

  • Experience programming in PyTorch

Nice to have:

  • Completed the course 02450.

  • TA experience in 02450.

Maximum number of students

3-4 student max pr. group up to three groups

Supervisors

Morten Mørup

Rasmus Aagaard

Tue Herlau ?

Contact information

Name: Morten Mørup

Address: Richard Petersens Plads, bld 321 room 118

E-mail: mmor@dtu.dk

Phone nr. Phone nr.+45 4525 3900

References

[[1] Chu, Wei, Zoubin Ghahramani, and Christopher KI Williams. "Gaussian processes for ordinal regression." Journal of machine learning research 6.7 (2005).]{.mark}

[[2] GPT-4 Technical Report, [https://arxiv.org/abs/2303.08774]{.underline}]{.mark}

[3] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, [https://arxiv.org/abs/1908.10084]{.underline}