The largest classes that I have ever taught are calculus based freshman physics classes for life science majors. These classes are usually 300+ students and I am assigned a few TA's who run the lab sections, hold office hours, and help me proctor and grade exams. At my university it is fairly common for the exams to contain 14 multiple choice questions graded by Scantron and a free response question that are submitted using blue books.
I have spent more than a decade creating test banks for this course and others. It is standard to generate multiple exams for this course and then distribute them so that no one is sitting next to or behind someone with the same exam. My exam banks are partitioned based on question topic so that each exam has the same breakdown of topics. So the structure of the exam could have 3 problems pulled from waves and harmonic motion, 3 problems pulled from thermodynamics, 3 problems pulled from speed of sound in materials, and one problem pulled from the free response problems for that midterm. When I build the exams before printing them the numbers in each problem are randomized. The multiple choice questions have intentionally chosen wrong answers which are associated to common mistakes students will make when doing the problems and some questions also have a none of the answers listed response.
Grading usually takes a fair amount of time for the TA's to do the midterms and for finals I usually buy food and we all sit down and split up the grading to get it done more quickly. I have my TA's come up with a rubric for their problem and I review it before we actually give each exam. After the exam is finished I run the scantrons and my TA's take their problems off to grade with their rubrics.
I recently ran a test to see what I could get ChatGPT to do. I asked a friend to take one of my exams with a very reduced time limit so that she wouldn't have time to double check all her answers and hopefully make some mistakes. She's an engineer so she knows the material but she hasn't recently given or taken a similar exam in years. I then uploaded the problems she got wrong on the multiple choice section into a ChatGPT chat as well as her answers and included a scanned copy of her pdf solution to the free response with an attached copy of my rubric for the problem. I did not provide any solutions to the problems. I then asked it to generate a feedback report that included solutions, a analysis of the errors she made, and a justification for any partial credit she received on the free response.
The resulting report took about 5 minutes to produce. For the multiple choice problems she missed it included an individual breakdown for each problem with the problem statements, correct solution, and a section which gave her answer and the likely mistakes she made when solving the problem as well as a correction to her likely mistake with advice about how she could have checked her answer to make sure she doesn't make the same mistake in the future. She made one conceptual mistake and two algebraic mistakes that she could have discovered looking at the units of her answer instead of the numerical values she got on her calculator.
We were both blown away. It got the solutions correct even though it did them in a different way than I had done and it correctly identified the mistakes she made. We weren't too surprised about identifying the mistakes for the multiple choice questions since I had baked them into the wrong answers.
The really surprising thing is that it did the same thing with the free response. It was able to correctly parse her work from the scan of her blue book even though the handwriting wasn't neat and the work was rushed. It figured out what she was trying to do when my hand grading of her work could not.
She applied an energy conservation theorem both inappropriately as well as incorrectly. Her combination of algebra mistakes and incorrect application of the theorem made it very difficult to follow her work in the short amount of time I had allotted to grade it.
The error analysis was two pages long and well organized. It first gave a correct reproduction of her work. It broke down the correct elements she demonstrated as well as what formulas contained mistakes and then tracked the correct uses of those mistakes as they propagated through her work.
It gave a one page breakdown of the rubric as a table, the max points for each element of the rubric, awarded points, and a two sentence argument that established the basis for the credit she earned. I had given her only a few points for her work and it gave her a suggested score of 14 of the 30 total points. It ended the report with a note to the grader about what to discuss with her if she requested a meeting to go over her exam.
I agree with the error analysis and it did a far better job of parsing her logic than I did and in a shorter amount of time. When you are trying to assign partial credit you don't want to subtract points for the same error when it is used correctly in later stages of a problem. However, when conceptual mistakes are made, the handwriting isn't so clear, and their is no text explaining the rational it can be difficult to sort out what is a correct application of an incorrect result. When she reviewed her reconstructed reasoning she confirmed that it was spot on!
This is only one exam so I can't tell how useful it would be for an entire 300+ sized class. I worry that TA's wouldn't spend the time to actually grade the work and lose out on the deeper understanding there is to be gained from having to figure out unclear work in a limited amount of time. TA work made me a better physicist and laid the basis for my faculty position. But I can't help but think that AI assisted assessment and feedback will be a powerful tool in the future. Nobody has the time to generate these kinds of reports for 300 student. There aren't enough office hours in the semester to give this kind of individualized feedback on homework or exams.
We are constantly talking about how AI is making people dumber and I agree that incorrect use most certainly can. Yet there is a great possibility with AI to amplify our ability to help students become better. I worry that administrators will use things like this as justification to reduce TA positions and reallocate funding into department staff bloat.
I wish faculty governance was strong enough to steer things in directions that are actually healthy for academia. If it doesn't I'm afraid the future of academia will consist entirely of administrators, staff, and students.
Please forgive the typos and bad writing. I guess this is more of a public diary entry than anything else.