At The Heart of Education, An Insoluble Problem
From Forbes.com
How do you measure what is going on inside a person’s mind?
Designing assessments is one of the most challenging pieces of education (and vast numbers of classroom teachers are required to do it on their own). There are numerous obstacles on the path to assessments that tell us what students actually understand.
If the assessment task is too simple, it becomes hard to distinguish between understanding and mimicry. We can teach a barely verbal infant, a parrot, and an AI bot to correctly complete the phrase “two plus two equals...” but that doesn’t mean they actually understand what the phrase means.
Even assessments of more complex learning can result in regurgitation rather than actual comprehension. Every student has been in at least one class that could be passed by simply spitting back to the teacher what she had delivered to the class.
The clearer a teacher’s expectations are, the easier it can be to simply game the system. But if the teacher’s expectations are not clear, she will have trouble deciding whether the students failed to comprehend the material or to understand what the assessment requires. Finding the sweet spot is difficult.
More complex assessments also run into the question of how much of the students’ troubles are a failure in understanding and how much are trouble in conveying that understanding. If you have asked for an essay answer, did the student come up short because of a lack of understanding, or because of a lack of writing skill? Or did the question itself include some cultural bias that interfered with student understanding of the task?
The common assumption is that if the student really knows the material, she’ll have no trouble expressing her understanding. But how many adults have had the “I know what I want to say, but I can’t quite figure out how to say it so you’ll understand” experience? Many persons have trouble translating understanding into communication, and many other persons can communicate quite effectively about things they do not understand at all.
1000000%. I have been saying this, in my former capacity as SLO assessment coordinator (yawn), for about 10 years. Coming up with ways to try to figure out what is inside a person's head is perhaps the most interesting and challenging part of teaching - because assessments are performances. The trick is to come up with a performance that (1) can only be done by someone who knows what you want them to know, and (2) that gives you a good idea of how that person is processing the knowledge you're trying to get them to cultivate.
So, not hard to come up with an assessment to see if someone can do the splits: you just say, Do the splits! and if they can do the splits, well, they can do the splits.
But much harder to know whether your discussion of confirmation bias, along with analysis of various examples of same, is helping someone to scrutinize their own thinking. Any task you come up with, in this category, is going to involve putting the student on the spot in some way - to perform. And people approach performances with all kinds of other baggage. Some people are great at figuring out just exactly what you're looking for, in very much the same way as an LLM. Some overthink things. Some are shy, some think you're asking about something else....
The thing is, again: Assessment is mostly a *performance.* It is NOT a measurement. You can't measure knowledge or skill or attitudes. When I think of the oceans of money and time we've pissed away on this completely worthless endeavour, I could weep.