1000000%. I have been saying this, in my former capacity as SLO assessment coordinator (yawn), for about 10 years. Coming up with ways to try to figure out what is inside a person's head is perhaps the most interesting and challenging part of teaching - because assessments are performances. The trick is to come up with a performance that (1) can only be done by someone who knows what you want them to know, and (2) that gives you a good idea of how that person is processing the knowledge you're trying to get them to cultivate.
So, not hard to come up with an assessment to see if someone can do the splits: you just say, Do the splits! and if they can do the splits, well, they can do the splits.
But much harder to know whether your discussion of confirmation bias, along with analysis of various examples of same, is helping someone to scrutinize their own thinking. Any task you come up with, in this category, is going to involve putting the student on the spot in some way - to perform. And people approach performances with all kinds of other baggage. Some people are great at figuring out just exactly what you're looking for, in very much the same way as an LLM. Some overthink things. Some are shy, some think you're asking about something else....
The thing is, again: Assessment is mostly a *performance.* It is NOT a measurement. You can't measure knowledge or skill or attitudes. When I think of the oceans of money and time we've pissed away on this completely worthless endeavour, I could weep.
1000000%. I have been saying this, in my former capacity as SLO assessment coordinator (yawn), for about 10 years. Coming up with ways to try to figure out what is inside a person's head is perhaps the most interesting and challenging part of teaching - because assessments are performances. The trick is to come up with a performance that (1) can only be done by someone who knows what you want them to know, and (2) that gives you a good idea of how that person is processing the knowledge you're trying to get them to cultivate.
So, not hard to come up with an assessment to see if someone can do the splits: you just say, Do the splits! and if they can do the splits, well, they can do the splits.
But much harder to know whether your discussion of confirmation bias, along with analysis of various examples of same, is helping someone to scrutinize their own thinking. Any task you come up with, in this category, is going to involve putting the student on the spot in some way - to perform. And people approach performances with all kinds of other baggage. Some people are great at figuring out just exactly what you're looking for, in very much the same way as an LLM. Some overthink things. Some are shy, some think you're asking about something else....
The thing is, again: Assessment is mostly a *performance.* It is NOT a measurement. You can't measure knowledge or skill or attitudes. When I think of the oceans of money and time we've pissed away on this completely worthless endeavour, I could weep.