AI is not responsible for the rise if bad writing assessment, but it is promising to provide the next step in that little journey to hell. Let me offer a quick recap of bad writing assessment, much of which I experienced first hand here in Pennsylvania. The Keystone State a few decades back launched the Pennsylvania System of School Assessment (PSSA) writing assessment. Assessing those essays from across the state was, at first, a pretty interesting undertaking-- the state selected a whole boatload of teachers, brought them to a hotel, and had them spend a weekend scoring those assessments.
absolutely brilliant post! And on the topic of automated scoring of essays, I wrote this almost exactly 8 years ago regarding the RI state assessment (then the PARCC):
Unfortunately for us in RI, we have a Strategic Plan for Public Education 2015-2020 that emphasizes “innovative” teaching/learning via digital programming. In addition, our Commissioner of Education, Ken Wagner, is enthusiastically committed to the full monty of digitized learning, going so far as to praise the use of automated scoring for essay responses on the state assessment, the PARCC (Partnership for Assessment of Readiness for College and Careers). Commissioner Wagner’s remarks on this topic can be found on the video of the April 5, 2016 meeting of the RI Council on Elementary and Secondary Education.
When Wagner was making his glowing comments about how great automated scoring is, and how fortunate we are in RI to be able to participate in this for scoring the PARCC, he declared that scoring by algorithm is more efficient and just as good as if not better than scoring by teachers. Hasn’t Wagner read the accounts of college graduates without teaching degrees or experience being recruited on Craigslist by Pear$on to score the tests? The quantity of tests they are expected to score and the rigid criteria they are expected to use cannot possibly result in valid scores for students. So maybe, yes, it doesn’t matter if you switch to computer scoring and don’t have to bother with providing low wages and no benefits to temporary workers who have no idea what they’re doing. Please see this post by Leonie Haimson for further insight on automated scoring. https://www.washingtonpost.com/news/answer-sheet/wp/2016/05/05/should-you-trust-a-computer-to-grade-your-childs-writing-on-common-core-tests/?postshare=841462480189631
...
There was another comment by the Commissioner that was jaw-dropping. When the high school student on the Council described a loss of instructional time due to insufficient technology in some schools during test administration, so that schedules are interrupted for weeks, Wagner insisted that it’s necessary to get all schools to use the online version, rather than the paper and pencil version of the PARCC, as soon as possible. He insisted that this is important not only for the testing, but for an underlying instructional purpose. He stated:
“We can’t think about student engagement unless we have a serious strategy around digital learning.”
I can’t think of a more misguided understanding of student engagement, can you?
I absolutely agree with this post. Evaluations should not be done by AI if the real goal is to teach children to think independently & critically with real intelligence. This is also one of the reasons that I don’t use AI on my SubStack publications: I want a real connection with real people.
Way back in 1969 I was due to graduate with a BS but I had left off a writing requirement. Luckily there was a test to take in lieu of a class, so I took the test. Given four prompts to write upon, I chose the easiest (for me, of course). Then I wrote a short introduction, followed by a longer "body" paragraph, and then a short conclusion. I made very effort to write as an eighth-grader would. No long sentences, no big words, etc.
I was just about to turn in my Blue Book, when I stopped my self, went back and wrote an outline of what I had written on the inside cover. (English teachers like outlines at the time.) The next week I found my name on a long computer printout with a capital P next to it, ta da!
That we would accept eighth-grade writing from college graduates was depressing, but I was gaming the system somewhat, so I surely couldn't play and ethical cards, now could I?
Oh, I ended up being a Chief Reader for and ETS program, reading essays written by teachers trying to add sciences to their list of subjects they were qualified to teach. The whole team, of course, was of science nerds (Me, chemistry.) and we were dismayed at the lack of redeeming qualities in the writings of purported classroom teachers, to the point we would leave a reading weekend depressed. We even wrote to the ETS higher ups regarding the abysmal writing quality.
absolutely brilliant post! And on the topic of automated scoring of essays, I wrote this almost exactly 8 years ago regarding the RI state assessment (then the PARCC):
Unfortunately for us in RI, we have a Strategic Plan for Public Education 2015-2020 that emphasizes “innovative” teaching/learning via digital programming. In addition, our Commissioner of Education, Ken Wagner, is enthusiastically committed to the full monty of digitized learning, going so far as to praise the use of automated scoring for essay responses on the state assessment, the PARCC (Partnership for Assessment of Readiness for College and Careers). Commissioner Wagner’s remarks on this topic can be found on the video of the April 5, 2016 meeting of the RI Council on Elementary and Secondary Education.
When Wagner was making his glowing comments about how great automated scoring is, and how fortunate we are in RI to be able to participate in this for scoring the PARCC, he declared that scoring by algorithm is more efficient and just as good as if not better than scoring by teachers. Hasn’t Wagner read the accounts of college graduates without teaching degrees or experience being recruited on Craigslist by Pear$on to score the tests? The quantity of tests they are expected to score and the rigid criteria they are expected to use cannot possibly result in valid scores for students. So maybe, yes, it doesn’t matter if you switch to computer scoring and don’t have to bother with providing low wages and no benefits to temporary workers who have no idea what they’re doing. Please see this post by Leonie Haimson for further insight on automated scoring. https://www.washingtonpost.com/news/answer-sheet/wp/2016/05/05/should-you-trust-a-computer-to-grade-your-childs-writing-on-common-core-tests/?postshare=841462480189631
...
There was another comment by the Commissioner that was jaw-dropping. When the high school student on the Council described a loss of instructional time due to insufficient technology in some schools during test administration, so that schedules are interrupted for weeks, Wagner insisted that it’s necessary to get all schools to use the online version, rather than the paper and pencil version of the PARCC, as soon as possible. He insisted that this is important not only for the testing, but for an underlying instructional purpose. He stated:
“We can’t think about student engagement unless we have a serious strategy around digital learning.”
I can’t think of a more misguided understanding of student engagement, can you?
https://resseger.wordpress.com/2016/05/27/story-telling-species/
I absolutely agree with this post. Evaluations should not be done by AI if the real goal is to teach children to think independently & critically with real intelligence. This is also one of the reasons that I don’t use AI on my SubStack publications: I want a real connection with real people.
This is an important essay Peter, you make an important point re how we already ask humans to grade like machines. Thanks for sharing this.
Way back in 1969 I was due to graduate with a BS but I had left off a writing requirement. Luckily there was a test to take in lieu of a class, so I took the test. Given four prompts to write upon, I chose the easiest (for me, of course). Then I wrote a short introduction, followed by a longer "body" paragraph, and then a short conclusion. I made very effort to write as an eighth-grader would. No long sentences, no big words, etc.
I was just about to turn in my Blue Book, when I stopped my self, went back and wrote an outline of what I had written on the inside cover. (English teachers like outlines at the time.) The next week I found my name on a long computer printout with a capital P next to it, ta da!
That we would accept eighth-grade writing from college graduates was depressing, but I was gaming the system somewhat, so I surely couldn't play and ethical cards, now could I?
Oh, I ended up being a Chief Reader for and ETS program, reading essays written by teachers trying to add sciences to their list of subjects they were qualified to teach. The whole team, of course, was of science nerds (Me, chemistry.) and we were dismayed at the lack of redeeming qualities in the writings of purported classroom teachers, to the point we would leave a reading weekend depressed. We even wrote to the ETS higher ups regarding the abysmal writing quality.
The ending of this post is absolutely jarring and thought-provoking.
From dystopias of our own making, Good Lord, deliver us.
The old article by a scorer may be this one:
https://monthlyreview.org/2010/12/01/the-loneliness-of-the-long-distance-test-scorer/
Thank you. It's not the same one, but it is just as good.