Originally published September 29, 2009 at http://artofeducating.wordpress.com
My tech-savvy colleague Bill Ferriter, The Tempered Radical, just put up a good post about a problem with data – “My (Current) Data Nightmare.” He wants to use data to inform his reading instruction (I say inform not drive because we’re professionals, not cows; data are not whips, they’re tools). His district is using the Blue Diamond Instructional Management Suite to give formative reading assessments every three weeks or so. Bill’s issue is that the Blue Diamond assessment reports break the questions and student responses down by North Carolina’s English Language Arts objectives (or numbered indicators), which are only slightly less overarching and vague than the top-level curriculum goals. He wants to pinpoint which specific reading skills his students need work on. I have a couple ideas to offer Bill and anyone else struggling with this problem.
Even when you have a standardized assessment reliable enough to make valid inferences, I think you still have to analyze assessment results on an item-by-item basis. Bill suggested tagging questions by discrete skill related to the state objective being targeted. That’s nice, but it still wouldn’t tell you what you need to know. A student might have mastered a specific reading skill, but missed the question for a completely unrelated reason. Maybe the student didn’t read the passage carefully enough, maybe she didn’t know an important vocabulary word, maybe the question itself was vague, or maybe she didn’t have some vital piece of background knowledge. The point is that you won’t know until you examine the questions themselves and reflect on patterns using knowledge of your students and the content. Skill tags on each question might be helpful, but chances are, at the very least, you’re going to need to double-check a few questions that a class missed at high percentages. I did item-by-item analyses (and better yet, had students do their own item-by-item analyses) of standardized interim reading assessments given by Charlotte-Mecklenburg Schools in North Carolina. Though I have never been a supporter of that district’s tendency toward over-assessment, I was able to use the assessments in a formative manner to learn about what my students knew and what they needed to know.
Take a look at my sample: Teacher Interim Analysis.
After a year or so of these interim assessments, I quickly realized that the indicated objectives were irrelevant to why students got the questions right or wrong, so I stopped bothering with the objectives. Notice in the above sample that I tagged the questions myself by type of question, my theory at the time being that students were missing questions because of the nature of the questions themselves (INF = inference, LIT = literary, BAS = basic, and VOC = vocabulary in context). The highlighted cells indicate percentages that I used to identify problem areas for particular classes.
The point is that an analysis like this really helped me target my instruction. On almost every district multiple-choice assessment, I also had my students do an item analysis. They learned from identifying patterns and reflecting on those patterns, and I learned even more from their analyses. Take a look at my sample: Student Interim Analysis. Her reflection is exactly what I was designing for. Doing the analysis gave her a chance to get some points for “corrections” (we were required to count these assessments as significant test grades) and it gave me a chance embed test-taking and reading strategies for timed passages into the instructional flow.
I suspect Bill’s Blue Diamond three-week assessments are not as extensive as my interim assessment examples, but item-by-item analyses are just as useful with shorter, quicker assessments. You can even set it up so students track patterns and/or progress from mini-assessment to mini-assessment.
See my sample: Student Mini Analysis.
In his blog, Bill said he wants concrete, tangible evidence to 1) identify students in need of enrichment and remediation, and 2) identify and share effective instructional practices with other teachers. Because reading skills are so interrelated, I identified students in need of enrichment and remediation by overall performance percentages or when particular issues cropped up in their assessment analyses and/or reflections. The only way to identify and share effective instructional practices with other teachers is to get feedback from students on instructional processes using reflections, student self-assessments, surveys, etc. Blue Diamond and other canned programs are not going to be able to do that. The best programs can do is identify teachers with the highest percentage passing rates. Teachers with good strategies might be identified by objective, but just as with analyzing student results, those objectives don’t mean much. Did that teacher who was successful with 5.01 teach a particular skill related to the objective? Did his success have to do with curricular choices, a style of group work, a particular activity, or an overall focused classroom atmosphere? We can’t know until we dig into the data ourselves and ask.
Does all this analysis and reflection take time? Of course, but not that much, and even less for you if you can get your computerized or web-based program to print out assessment results by item and percentage of students who got it correct or incorrect. The time is spent in worthy analysis and reflection. In the examples I’ve shared above, the assessment becomes part of learning and truly formative. Besides, I have it my mind that it is our job – as the professionals – to make the important inferences from and interpretations of the data. Data and assessments, whether they come from Blue Diamond or a teacher-made test, can inform, but they cannot create learning. That’s what teachers do.
For a great formative assessment article that highlights the importance item analysis, see Stephen and Jan Chappuis’ piece, “The Best Value in Formative Assessment,” in the Dec 2007/Jan 2008 edition of Educational Leadership.