THE GOSPEL OF BAD DATA: BATTLE ADVICE FOR INDIVIDUAL EDUCATORS AND INSTITUTIONS

In the accountability era, data have become a weapon; opposing groups lob data back and forth to discredit each other and create support for pre-determined opinion and philosophy.  Truth be told – especially if you don’t have much concern about research methodology (maybe only that it is scientifically based, whatever relative definition of that term is conveniently picked) – there is data out there for you.  Especially in the murky waters of what works in education, data can be found, generated, or – when necessary scientifically fabricated – to prop up almost anyone’s point of view.

These days, what gets accepted as fact is almost never a matter of how rigorous the research process is or how valid the data are.  As long as you use the word data in some form or fashion, you can create generally accepted fact.  You just need to get your message out in the media in a way that can be picked up by the general public, preferably in a big box movie or on a friendly, graphic website.  Just a couple weeks ago, the National Council on Teacher Quality (NCTQ), in partnership with U.S. News & World Report, published a damning report of traditional teacher education in the U.S.  The methods used to gather data bordered on the ridiculous and the organization’s bias against traditional teacher education is obvious in everything from the language of the report to the make-up of its board.  Nevertheless, the NCTQ Teacher Prep Report made headlines and people listened to the message.  The report was released on Tuesday, June 11th.  As soon as I opened msn.com Tuesday morning I saw reference to the NCTQ results; I heard related reporting on NPR on my way in every morning from Tuesday on; Indeed, in my first meeting Tuesday morning one of my colleagues mentioned an article published in the Chronicle of Higher Education, which headlined “’An Industry of Mediocrity’: Study Criticizes Teacher-Education Programs.”  Snappy phrases like “an industry of mediocrity” are just too juicy for even a respected media outlet not to recycle and proliferate.    

As Chair of Teacher Education at Newberry College, I do have a dog in this fight, but philosophically, when it comes to teacher preparation, I fall on both sides of the fence.  I have prepared teachers through traditional means but also through alternative means.  I am not opposed to Teach for America (though I’d like to find more than a handful of TFA candidates who have spent more than the required two years in the classroom before moving on to bigger and better things).  It’s astounding to me that no one seems to acknowledge the fact that venture philanthropies and for-profit education businesses have much to gain from the dissolution of traditional avenues and governmental gateways in teacher preparation and licensure, as well as school funding and governance.  Who are we kidding?     

But this is not a post about teacher education.  What interests me is how a vague reference to data takes opinion and makes it into gospel, gospel that cannot be denied.  Any teacher education program sounding off against the NCTQ report – especially those labeled with NCTQ’s handy “consumer alert” – sound defensive and thus culpable.  To attack NCTQ or NCTQ’s data-gathering methods publicly just implies one is a special interest group out to cover its butt with semantics and excuses.  Institutions are caught between a rock and a hard place.  Respond and reinforce the implication that you are culpable; don’t respond and let what is, at worst, thinly veiled slander become accepted fact.  Public school districts, principals, and teachers in South Carolina and across the nation face this same conundrum.  We know that many of the data systems used to evaluate effectiveness are flawed, in great part because high-stakes, large-scale testing used to determine effectiveness are not reliable indicators of student learning at the individual or classroom levels. 

As a teacher, there were years that I had all the highest scoring students in the school on the state standardized reading test and there were years that my students barely made it past the test to eighth grade.  Every year the students changed, the test changed, the scoring methods changed, the way growth was calculated changed – I was the constant in the classroom.  What I did played only one part in determining my students’ and school’s standardized test results.  But like my school, I would have seemed defensive had I questioned or refused to acknowledge what I know is a flawed system of evaluation. 

Our best bet as educators – when it comes to standardized testing – is to make the best of a bad situation and try to help it improve.  You, the American public, like a number that can be sliced, diced, and statistically compared with a slew of other numbers?  Then sure – let’s try to evaluate my instructional effectiveness as a classroom teacher using standardized tests.  As a data literate instructor I have a pretty good (but not guaranteed) chance of making the game work in my students’ favor while actually teaching them to high levels in real time  But, while I accept a certain amount of the gospel for what others believe it to be, I will advocate for better evaluation and real learning.  I will pay an appropriate amount of attention to data from those tests – which is not as much as many people think – whether the results are positive or negative.  I will share more reliable assessments, useful data, and valid inferences with all the stakeholders in my classroom – students, parents, administrators, and colleagues – all while educating, inspiring, drawing attention to, and involving them in what everyone really wants: a truly fabulous, effective learning environment.      

Public education and teacher education institutions need to do the same.  When it comes to reports and report cards – damning or otherwise – we must respond, but an all-out battle will not make us look any better.  Ignoring the gospel won’t either.  On an institutional level, we must advocate for better evaluation and real learning, and we must pay an appropriate amount of attention to reports and report cards like NCTQ’s – which is not as much as the report writers think – whether our results are positive or negative.  We must develop and share more reliable assessments, useful data, and valid inferences with all our stakeholders – all while educating, inspiring, drawing attention to, and involving them in what everyone really wants: an institution that cares about people and is effective at achieving important outcomes related to their long-term success. 

CHERRY PICKING STANDARDS

Originally published November 25, 2009 at http://artofeducating.wordpress.com

A couple weeks ago I spent a bit of time talking over teachers’ use of classroom standards with Stephen Sawchuk of Education Week.  Standards are a big topic right now, especially as the Common Core State Standards Initiative finalizes national standards for English language arts (ELA) and mathematics.  Almost all 50 states have signed on to adopt and implement whatever standards the National Governors Association (NGA) and Council of Chief State School Officers (CCSSO) develop with very little fuss.  Why is this?  Perhaps states are desperate for federal funding; they are required to sign on in order to compete for current funds.  Perhaps, at this point, we see ELA and mathematics content as uncontroversial.  Perhaps states know that national standards will have to be broad, at least as broad (in the case of ELA) as the standards they are already using.  After all, states will only be required to use 85% of the Common Core Standards, and how different can they really be?  Perhaps states think adopting the Common Core will only require minor shifts in verbiage, not major shifts in systemic thinking.   Perhaps, after all, we recognize the fact that standards have very little, if any, impact on daily classroom instruction.

Yes, I know…teachers have to write the day’s standards on their boards, list them in lesson plans, and address them when preparing students for state standardized assessments.  But that doesn’t mean that teachers actually understand and use standards.  In my experience, what happens is that teachers design great lessons around great content, then cherry pick standards to fit that lesson and content.  In my subject area – English – you can design almost anything and find standards that fit.  Over the last two years, working with my department chair to make sure our department’s classroom tests and tasks assess standards at high levels, I’ve come to believe that teachers design first and look at standards later.

That’s not surprising.  Years ago, while on a Fulbright to the University of Canterbury in New Zealand, I spent a year and a half analyzing the factors influencing teachers’ use of the new national curriculum, English in the New Zealand Curriculum (Ministry of Education, 1994).  I found that the national standards did not impact teachers’ individual pedagogy directly.  Instead, teachers tended to formulate practice and respond to curriculum documents based on the same set of factors:  Experience, Management Purposes, Consciousness of Professional Environment, Teacher Interests, Context Constraints, Students, and Teacher Beliefs.

In a U.S. context high-stakes standardized assessments also play into teachers’ thinking.  At least in tested subjects, we want to address standards because those standards are what is purportedly being assessed on the tests by which we, our students, and our schools are evaluated.  In the back of our minds, however, we know how impossible (and unethical) it is to match our teaching to the questions on a high-stakes test.  Standards themselves – as they should be – tend to be broad and many, too broad and too many for one test to cover or even one teacher in one, 180-day year to teach well.  Teaching/testing mismatches are inevitable.  “There are enormous and typically unrecognized mismatches between what’s tested and what’s taught.  If you look carefully at what the items in standardized achievement tests are actually measuring, you’ll often find that half or more of what’s tested wasn’t even supposed to be taught in a particular district or state” (Popham, 2001).  Deep down teachers know this.  My low-level middle school students rarely missed questions because they didn’t know the standard.  Often, what tripped them up was a vocabulary word in the question itself.

According to the NGA and CCSSO, their common core state standards will be fewer, clearer, and higher.  That’s great, but it’s not the point.  In the state of education right now, standards only inform practice minimally, if at all.  Even if we create a massive national standardized assessment program and all the test-prep workbook and workshop nonsense to go with it (for which I am sure the Common Core partners – Achieve, ACT, and the College Board – see the potential), teachers will still cherry pick.

Standards are targeted only when a teacher first looks at the standards, then designs.  What that takes is a very savvy teacher, one who can analyze the language of the standards, translate them into learning targets, design instruction leading to those targets, and design assessments that reliably verify learning of those targets.  Standards and assessment go hand in hand.  We need teachers who can both deduce learning targets and induce the big picture.  We need teachers who can get their students talking about standards, not teachers who cherry pick from a curriculum document to write standards on the board or in their lesson plan books.  The educational environment right now encourages teacher compliance, not true understanding and creativity, and it is understanding and ingenuity that will make the standards rubber hit the classroom road.  For that what we need is not mandates or legislation, but strong instructional leadership and professional development focused on increasing teachers’ assessment literacy from pre-service to tenure.

References

Ministry of Education.  (1994).  English in the New Zealand curriculum.  Wellington, New Zealand: Learning Media.

Popham, W. J.  (2001).  The truth about testing: An educator’s call to action.  Alexandria, VA: ASCD

NO MORE DATA NOTEBOOKS

Originally published October 25, 2009 at http://artofeducating.wordpress.com

People have been writing to me lately about data notebooks.  Principals ask, “How do you implement them in a school?  How do you get teachers to buy in?”  Teachers ask, “What data should I be looking at, especially if my subject area isn’t tested?”  

No doubt, data notebooks are a sexy idea.  They have potential for getting data out of faculty meetings or the principal’s computer and into classrooms, the only place where deep down we know data can make a difference.  I spent years in North Carolina as a trainer and curriculum writer for the NC Teacher Academy getting teachers to start keeping data notebooks.  My conference sessions on the subject were always packed.  At the time, my ideas about data notebooks were inspired by Steve Edwards, a former school principal who contributed work to Alan Blankstein’s book, Failure Is Not an Option (2004).  In 2004, at a training for NC Teacher Academy trainers, Steve (now President and CEO of Edwards Educational Services, Inc.) talked a little about he used data notebooks with teachers.  In his system, teachers were given and/or assigned what to put in their notebooks.  From what he said I re-envisioned data notebooks as a learning tool for teachers to research the effects of instruction in their classrooms.  For me it was less about putting certain data in front of teachers’ noses, and more about empowering teachers to think critically about student outcomes and develop capacity for gauging and building student understanding.   When we revised NCTA’s data module in 2005 to develop “Using Data to Create Classroom Learning Communities,” the backbone of the weeklong course was having teachers build their own notebooks based on three dimensions: when the data are collected (daily, periodically, or annually), the types of data collected (based on Victoria Bernhardt’s four types of data: student learning, demographic, perception, and process), and the type of teacher thinking involved (based on National Board requirements: description, analysis, and reflection).  I created a Ms. Smith’s Sample Data Notebook based on my own classroom data for use as a model.

Ideally, the process of putting together a data notebook should force teachers to think critically about practice and allow them to make accurate and timely decisions and interventions.  What often happened, however, was that teachers got stuck on making pretty or correct notebooks and filling them up, not asking the questions that led to assessment design and data collection to inform instruction.  When it’s just about collecting for a notebook, valuable time is wasted, yet another useless paperwork requirement comes into being, and most tragically, teachers and whole schools get distracted.  When that happens, data notebooks can actually work to decrease student achievement.  

I left the NC Teacher Academy in 2007.  In my own work, I progressed from physical notebooks to digital notebooks at both the classroom and school levels, experimenting with the ways in which data can and should be shared among teachers, administrators, students, and parents.  I also focused on helping teachers uncover and ask important questions about their practice, students, and curriculum.  When teachers ask me what should go in their data notebooks, I say, “Whatever answers your questions.”  The questions must come first.

What I find, though, is that teachers don’t ask questions.  I don’t know if they used to, but they certainly aren’t now.  Teachers expect to be told what to do.  I believe we can thank a lot of the standardization policies we implemented and continue to implement today for this sheep-like mentality: pacing guides, teacher-proof textbooks, standardized assessment regimes, and all the mandates of a mechanistic bureaucracy.  In 1994, Kevin Harris wrote, “Within a larger process of adopting forms and processes of corporate managerialism, professionals such as teachers are being redefined as straight-out contract employees subject to direct management, and are becoming positioned in such a way that their expertise and professional knowledge is decreasingly called upon with regard to decision-making in areas central to the needs and requirements of those whom they teach and serve.  Teachers are having to leave decision-making in areas such as curriculum and educational goals and purposes to others and instead become efficient managers….”  We are reaping the natural effects of what Michael Apple (2004) calls the deprofessionalization of teachers.

Today’s emphasis on data can either reinforce deprofessionalization or make teachers the masters of their own classrooms (and, as masters, responsible for producing significant student learning).  Truthfully, I hope for even more and we need more these days.  I want teachers to become the masters of their schools, working through collaboration and inquiry to meet the needs of the many children they serve.  Data certainly play a role in that vision, but as servants to teacher inquiry, used to inform and empower instruction, not drive it.  So for now, forget data notebooks.  Think data and assessment literacy, which gives teachers a lens through which to view data, ask questions of it, and use it to improve practice. 

References

Apple, M. W.  (2004).  Ideology and curriculum (3rd ed.).  New York, NY:  RoutledgeFalmer.

Blankstein, A. M.  (2004).  Failure is not an option: six principles that guide student achievement in high-performing schools.  Thousand Oaks, CA: Corwin Press.

Harris, K.  (1994).  Teachers: Constructing the future.  London, England: The Falmer Press.

MAKING STANDARDIZED DATA WORK

Originally published September 29, 2009 at http://artofeducating.wordpress.com

My tech-savvy colleague Bill Ferriter, The Tempered Radical, just put up a good post about a problem with data – “My (Current) Data Nightmare.”  He wants to use data to inform his reading instruction (I say inform not drive because we’re professionals, not cows; data are not whips, they’re tools).  His district is using the Blue Diamond Instructional Management Suite to give formative reading assessments every three weeks or so.  Bill’s issue is that the Blue Diamond assessment reports break the questions and student responses down by North Carolina’s English Language Arts objectives (or numbered indicators), which are only slightly less overarching and vague than the top-level curriculum goals.  He wants to pinpoint which specific reading skills his students need work on.  I have a couple ideas to offer Bill and anyone else struggling with this problem.

Even when you have a standardized assessment reliable enough to make valid inferences, I think you still have to analyze assessment results on an item-by-item basis.   Bill suggested tagging questions by discrete skill related to the state objective being targeted.   That’s nice, but it still wouldn’t tell you what you need to know.  A student might have mastered a specific reading skill, but missed the question for a completely unrelated reason.  Maybe the student didn’t read the passage carefully enough, maybe she didn’t know an important vocabulary word, maybe the question itself was vague, or maybe she didn’t have some vital piece of background knowledge.  The point is that you won’t know until you examine the questions themselves and reflect on patterns using knowledge of your students and the content.  Skill tags on each question might be helpful, but chances are, at the very least, you’re going to need to double-check a few questions that a class missed at high percentages.  I did item-by-item analyses (and better yet, had students do their own item-by-item analyses) of standardized interim reading assessments given by Charlotte-Mecklenburg Schools in North Carolina.  Though I have never been a supporter of that district’s tendency toward over-assessment, I was able to use the assessments in a formative manner to learn about what my students knew and what they needed to know.

Take a look at my sample: Teacher Interim Analysis.

After a year or so of these interim assessments, I quickly realized that the indicated objectives were irrelevant to why students got the questions right or wrong, so I stopped bothering with the objectives.   Notice in the above sample that I tagged the questions myself by type of question, my theory at the time being that students were missing questions because of the nature of the questions themselves (INF = inference, LIT = literary, BAS = basic, and VOC = vocabulary in context).  The highlighted cells indicate percentages that I used to identify problem areas for particular classes.  

The point is that an analysis like this really helped me target my instruction.  On almost every district multiple-choice assessment, I also had my students do an item analysis.  They learned from identifying patterns and reflecting on those patterns, and I learned even more from their analyses.  Take a look at my sample: Student Interim Analysis.   Her reflection is exactly what I was designing for.  Doing the analysis gave her a chance to get some points for “corrections” (we were required to count these assessments as significant test grades) and it gave me a chance embed test-taking and reading strategies for timed passages into the instructional flow.

I suspect Bill’s Blue Diamond three-week assessments are not as extensive as my interim assessment examples, but item-by-item analyses are just as useful with shorter, quicker assessments.  You can even set it up so students track patterns and/or progress from mini-assessment to mini-assessment. 

See my sample: Student Mini Analysis.

In his blog, Bill said he wants concrete, tangible evidence to 1) identify students in need of enrichment and remediation, and 2) identify and share effective instructional practices with other teachers.   Because reading skills are so interrelated, I identified students in need of enrichment and remediation by overall performance percentages or when particular issues cropped up in their assessment analyses and/or reflections.  The only way to identify and share effective instructional practices with other teachers is to get feedback from students on instructional processes using reflections, student self-assessments, surveys, etc.  Blue Diamond and other canned programs are not going to be able to do that.  The best programs can do is identify teachers with the highest percentage passing rates.  Teachers with good strategies might be identified by objective, but just as with analyzing student results, those objectives don’t mean much.  Did that teacher who was successful with 5.01 teach a particular skill related to the objective?  Did his success have to do with curricular choices, a style of group work, a particular activity, or an overall focused classroom atmosphere?  We can’t know until we dig into the data ourselves and ask.    

Does all this analysis and reflection take time?  Of course, but not that much, and even less for you if you can get your computerized or web-based program to print out assessment results by item and percentage of students who got it correct or incorrect.  The time is spent in worthy analysis and reflection.  In the examples I’ve shared above, the assessment becomes part of learning and truly formative.  Besides, I have it my mind that it is our job – as the professionals – to make the important inferences from and interpretations of the data.  Data and assessments, whether they come from Blue Diamond or a teacher-made test, can inform, but they cannot create learning.  That’s what teachers do.

For a great formative assessment article that highlights the importance item analysis, see Stephen and Jan Chappuis’ piece, “The Best Value in Formative Assessment,” in the Dec 2007/Jan 2008 edition of Educational Leadership.