the dangers of formative assessment without agility

Within a teaching context, formative assessment means gathering data about what students do and don’t get, how they’re thinking, etc. for the purpose of guiding ongoing teaching and learning. It’s assessment to improve learning, not to evaluate it. An implication is that assessment is only formative if the information gathered is actually used to inform decision-making by the teacher and/or students.

An intriguing research result is that formative assessment may actually be counterproductive if the teacher doesn’t have adequate strategies for responding to that information. Here’s a quote about that from a paper by Dylan Wiliam:

What is less clear is what exactly constitutes effective classroom assessment. Although the studies cited above indicate that assessment for learning can improve learning, several studies have found conflicting results. For example, in a study of 32 fifth-grade teachers in Germany, Helmke and Schrader (1987) found that teachers who had an accurate knowledge of their students (as measured by the teachers’ ability to predict achievement test scores) were associated with higher levels of achievement only when the teachers also showed a high range of instructional techniques. Students taught by teachers who had a high knowledge of their students’ achievement but lacked a range of instructional techniques actually performed worse than students taught by teachers who did not know their students’ achievement. This study seems to indicate that collecting data if one cannot do anything with it is counterproductive.

Furthermore, even when teachers do manage to use information about student achievement to adjust or individualize their instruction, teachers may lack the ability to do so effectively. For example, in a 20-week study of 33 teachers in elementary and middle schools, Fuchs, Fuchs, Hamlett and Stecker (1991) found that teachers who received feedback on the achievement of students with learning difficulties in their classes made more adjustments to their teaching programs than teachers not given this information. However, the achievement of these students was improved only when this feedback was accompanied by advice from a computerized “expert system”, because the teachers not given the feedback from the expert system tended to re-explain how to do problems with the same algorithms that had led to previous failure.

Source: Wiliam, Dylan. “Keeping Learning on Track: Classroom Assessment and the Regulation of Learning.” In Second Handbook of Mathematics Teaching and Learning. Edited by Frank K Lester. Greenwich, CT: Information Age Publishing, 2007. pp. 10-11. [PDF preprint]

Posted in Educational Research, Learning & Teaching, classroom response systems | 2 Comments

commenting on this blog

I’ve been having difficulty getting comments to stay open on this blog. (I think I’ve got a bad interaction between a couple of WordPress features/plug-ins.) It looks like I’ve got it sorted out, but I’ve thought that before, too.

It is my intention that comments be open to the public. If you want to comment on a post and find you can’t, by all means please drop me a note (ian [at] ianbeatty [dot] com) and I’ll try to fix it. (Again.)

Posted in Blogging | Leave a comment

why are clicker questions hard to create?

I and my colleagues have, since 2005, been running a large research project that involves giving classroom response systems (CRSs, a.k.a. “clickers”) to middle and high school science and math teachers, spending copious time and energy (and consequently money) helping them to use those systems effectively in their teaching, and studying the heck out of their varied CRS learning experiences. (For more about the project, see its web page. Thanks to the National Science Foundation for funding it—your tax dollars at work via grant# TPC-0456124.)

Of the forty-some teachers we’ve worked with to some degree or another, by far the number one difficulty they’ve reported is the challenge of regularly creating effective clicker questions to use in class. The characteristics that make a question “work”—meaning engage students in quality classroom discussion and promote learning—are not obvious, and typical back-of-the-chapter or quiz-type questions will fail miserably. In the project’s professional development meetings, we’ve spent a great deal of time talking about question creation, and I’ve developed various frameworks in an attempt to help make it more science and less art.

This semester, in prepping my own Conceptual Physics class, I’ve run into exactly the same difficulty. “Today I’m teaching topic X, and I need some good questions. Um, ah, hmm…” Not so easy, even with all the frameworks and such.

One flash of insight I had recently is that, at least for me, it’s not really creating questions that’s tough. The hard part is figuring out what I want my students to learn from the class, and casting that in terms of what I want my students to be able to do. I’ve been trying to shift my thinking from “the material” to “the demonstrable, assessable learning outcomes” (cf. The Myth of Coverage).

Once I can articulate what I would like my students to be able to do after the class, it’s generally relatively easy to invent a few good clicker questions. I just formulate a question asking them to do that (in a particular context), and then much of the class activity is me helping them struggle through the process as they learn how. (This is the principle we’ve called “Question-Driven Instruction”, as articulated in Beatty & Gerace 2009 and elsewhere.)

Which all means that when someone says “Creating good clicker questions is hard”, I’m now inclined to hear that as “Thinking in terms of demonstrable student learning outcomes rather than topic coverage is hard.” And I agree. I also think it’s one of the many desperately needed shifts to how we conceive of this whole enterprise we call organized schooling.

I’m not saying that this is the only difficult aspect of creating good questions, but it’s definitely key for me. I’m curious what others think. If you’ve taught with a classroom response system, what do you think? Does that ring true? Do you have any similar or conflicting experiences to share? Comments are open…

Posted in Learning & Teaching, Pedagogy, classroom response systems | 2 Comments

coming soon: theory meets reality

This blog has been dormant for way too long.

Last January, I moved from Massachusetts to North Carolina, and started a new job as a Physics professor. Spring was largely transition, teaching one light course here and making several long trips back north to keep the research project there going. Then came the summer, with a greater-than-usual blitz of travel and urgent work.

This fall, I started here for real. Now I’m really teaching! (And quite a lot of work it is, too.)

I’ve taught before, sort of: lots of labs and discussion sections as a TA, an outdoor leadership program for high school students, short and long teacher professional development programs… but that’s not the same has having responsibility for a full-scale university “lecture” course with ~60 students and 3 contact hours per week.

This fall, I’ve been teaching Conceptual Physics, a general education course with 55-ish students, drawing from all four class years (most heavily from freshmen and seniors) and almost every major on campus except Physics. I have, of course, been using a classroom response system (CRS, a.k.a. “clickers”). I cannot imagine teaching a course even a third this size without it; it would be like teaching without a whiteboard or a data projector. It would like becoming deaf in the classroom.

It has been interesting to see how all the pedagogical theory that I and my colleagues have been developing has fared. It isn’t easy! I’m encountering many of the same difficulties that the high school teachers in our project have voiced—problems I’ve tried to help them resolve with all kinds of sage advice.

I am finding, of course, that it isn’t quite as easy as I’d thought. (I’m envisioning many of our teachers nodding with a small smile of vindication, and thinking “See?” Touché.) It’s not so much that I’ve been wrong, as that I’m seeing new dimensions and nuances to the problems and the solutions. In fact, having to go through many of the same CRS learning issues as my teachers is really quite instructional.

I’ll try to document some of those insights in forthcoming blog posts. One question that I’m wrestling with, however, is this: just how much should I “let it all hang out” in a public forum that my students might quite possibly discover?

Maybe I shouldn’t worry about that. I find that my faith in my basic pedagogical principles and outlook are being strengthened, not weakened, by the experience. If anything, I’m entertaining even more radical thoughts about how we can re-envision the educational enterprise. (That might scare some of you who know me well.)

Anyway, the next post will focus on my latest insight about what makes creating good CRS questions difficult, at least for me. Hint: It’s not actually about the questions, though it initially appears that way.

Posted in Blogging, Learning & Teaching, Me, Physics Education Research, classroom response systems | Leave a comment

new type of cloud discovered?

From the “just when you think you’ve got it all figured out” department:

Posted in Science | Leave a comment