(I still think I’ve got too many fine-grained and overlapping standards, though. That does cause headaches.)

One thing I’ve realized while developing assessments for this first “unit” of the course — kinematics and vectors — is that my choice of standards can make assessment harder or easier. Some specific realizations:

• Too many standards makes it hard for me to adequately assess, and re-assess, them all. Fewer is better (though too few loses the laser-sharp-feedback quality of SBG).

• Standards that are too “basic”, and which are necessary steps towards the harder ones, aren’t necessary to articulate as standards on their own; students must learn to do them anyway in order to do the higher-level ones. Having them in the list simply clogs up assessments. Example: I can draw or interpret motion diagrams (strobe diagrams).

• Standards that don’t “factor” cleanly are difficult to assess and give separate mastery ratings for. Example: I can use 2D/3D constant-acceleration kinematics (graphical analysis and/or formulae) to analyze an object’s motion, working with numbers or variables and it’s too-close cousin, I can use the projectile motion model to analyze physical situations. I can articulate a distinction between these two — make the second be about recognizing the independence of the two coordinates and the acceleration in each, and the first be about “doing” the subsequent kinematics — but it’s awkward and unclear.

• Some really seem to belong as part of a larger standard, not hanging out on their own. Example: I can determine or reason about an object’s instantaneous acceleration.

So now I’m rewriting my unit 2 standards. (With fewer standards in later units, I’ll have to add some weighting factors to avoid overly-counting kinematics in the final grade.)

Why? For one thing, I’m already a full day behind my planned schedule. Ugh. 50 minutes is so short! Something’s gotta give. I’m feeling the voices of 100+ standards screaming at me to assess, reassess, reassess!

For another, I think I’ve been so concerned about my SBG implementation that I’ve lost the forest for the trees. I forgot to introduce students to what physics is all about and why it’s worth studying, beyond a 3-minute definition (okay, three definitions, including “Whatever physicists feel like studying”). Wait, what’s the point of all these kinematical vector thingies?

You’d never know from my teaching that I recently read Perkins’ Making Learning Whole.

Pondering what to do on Wednesday (and thereafter) to save this course without looking incompetent and losing students’ confidence…

• Teachers Matter (NYT)

It’s not “the system” or “the curriculum” or “the standardized tests” that we should be paying attention to, except insofar as they get in the way of good teachers trying to do their thing.

Some initial thoughts about my experiences with and realizations about SBG:

Choice of specific standards is absolutely critical, and one key choice is “grain size”. I could identify a few larger, more general capacities to assess (extreme example: “I can use work and energy ideas to analyze situations and solve problems”). Alternatively, I could unpack those into a plethora of highly targeted standards (“I can draw velocity vs. time graphs for constant-acceleration problems based on a motion diagram”, “I can draw acceleration vs. time graphs for constant-acceleration problems based on a motion diagram”, “I can draw acceleration vs. time graphs for constant-acceleration problems based on a velocity vs. time graph”, etc. etc. etc.). Somewhere in between these extremes is a sweet spot that optimally balances specificity of feedback to the student with practicality of assessment and tracking.

I seem to be on track to have a bit over a hundred standards in this course, at a rate of about 6-8 per chapter. That’s 3-4 per class meeting, more or less. That seems like a lot, and more than many other SBG practitioners seem to have — but I’m having a great deal of difficulty combining them into more coarsely-grained standards without doing violence to my sense of what the “things” to be learned really are. To put it another way: The topics seem to naturally cleave along certain lines, and allowing that gets me to where I am.

Despite that last sentence, standards can be divided along various lines, and different ways of grouping sub-elements can align more or less well with the organization of my textbook and accompanying workbook, easier ways of assessing, etc. I initially brainstormed a list of standards, but have been doing some refactoring as I went through and correlated them with textbook sections and daily class plans.

SBG drives me to assess (and reassess) EVERYTHING I want students to seriously try to learn, rather than allowing me to sample a subset of the learning goals. I suppose I could simply not assess some of the standards and let them drop out of the grading scheme, but I currently feel that if it’s on the standards list, I ought to assess it. And that’s a lot! Which leads to my next realization:

Articulating learning standards makes me much more aware of what I’m actually asking students to learn (more than I would be with a traditional by-topics list), and there’s a freaking lot of stuff for intro physics students to learn. Wow. No wonder physics is hard!

If I want a relatively simple grade calculation — each student gets a 0-4 mastery rating on each standard, and the final grade calculations consists of averaging all those ratings and then mapping to a letter grade — then the number of standards per general topic had better be proportional to the topic’s importance, since that determines its weight in the overall grade. I find it tempting to split early chapters into many fine-grained standards (e.g., specific kinematics graphing skills, specific types of motion, etc.), but leave later chapters as more holistic standards (use the Impulse-Momentum principle to analyze collisions). Unfortunately, that overly weights the early stuff. I can either weight different standards differently, or unpack the later standards into finer-grained components… which is probably beneficial to both me and the students, but darn, it’s hard work!

Unless I want to box myself into having to assess each standard multiple times, in different ways (for different levels of mastery), or having different mastery scales for different standards, I’d better construct my standards such that only one assessment probe is necessary for each. That can mean peeling “advanced” mastery levels off of the top end of the mastery rubric and creating new standards specifically targeting those. For example: Instead of having the top mastery rating be reserved for “Can recognize need to apply this within a complex scenario and figure out how to connect to other principles” (which takes a different exam question than “Can apply to a straightforward situation when prompted”), I can have a separate standard for “Identify which principle(s) apply to a complex situation” and “Combine multiple principles to solve a problem”. Put another way: If every standard has an “above and beyond” level, I need to assess every student for that level of mastery on every standard, and that’s probably unrealistic. Better to have a few explicit “above and beyond” standards.

Reassessment is the heart of SBG — it’s what makes assessment formative, and lets students learn from their mistakes and keep making progress — but it’s also looking like the hardest part to implement, at least in my context (60 students, three 50-minute classes per week, the fact that giving up my free afternoons/days to a stream of reassessing students would kill my research efforts). I’ve been very cagey about not promising anything specific about reassessment yet in this course, but I can’t keep that up much longer.

The other big question, of course, is whether students really will do the work –reading, workbook, homework, etc. — without having those be graded. Most students do end up in the trap of running from deadline to deadline, only focusing on whatever is “due” next and prioritizing tasks by grade impact.

Stay tuned. This is very much a work in progress.

Last night I leafed through the first four chapters of Knight’s Physics for Scientists & Engineers — everything up to but not including forces — and scribbled down some potential “standards”, were I to be so bold as to try SBG next semester when I teach General Physics I w/Calculus. It’s definitely not a final set, but here’s the unedited list, presented for discussion.

Note 1: Some of these “standards” span or relate to more than one Knight chapter. I’m listing them here under the first such chapter.

Chapter 1: Concepts of Motion (i.e., “Basics”)

1. Convert quantities between different units.
2. Know and (where appropriate) employ SI units for all physical quantities used.
3. Report and interpret numerical values for calculations or measurements, including appropriate units, unit prefixes, scientific notation, and significant figures.
4. Determine values of kinematic variables corresponding to described, depicted, or observed motion, and interpret values by describing or depicting the resulting motion (including proper use of algebraic signs for direction).
5. Produce, interpret, and interrelate graphs and motion diagrams of an object’s motion.
6. Know and apply the definitions of fundamental kinematics quantities.
7. Make reasonable order-of-magnitude (Fermi) estimates of physical quantities.
8. Identify correct and incorrect expressions via dimensional analysis and/or limiting-case arguments.

Chapter 2: Kinematics in One Dimension

1. Use the particle model and constant-acceleration kinematics formulae to produce a complete description of an object’s motion (numerical or symbolic) from partial information. [1D for chapter 2, 2D or 3D later.]
2. Use basic calculus (derivatives and integrals) to interrelate functional forms for kinematic quantities.
3. Use “free-fall” as a model to analyze real physical situations.
4. Use the “inclined plane” as a model to analyze real physical situations.

Chapter 3: Vectors and Coordinate Systems

1. Define and use a Cartesian coordinate system to describe an object’s location and motion.
2. Interrelate the values of kinematic variables in two different coordinate systems (including translations, rotations, and Galilean relative motion), including “relative velocity” problems.
3. Execute vector algebra (addition, subtraction, components, magnitude and direction) both graphically and algebraically.
4. Represent, interpret, and interconvert between vector representations (graphical, component n-tuple, component unit-vector, magnitude & direction).
5. Apply vectors and their properties where relevant when “using physics”. [Ick! But see note 3 below.]

Chapter 4: Kinematics in Two Dimensions

1. Use “uniform circular motion” as a model to analyze real physical situations.
2. Use “accelerated circular motion” as a model to analyze real physical situations.
3. Use “projectile motion” as a model to analyze real physical situations. [See note 4 below.]
4. Use angular kinematics in direct analogy to linear kinematics.

Note 2: All standards should carry the implicit rider “…and justify the applicability of the tools used, or identify when and why the task is not possible given those tools.”

Note 3: I’m unsure how to work in “applying” a specific thing (e.g., vectors and vector algebra) in addition to “knowing” it. I’m driving at the difference between active and passive vocabulary here: Yeah, so a student can do a vector algebra problem when presented with one, but will she identify the need to do vector algebra within an authentic context and apply it properly there? I could add a specific standard for that, but am afraid of proliferating standards.

Note 4: “Apply XXX as a model to analyze…” should be interpreted to include applying it to a piece or portion of a system or of an object’s motion, and stringing together multiple models or tools as necessary to solve a multi-part problem. (For example, inclined-plane as a model for a skier on a slope, followed by accelerated circular motion for a curved ramp at the bottom, followed by projectile motion for sailing through the air.) Or, should there be a separate standard for “Analyzing situations that require combining multiple ‘models’ or ‘kinds of motion’”?

I spent nine class days on these four chapters last time through the course (including an integrative pre-exam review day), so that comes to a hair more than two standards per day. Reasonable? Excessive? Thoughts about the grain-size of these standards?

Update: Since posting this, I’ve switched from Textile to Markdown for writing on this blog, since the best Textile plugin I could find for WordPress had some ugly bugs. Unfortunately, one of those bugs affected list numbering, with the result that the 21 standards above were numbered 1-21, rather than having the numbering reset for each chapter’s list. (The former may be preferable for this particular post, but the latter is technically correct.) So, numbers in the comments below may not correctly identify the standards they were meant to indicate. Apologies…

At the moment I’m more interested in lessons we can learn from video game design and take into more traditional, classroom-based instruction than I am in creating an actual video game that teaches physics. (The latter, however, is also a fascinating challenge to contemplate.)

In that vein, a definition by Bernard Suits (quoted in Jane McGonigal’s excellent book Reality is Broken) caught my attention:

Playing a game is the voluntary attempt to overcome unnecessary obstacles. (p.22)

The key word here is “voluntary”. McGonigal makes a case that if it isn’t voluntary, it isn’t a game, and many of the remarkable phenomena associated with game-playing disappear. The entire psychology changes.

Yes, attending university is in principle a voluntary choice, as is one’s major; but beyond that, we pretty much tell students what courses they must take and what they must do along the way to succeed, and keep them in line with grades and transcripts. Does that doom any attempt to make learning more deeply game-like?

What I’m getting at is that the very structure of our educational system frames learning activity as an externally-motivated, externally-directed, authority-laden series of tasks and assessments. I’m concerned that trying to embed a novel learning micro-environment — say, a gaming-inspired self-paced learning activity — into such a matrix could be doomed to failure, not because of the micro-environment’s worth but because of drastic dissonance with the matrix.

If I’m even more ambitious and try to construct an entire course as something analogous to a game, I still have to assign a grade at the end, and students know it.

Those of us who would like to experiment with gaming-inspired alternative paradigms and challenge some of our fundamental assumptions about what instruction should look like, and who don’t have the luxury of creating an entire parallel educational system to do our testing in, need to worry about such things.

“Playing a game is the voluntary attempt to overcome unnecessary obstacles.” — Bernard Suits, quoted in Jane McGonigal’s Reality is Broken

Is learning physics a game? Is doing physics a game? Does it depend on how obligated we feel to do any particular task? Is attending university voluntary (or compelled by social and/or economic considerations), and if so, does that make the whole endeavor a game? Taking any particular course may or may not be voluntary; doing homework, lab reports, etc. rarely is.

Why does this matter? Because in general, people like games, and often reach their best performance (think flow state) while playing games. Perhaps we ought to be learning from the game design industry.

Perhaps the human organism is well-adapted enough that if we can give learners the freedom to pursue their own learning, with an adequately rich and suitably organized and accessible array of resources, they would naturally find a highly optimal balance of these factors?

In other words, maybe people know how to learn better than we know how to teach, and we just have to figure out how to let them do it. And stop convincing them that learning something like Physics means doing it “the school-like way.”

Afterthought: I suspect we can’t do this because so much of the educational enterprise is designed to get learners to learn things they’re not particularly invested in learning. Maybe we need to revisit that?

• http://bit.ly/beatty-aapt-2010

Update: Here’s another link, in case bit.ly goes under:

• http://goo.gl/XPgM

Timing: towards the close of a unit on magnetism, after gravitation and electrostatics have been taught. (No guarantees that they’ve been learned, though.)

Question: If you were a superhero, which power would you rather have?

1. Change the mass of things.
2. Change the charge of things.
3. Change the magnetization of things.

Why is this a great question?

For one thing, it’s pretty obvious that even if the instructor might have a preferred answer (which he didn’t), there’s no “correct” answer in an absolute sense. So, students can relax a little and explore what they actually think.

I generally run this one without a small-group discussion phase before the poll. When the poll has been taken and students begin voicing their choices and reasons in the whole-class discussion phase, it rapidly becomes clear that there are many different ways to think about the question.

And then the shift happens. Students stop thinking about their goal as “come up with the most correct thing to say” (or worse, “come up with the thing the instructor wants to hear”), and start thinking about it as “come up with the most clever thing to say”. The interaction has been reframed. Score!

Somebody picks “mass” so that they could make a bullet harmless by dropping its mass to near-zero, or so they could throw a pebble and then increase its mass hugely so that it would punch through a wall. Someone else says “Wait a minute, if you increased its mass, would its velocity stay the same, or would its velocity become tiny so that its momentum was conserved?” Everyone looks at me, and I shrug and say “I guess it depends on how this superpower works, eh?” I couldn’t care less how it would work, but I’m very happy that the connection between mass, inertia, and momentum has been articulated. Score!

Someone else says they picked “charge” because they could make lightning zap things. I ask if anyone picked “charge” for a different reason, and someone else says that if they could control charge, they could make things attract or repel, which means they could make things (including themselves) levitate or fly. “Mass affects gravity, which is only for attracting; charge affects the electric force, which can attract or repel.” Score!

Then another student piggybacks on that, saying “But wait, the magnetic force can make things twist and turn as well as attract or repel. Wouldn’t that be more useful?” Someone replies “Huh?”, and a short clarification dialog ensues. I smile. Then a student asks whether the power means that non-magnetic things could be made magnetic, or only that magnetic things could be made more or less magnetic, or have their polarities switched, or what. I shrug again, happy that the distinction has been voiced. Score!

Then one student who’s been quiet all along speaks up. She says “I think I want to control charge, because that’s what brain cells use to communicate, so I could alter people’s thoughts. Maybe I could alter computer programs, too.” Eyes widen throughout the room. Score!

In the resulting silence, I innocently inquire whether she’d need to be able to sense all the charge patterns flowing around — and even harder, interpret them — to make that power useful. She looks nonplussed, and then says “Well, at least I could scramble someone’s head pretty well, maybe give them amnesia!” Laughter.

The actual discussion doesn’t flow quite this smoothly and efficiently, of course; a fair number of less interesting, or less defensible, or less comprehensible assertions are made, and I do a little prompting and steering to bring out some of these points. Nevertheless, I’ve accomplished three very important tasks: We’ve compared and contrasted gravity, the electrostatic force, and the magnetic force, and the roles that mass, charge, and magnetism play within those; I’ve engaged the students in creative, open-ended thinking to apply abstract physics ideas to real-world (okay, comic-book-world) things; and I’ve gotten the students to enjoy physics class. Triple score!

This one question nicely instantiates all four principles of Technology-Enhanced Formative Assessment (TEFA):

• Motivate and focus student learning with question-driven instruction (QDI);
• Develop students’ understanding and scientific fluency with dialogical discourse (DD);
• Inform and adjust teaching and learning decisions with formative assessment (FA); and
• Help students develop metacognitive skills and cooperate in the learning process with meta-level communication (MLC).

For more about TEFA (probably far, far more than you really want to know), see Beatty & Gerace (2009), Technology-Enhanced Formative Assessment: A research-based pedagogy for teaching science with classroom response systems, Journal of Science Education and Technology 18(2):146-162.

BTW, my inspiration for this question was a biology question by Cathy Wanat of Northampton (MA) High School (since retired). It showed a photograph of a long buffet table loaded with different food dishes, with lines of people moving along both sides as they added food to their plates. The question was “Which of the following is most like this picture?”, and the answer choices were various parts of the digestive system: mouth, esophagus, stomach, small intestine, etc. She said the resulting class discussion was mind-blowing. Thanks, Cathy!

An intriguing research result is that formative assessment may actually be counterproductive if the teacher doesn’t have adequate strategies for responding to that information. Here’s a quote about that from a paper by Dylan Wiliam:

What is less clear is what exactly constitutes effective classroom assessment. Although the studies cited above indicate that assessment for learning can improve learning, several studies have found conflicting results. For example, in a study of 32 fifth-grade teachers in Germany, Helmke and Schrader (1987) found that teachers who had an accurate knowledge of their students (as measured by the teachers’ ability to predict achievement test scores) were associated with higher levels of achievement only when the teachers also showed a high range of instructional techniques. Students taught by teachers who had a high knowledge of their students’ achievement but lacked a range of instructional techniques actually performed worse than students taught by teachers who did not know their students’ achievement. This study seems to indicate that collecting data if one cannot do anything with it is counterproductive.

Furthermore, even when teachers do manage to use information about student achievement to adjust or individualize their instruction, teachers may lack the ability to do so effectively. For example, in a 20-week study of 33 teachers in elementary and middle schools, Fuchs, Fuchs, Hamlett and Stecker (1991) found that teachers who received feedback on the achievement of students with learning difficulties in their classes made more adjustments to their teaching programs than teachers not given this information. However, the achievement of these students was improved only when this feedback was accompanied by advice from a computerized “expert system”, because the teachers not given the feedback from the expert system tended to re-explain how to do problems with the same algorithms that had led to previous failure.

Source: Wiliam, Dylan. “Keeping Learning on Track: Classroom Assessment and the Regulation of Learning.” In Second Handbook of Mathematics Teaching and Learning. Edited by Frank K Lester. Greenwich, CT: Information Age Publishing, 2007. pp. 10-11. [PDF preprint]

Of the forty-some teachers we’ve worked with to some degree or another, by far the number one difficulty they’ve reported is the challenge of regularly creating effective clicker questions to use in class. The characteristics that make a question “work” — meaning engage students in quality classroom discussion and promote learning — are not obvious, and typical back-of-the-chapter or quiz-type questions will fail miserably. In the project’s professional development meetings, we’ve spent a great deal of time talking about question creation, and I’ve developed various frameworks in an attempt to help make it more science and less art.

This semester, in prepping my own Conceptual Physics class, I’ve run into exactly the same difficulty. “Today I’m teaching topic X, and I need some good questions. Um, ah, hmm…” Not so easy, even with all the frameworks and such.

One flash of insight I had recently is that, at least for me, it’s not really creating questions that’s tough. The hard part is figuring out what I want my students to learn from the class, and casting that in terms of what I want my students to be able to do. I’ve been trying to shift my thinking from “the material” to “the demonstrable, assessable learning outcomes” (cf. The Myth of Coverage).

Once I can articulate what I would like my students to be able to do after the class, it’s generally relatively easy to invent a few good clicker questions. I just formulate a question asking them to do that (in a particular context), and then much of the class activity is me helping them struggle through the process as they learn how. (This is the principle we’ve called “Question-Driven Instruction”, as articulated in Beatty & Gerace 2009 and elsewhere.)

Which all means that when someone says “Creating good clicker questions is hard”, I’m now inclined to hear that as “Thinking in terms of demonstrable student learning outcomes rather than topic coverage is hard.” And I agree. I also think it’s one of the many desperately needed shifts to how we conceive of this whole enterprise we call organized schooling.

I’m not saying that this is the only difficult aspect of creating good questions, but it’s definitely key for me. I’m curious what others think. If you’ve taught with a classroom response system, what do you think? Does that ring true? Do you have any similar or conflicting experiences to share? Comments are open…

Last January, I moved from Massachusetts to North Carolina, and started a new job as a Physics professor. Spring was largely transition, teaching one light course here and making several long trips back north to keep the research project there going. Then came the summer, with a greater-than-usual blitz of travel and urgent work.

This fall, I started here for real. Now I’m really teaching! (And quite a lot of work it is, too.)

I’ve taught before, sort of: lots of labs and discussion sections as a TA, an outdoor leadership program for high school students, short and long teacher professional development programs… but that’s not the same has having responsibility for a full-scale university “lecture” course with ~60 students and 3 contact hours per week.

This fall, I’ve been teaching Conceptual Physics, a general education course with 55-ish students, drawing from all four class years (most heavily from freshmen and seniors) and almost every major on campus except Physics. I have, of course, been using a classroom response system (CRS, a.k.a. “clickers”). I cannot imagine teaching a course even a third this size without it; it would be like teaching without a whiteboard or a data projector. It would like becoming deaf in the classroom.

It has been interesting to see how all the pedagogical theory that I and my colleagues have been developing has fared. It isn’t easy! I’m encountering many of the same difficulties that the high school teachers in our project have voiced — problems I’ve tried to help them resolve with all kinds of sage advice.

I am finding, of course, that it isn’t quite as easy as I’d thought. (I’m envisioning many of our teachers nodding with a small smile of vindication, and thinking “See?” Touché.) It’s not so much that I’ve been wrong, as that I’m seeing new dimensions and nuances to the problems and the solutions. In fact, having to go through many of the same CRS learning issues as my teachers is really quite instructional.

I’ll try to document some of those insights in forthcoming blog posts. One question that I’m wrestling with, however, is this: just how much should I “let it all hang out” in a public forum that my students might quite possibly discover?

Maybe I shouldn’t worry about that. I find that my faith in my basic pedagogical principles and outlook are being strengthened, not weakened, by the experience. If anything, I’m entertaining even more radical thoughts about how we can re-envision the educational enterprise. (That might scare some of you who know me well.)

Anyway, the next post will focus on my latest insight about what makes creating good CRS questions difficult, at least for me. Hint: It’s not actually about the questions, though it initially appears that way.

A thoughtful post by sciencegeekgirl.

Until New Year’s Day, 1995, when an oil rig in the middle of the North Sea recorded a 100-foot wave going by during a storm. Oops.

By examining where most rogue wave reports come from (many off the SE coast of South Africa), some bright scientist figured out that when big waves and wind meet an oncoming current (such as storms from the South Atlantic moving up towards the Indian Ocean, meeting the warm narrow fast Aghulas Current flowing the other way along the African coast), waves can become much larger and steeper.

Okay, problem solved, scientists happy. Shipping industry happy too, because now they can just tell their captains what to steer around.

Until March 2001, when two cruise ships in the middle of the South Atlantic each met 100-foot rogue waves, only days apart, and were almost destroyed. There were no currents or other local features to explain the occurrences.

So an intrepid scientist in Germany got access to a radar satellite capable of measuring wave heights with sufficient precision, and started scanning satellite data for evidence of waves so high they shouldn’t exist. To everyone’s surprise, including her, she found dozens within a three-week window. All over the place.

A physicist in Italy (but with a notably American accent) thinks that ocean waves can be modeled by the “nonlinear Schrödinger equation”. One of his model-generated solutions looks almost exactly like the profile of the 1995 New Years’ Day wave that was recorded by the oil platform. If he’s right, it means that there are two kinds of waves out there: the normal sinusoidal “linear” waves, and — hidden among them — some scattered nonlinear waves with subtly different shapes. These rogues are usually indistinguishable from the linear waves, but every once in a while they go into a strange mode where they suck energy from their adjacent neighbors and rear up into monsters for a little while, then subside again.

Mariners have described this as a “rogue sea”. I swear I’ve seen this phenomenon myself, when watching a stormy ocean from shore; every now and then an unusually large wave seems to rise up somewhere, then drop again.

Part of what fascinates me about this story is the utter faith that “scientists” (as a monolithic entity, granted) had in the “linear model”: so much that they flat-out rejected numerous and continuing eyewitness accounts of rogue waves. (Let’s assume that the program’s representation of prevailing attitudes is accurate.) And the shipping and ship-design industries had complete faith in what scientists said, and built and navigated their ships accordingly. The world’s fleet of ocean-faring vessels can generally withstand 40-foot seas, but not 100-foot waves with steep faces and deep troughs. Because, after all, those can’t exist.

It took the incontrovertible evidence of the 1995 New Years’ Day wave recorded by the oil rig to cause a reexamination of the belief that such waves can’t happen, and further in-your-face evidence (literally, for too many crew and passengers) to cause a reexamination of our model for ocean wave dynamics in general. The unexplained disappearance of one ship per week was insufficient.

I suppose that shouldn’t be too surprising. One of the earliest findings of educational research was how firmly entrenched “misconceptions” are, and how emotionally difficult it is for students (or anyone) to let go of a model that has proven at least partially successful for them.

I gather that the Schrödinger model is not yet firmly established, but the existence of rogue waves is. Personally, I’ll think twice next time I consider taking a boat out in rough seas… Or next time I hear a scientist arguing that some piece of data “must” be spurious because it doesn’t fit the model we “know” is true.

The question is, what do **you** believe so strongly that you might be rejecting contradictory evidence?

My team — the UMass Physics Education Research Group (UMPERG) — is moving! The University of North Carolina at Greensboro wants to build a robust, interdisciplinary, inter-departmental effort in science and math education research, outreach, and teacher preparation, and they’re serious enough to put some resources behind it.

To kick things off, they’ve hired my mentor and my group’s senior member, Bill Gerace, to fill the newly-created post of Helena Gabriel Houston Distinguished Professor of Science Education (a chair generously endowed by the C. D. Spangler Foundation — thanks, folks). He retired from UMass and began at UNCG this past summer. I’m moving in January, and my other colleague, Bill Leonard, starts at UNCG next summer. Whee!

(For those of you not fully up-to-date on UMPERG’s status, Jose Mestre moved to UIUC a few years ago, and Bob Dufresne has been primarily occupied with his Reading Recovery publishing business for a while.)

We’ve got the ambition and the mandate to do some big things at UNCG, such as building an inter-departmental “center” for science and mathematics education research and outreach, and creating a graduate degree program in Physics Education Research and/or Science Education Research. So, if you’re a potential graduate student, sabbatical visitor, collaborator, or faculty hire — or if you’re just curious — stay tuned.

The inevitable disruption is a partial explanation for my relative silence on this blog of late. 2008 has been a very, very tumultuous year, in both good and bad ways, for many reasons. My list of ideas for blog topics is getting longer, though…

(Yeah, it’s from 1999, but I just discovered it today — thanks to Michele Martin at The Bamboo Project.)

1. Read a book on the subject, cover to cover, to get my head around the “big picture”;

2. Try one or two little toy projects as a “proof of concept”, just to make sure I have all the pieces to at least get started; and

3. Throw myself into an ambitious, real project that is well beyond my skill level, and figure things out on the fly by frequently going to the text and other relevant documentation as needed.

I find that wrestling with the big project provides the motivation and the context to help me bring it all together.

This contrasts with the way that most academic subjects at most educational levels in most parts of the world are taught: leading students along a carefully-engineered path to understanding, one step at a time. My gut and my personal learning experiences tell me that we’d be better off “throwing students into the deep end”, as I argued in _Travel, Teaching, and Intellectual Saturation_. The problem is that I don’t have a clear idea of what this would look like in practice, and I don’t have any _evidence_ that it would actually work. (Perhaps students would be too frustrated to persevere?)

Well, at a meeting today with several high school teachers participating in my current research project, one teacher gave me a glimpse of how it might be implemented, along with reason to believe that it can work.

Darcy (not her real name) is teaching 9th grade algebra, with heterogeneous (mixed achievement level) classes. Largely as a result of our project, she has been experimenting with her teaching style. Today she reported that with one class in particular, she’s been developing a class dynamic where she gives the students a problem to figure out, and then lets them spend perhaps 3/4 of the 90-minute class working together on it. The whole class works cooperatively, with small-group side conversations splitting off and rejoining the main discussion. Sometimes students go to the board to draw something, and sometimes another student will go to another board to disagree. When students look to Darcy for input, she puts on her best poker face and ignores them.

Remarkably, all but one or two students engage. I asked whether a few know-it-all students dominate the discussion, and she said no, all students’ contributions seem to be valued.

Eventually, when the class has reached a solution, Darcy will retake the helm, explore their solution, and often suggest alternative ways that they could have reached it.

Here’s the kicker: I asked Darcy whether she had trouble covering material at a sufficient pace when devoting so much time to student-directed discussion (cf. _the myth of coverage_). She said that quite the opposite happened: this class was ahead of every other 9th grade algebra class in the school. When my eyes widened, she explained that she’d rearranged the curriculum, starting off with the “hard topics” that were usually saved for later in the year. These provided the problems that students collaboratively wrestled with as described above. Then, later on in the course, she’d bring in the “easy” material that she’d skipped earlier, and the students could chew through that at a high rate — perhaps four sections per day.

Yes, it’s anecdotal evidence, but the story does illustrate one way of teaching by “throwing students into the deep end”. And it supports the rather counterintuitive idea that students learn faster when we put the hard stuff first.

I suspect that in addition, the _learning_ skills they develop are more useful in the “real world”…

The central process driving this is not globalization. It’s the skills revolution. We’re moving into a more demanding cognitive age. In order to thrive, people are compelled to become better at absorbing, processing and combining information. This is happening in localized and globalized sectors, and it would be happening even if you tore up every free trade deal ever inked.

The globalization paradigm emphasizes the fact that information can now travel 15,000 miles in an instant. But the most important part of information’s journey is the last few inches – the space between a person’s eyes or ears and the various regions of the brain. Does the individual have the capacity to understand the information? Does he or she have the training to exploit it? Are there cultural assumptions that distort the way it is perceived?

The globalization paradigm leads people to see economic development as a form of foreign policy, as a grand competition between nations and civilizations… But the cognitive age paradigm emphasizes psychology, culture and pedagogy — the specific processes that foster learning…

Excerpted from David Brooks, “The Cognitive Age”, in the NYTimes of May 2, 2008.

Korb Eynon and tribal fame: “Driving home late last night, I realized he’d done it again, 19 years after I left his classroom for the last time. Korb hadn’t impressed his thinking on me – he’d shared something that caused me to explore my own line of thinking. In other words, he’d taught. Just like he’s been doing for five decades. Thanks, Korb.”

Ask an educational researcher for his “teaching philosophy” and you’re likely to get a puzzled look and a long pause. These can be interpreted as “How do I condense years of research, literature reading, and theoretical development into a short answer?”

grounding

My philosophy of teaching draws from several research and philosophical traditions, as well as from the teaching experiences of myself and my colleagues. First and foremost, I am a constructivist (von Glasersfeld, 2007; Peschl, 2006). That term means many things to many people, but to me it means simply that knowledge and understanding cannot be “transmitted” between people; it must be constructed over time by each individual. In other words, learning is a deliberate process of sense-making that inevitably includes times of confusion, struggle, and reconciliation of difficulties. This relatively simple recognition has deep implications for instruction.

One implication is that communication cannot be taken for granted. All communication involves the sending of symbols that have no inherent meaning; meaning is intended by the sender and inferred by the recipient, and what meaning the recipient infers depends on his or her pre-existing expectations, assumptions, model of the sender, knowledge, and so on. As a teacher, that means I cannot presume that my spectacularly clear explanations communicate to the students what I intend them to. I need to model their interpretations as they model my intentions, and I need to “close the loop”, asking them to communicate back to me what they think they understood.

Another implication is that I do not “teach” so much as engineer a productive environment and set of stimuli for students to learn within, and provide coaching as they do so. (Note that this does not mean that lecture, or direct explanation, is always bad. Sometimes it is the appropriate stimuli to provide; nevertheless, I must remember that such lectures or direct explanations are not simply absorbed, understood, and immediately ready for future use.) Vygotsky’s notion of the zone of proximal development (Vygotsky, 1978) — that productive learning occurs within the space of challenges that students can succeed at with scaffolding, but not alone — suggests that I must continually tune the learning environment to students’ evolving capacities.

A third implication of constructivism is that students do not enter my classroom as blank slates; pre-existing knowledge, perceptions, perspectives, and experiences shape the understandings they construct in response to the environment and stimuli I provide. Thus, attempting to model their initial state, and track its subsequent evolution, is as important a component of my teaching job as designing my instruction.

Out of this perspective has grown the conceptual change tradition of educational research (Scott, Asoki & Leach, 2007), which studies the mechanisms by which students’ understanding of concepts evolves, the role of “misconceptions” in learning, and the like. More recent research, in what might be termed the knowledge in pieces tradition (Scherr, 2007), suggests that attending to what knowledge elements students have access to, and what contextual elements help to activate them, is more productive than considering what knowledge (or misconceptions) they “possess” (Redish, 2003; Hammer, Elby, Scherr & Redish, 2004; Dufresne, Thaden-Koch, Gerace & Leonard, 2005).

I am not just a constructivist, but a social constructivist informed by the sociocultural tradition of educational research (Carlsen, 2007; Mortimer & Scott, 2003). I see social interaction as essential to the internal knowledge-construction process, including student-to-student interactions as well as instructor-to-student ones. As Vygotsky observed (1987), the tools students use for internal cogitation are appropriated from social interactions. This implies that the classroom should be a place for exhibiting and exploring modes of thinking and argument, where students can see the process of “thinking science” modeled and where they can try it out themselves. Also, as Bakhtin observed (summarized in Wertsch, 1991, pp. 93-118), learning science largely means learning the social language of science (including conventions for thought and argument as well as vocabulary and grammar), and students must practice speaking a language to develop fluency. Thus, the classroom should be a place for students to practice “talking science”, with enough scaffolding from me to help them along, but not so much that I do the talking instead of them. In the very act of struggling to articulate their fuzzy thinking, students clarify their understanding of what they know, identify what they don’t, and often reach insights.

This has strong implications for what should occur in my classrooms. I do not see the classroom as a place for the dissemination of declarative content knowledge or the exhibition of proofs; those are more efficiently done through textbooks, multimedia, or other online resources. My classroom should be a place for dialogue and interaction, for exploration and confrontation and resolution. (In a large lecture hall, this is greatly facilitated by use of a classroom response system.)

My outlook is also shaped by the literature on student motivation and self-regulation (Koballa & Glynn, 2007; Wilson, 2006), and on the significance of students’ epistemological framing of the learning activities they engage in (Hammer, 1996; Hammer & Elby, 2003). Students are not black boxes, to whom instructional stimuli are applied and learning results; how they engage in learning activities matters tremendously, and as an instructor I must probe, model, monitor, and seek to influence that.

principles

Over time, I have distilled the practical implications for these (and other) pedagogical positions and educational research findings into four principles to guide instruction. These principles form the core of the “technology-enhanced formative assessment” (TEFA) pedagogy that my colleagues and I promote through in-service teacher professional development, and I would adhere to them in my own teaching.

The first principle is “Motivate and focus student learning with question-driven instruction.” This means posing tough, rich, meaty, often messy questions to students in order to contextualize and motivate subsequent learning, and often in order to catalyze or precipitate learning. It is grounded in the conceptual change tradition. It is motivated by an understanding that students perceive, process, and store information differently in response to a need, and that they “get” ideas by wrestling with the application of those ideas (Bransford et al., 1999, p. 139).

The second principle is “Develop students’ understanding and scientific fluency with dialogical discourse.” This means engaging students in discussion in which multiple ideas and ways of thinking are explored and contrasted, and in which students articulate and explore their own thinking. It is grounded in the sociocultural tradition.

The third principle is “Optimize teaching and students’ learning with formative assessment.” This means making students’ knowledge and thinking visible in order to adjust and optimize subsequent learning and teaching. It is motivated by an understanding that effective instruction requires detailed and current information about the specific students being taught, and that effective learning requires accurate self-knowledge (Wiliam, 2007). According to a seminal literature review by Paul Black and Dylan Wiliam (1998), “innovations” involving formative assessment produce learning gains that are among the largest ever found for educational interventions.

The fourth principle is “Help students cooperate in the learning process and develop metacognitive skills with meta-level communication.” This means communicating about communication, about cognition, about learning, and about the purposes of instructional experiences. It is grounded in literature on student motivation and self-regulation. It is motivated by an understanding that learning works better when students frame their participation appropriately and understand what they are supposed to be paying attention to.

I do not consider these four principles to be independent and arbitrary beliefs; they interlock and reinforce each other in a highly synergistic way. This can be seen in the way they are enacted in the TEFA “question cycle” — one specific way out of many of realizing the principles — which has been described elsewhere (Dufresne et al., 1996; Beatty, Leonard, Gerace & Dufresne, 2006).

references

Beatty, I. D., Leonard, W. J., Gerace, W. J., and Dufresne, R. J. (2006). Question driven instruction: Teaching science (well) with an audience response system. In Banks, D. A., editor, Audience Response Systems in Higher Education: Applications and Cases. Idea Group Inc., Hershey, PA.

Black, P. and Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1):7-74.

Bransford, J. D., Brown, A. L., and Cocking, R. R. (1999). How People Learn: Brain, Mind, Experience, and School. National Academy Press, Washington, D.C.

Carlsen, W. S. (2007). Language and science learning. In Abell, S. K. and Lederman, N. G., editors, Handbook of Research on Science Education, chapter 3, pages 57-74. Lawrence Erlbaum Associates, Mahwah, NJ.

Dufresne, R. J., Gerace, W. J., Leonard, W. J., Mestre, J. P., and Wenk, L. (1996). Classtalk: A classroom communication system for active learning. Journal of Computing in Higher Education, 7:3-47.

Dufresne, R. J., Thaden-Koch, T., Gerace, W. J., and Leonard, W. J. (2005). Knowledge representation and coordination in the transfer process. In Mestre, J. P., editor, Transfer of Learning from a Modern Multidisciplinary Perspective, chapter 5, pages 89-119. Information Age Publishing.

Hammer, D. (1996). More than misconceptions: Multiple perspectives on student knowledge and reasoning, and an appropriate role for education research. American Journal of Physics, 64:1316-1325.

Hammer, D. and Elby, A. (2003). Tapping epistemological resources for learning physics. Journal of Learning Sciences, 12:53-90.

Hammer, D., Elby, A., Scherr, R. E., and Redish, E. F. (2004). Resources, framing, and transfer. In Mestre, J. P., editor, Transfer of Learning: Research and Perspective. Information Age Publishing, Greenwich, CT.

Koballa, T. R. and Glynn, S. M. (2007). Attitudinal and motivational constructs in science learning. In Abell, S. K. and Lederman, N. G., editors, Handbook of Research on Science Education, chapter 4, pages 75-102. Lawrence Erlbaum Associates, Mahwah, NJ.

Mortimer, E. F. and Scott, P. H. (2003). Meaning Making in Secondary Science Classrooms. Open University Press.

Peschl, M. F. (2006). Modes of knowing and modes of coming to know: Knowledge creation and co-construction as socio-epistemological engineering in educational processes. Constructivist Foundations, 1(3):111-123.

Redish, E. F. (2003). A theoretical framework for physics education research: Modeling student thinking. In Vicentinni, M. and Redish, E. F., editors, Proceedings of the Varenna Summer School, “Enrico Fermi” Course CLVI. IOS Press, Amsterdam.

Scherr, R. E. (2007). Modeling student thinking: An example from special relativity. American Journal of Physics, 75(3):272-280.

Scott, P., Asoki, H., and Leach, J. (2007). Student conceptions and conceptual learning in science. In Abell, S. K. and Lederman, N. G., editors, Handbook of Research on Science Education, chapter 2, pages 31-56. Lawrence Erlbaum Associates, Mahwah, NJ.

von Glasersfeld, E. (2007). Key Works in Radical Constructivism. Sense Publisherss.

Vygotsky, L. S. (1978). The development of higher psychological processes. Harvard University Press.

Vygotsky, L. S. (1987). Thinking and speech. In Rieber, R. W. and Carton, A. S., editors, The Collected Works of L. S. Vygotsky. Plenum Press.

Wertsch, J. V. (1991). Voices of the Mind: A Sociocultural Approach to Mediated Action. Harvard University Press.

Wiliam, D. (2007). Keeping learning on track: Classroom assessment and the regulation of learning. In Lester, F. K., editor, Second Handbook of Mathematics Teaching and Learning, pages 1051-1098. Information Age Publishing, Greenwich, CT.

Wilson, T. D. (2006). The power of social psychological interventions. Science, 313:1251-1252.

Suggestions and opinions are welcome!

I probably need help with my Facebook profile, too.

I’m an educational researcher by profession. I tend to believe that I know a lot about how to teach well, especially physics. I’ve read the literature, attended the conferences, conducted research, engaged in countless discussions about teaching and learning, and published some papers. Yes, I’ve even designed and taught physics courses, though not much since finishing my Ph.D. (I’m on a research position, not a teaching one. Unfortunately.)

Which is why the following anecdote is acutely embarrassing.

This past July, my colleague, group leader, and travel buddy Bill Gerace and I spent two unexpectedly hot, humid weeks in Vitznau, Switzerland. We went to teach physics to hospitality management students (as we did in Singapore the previous summer). Does that seem bizarre? UMass has partnered with hospitality management schools in Singapore and Vitznau to offer a UMass baccalaureate. Students must fulfill regular UMass degree requirements, including “distribution” criteria of so many literature classes, so many science classes, etc. The partner schools used to ship their students over to UMass for a year or so to take all those courses, but someone figured out that it’s cheaper to send UMass faculty over to teach two-week intensive courses in various subjects. So, UMass asks its faculty for volunteers.

Knowing a good thing when he sees it, Bill jumped on the opportunity. He used the stipend to pay my travel expenses, so we both went more or less for free, inveterate travel junkies that we are. Bill taught, I helped out with computer tasks and improvised experiment/demo equipment, and I telecommuted to fulfill the duties of my “real” job. (Lest you think I’m a slacker, know that we committed to this trip before the big research grant providing my real job had been awarded.)

Back to the humility thing. A few days into the course, we reached the topic of “conservation of energy.” I have a way of explaining the concept that I think makes a whole lot of intuitive sense and should be brilliantly clear to students, so I asked Bill if I could teach that segment. He agreed, and I did. I tried to, anyway.

So I started, and introduced my analogy between conservation of energy and financial accounting, making the point that money is never created or destroyed, but moved from one account to another, to cash in your pocket, to credit (or less debt) on your credit account, etc. This is is just like energy: it gets shifted around from one form to another, one “place” to another, but the total amount remains the same. (Nobody asked about governments that print money.) This should be really accessible to students also taking management classes, right?

As it goes on, I get increasingly uncomfortable. Eyes are glazing over. A crunch on classroom space has pushed us into the computer lab for this class, and more than a little key-pecking and monitor-glancing is happening. I ask questions and get very little response; the answers I do get are tentative and unsure, more like guesses than opinions.

And then it hits me. I’m doing it: the classic IRE triadic pattern of classroom discourse, in which the instructor “initiates” with a question, the students “respond” with an answer, and the instructor “evaluates” the correctness of the response. No “uptake” or chaining of responses to responses, no true dialogic discourse or exploration of points of view. This is quizzing, not discussion. I’ve just read an entire damn book about patterns of discourse, nodding in agreement as the authors expounded upon the futility of IRE-based teaching, and here I am torturing perfectly nice foreigners with it.

It’s not that I don’t really understand the theory or the arguments against IRE. I very much do, to the point that it seems self-evident. Rather, IRE-style teaching is so deeply ingrained in me from 20-odd years of being a student (not counting preschool or the interminable stretch of my dissertation work) that I fell into it without even thinking.

So I bailed. I tag-teamed off to Bill almost mid-sentence. No one can improv physics like Bill, so he picked up smoothly and continued the lesson (with significantly less IRE and eye-glazing).

Licking my wounds later and reflecting on the experience, I realized I had been doomed from the very moment I first desired to teach that lesson. I began by thinking about what was inside my head — the cool analogy I was going to make — rather than about what was inside the students’ heads. Rule #1 of teaching:

It doesn’t matter what comes out of your mouth (or shows up on your PowerPoint slides). All that matters is what happens in the students’ minds, so find out what that is and interact with it.

The myth of coverage is a corollary of this.

The morals of this story?

1. There’s a huge gap between knowing and doing. We generally do what we’re patterned on, not what we would choose if we thought about it. Especially under stress or on the spot.
2. If we really want to impact the way science (or anything else) is taught, we must change the formative learning experiences of our future teachers. It’s a bootstrapping problem.
3. Don’t lose sight of the goal for even a moment: in this case, developing students’ understanding. Teaching cleverly is not synonymous with making learning happen.
4. Self-monitoring and reflection are very powerful learning tools. I learned more from that one experience than from dozens of learned papers and discussions. (Bill likes to say that “All learning is through trauma.” He’s using learning in a narrow, strong sense and trauma in a general, cognitive one.)

(If you click on the title links, the relevant “slide” should open in a new browser window. Subsequent slides should open in the same window, so if you resize the window to something about 1024 x 768 and drag it to the side of the narrative in this window, you should somewhat recreate the effect of the talk. Except for the part about me being rattled by technical problems and talking way too fast. Hopefully, your display is working better than my projector did during the talk. The slides are HTML, not PDF or images, so fonts and layout and such may vary.)

Title & Authors

Good evening.

This is the kind of talk where I tell you about a current research project, because it’s good for professionals to know what their colleagues are up to. It’s also the kind of talk where I slide in my own pedagogical opinions, because, well, I want to change the world.

If you hear anything brilliant in my talk, credit probably belongs to my colleagues in the UMass Physics Education Research Group. They’ve been thinking about this for longer than I.

Classroom Response Systems

How many of you know what a “classroom response system” is? Also known as an “audience response system”, “voting machine”, “polling system”, or “clicker system”? [Probe audience and adjust talk as appropriate.]

Briefly, for those of you who haven’t: A classroom response system provides a supplemental, technology-mediated channel of communication between instructor and students. [1]

It is a combination of hardware and software that allows an instructor to:

* Present a question to students in class;
* Have them submit answers;
* Immediately aggregate the responses; and
* Share the results with the whole class, usually as a histogram.

[Pause.] Are classroom response systems effective?

[Pause.] Please raise your hand if you think they’re educationaly effective… Who thinks they’re not so hot?… Who’s still sitting on the fence?…

That was a trick question.

Assessing Instructional Technology

This session is about “assessing the educational effectiveness of technology”, but technology doesn’t have educational effectiveness. At least not by itself.

What is the culinary effectiveness of a wok?

[Pause.] If you know how to use one and you’re trying to make a nice stir-fry, it’s quite effective. But if you have no clue what you’re doing in a kitchen, or you’re trying to make a quiche, it’s pretty darn useless.

Any evaluation of instructional technology must ask four questions:

* For what purpose is the technology being applied?
* How is it being applied?
* How well is the user doing it?
* How well does the technology enable or enhance the attempt?

Only the fourth is about the technology itself.

[Pause] That perspective motivates the project I’ll be describing.

Uses for Classroom Response Systems

Classroom response system can be tremendously powerful instructional tools, but they’re not a silver bullet. We’ve seen them used completely ineffectually, horribly abused, or used well for ends we don’t see much merit in.

Some of the different goals people use classroom response systems for include:

* Taking attendance
* Administering quizzes
* Provoking engagement [2]
* Checking for progress
* Promoting knowledge diffusion [3]

All of these typically involve sprinkling response system questions throughout “normal” instruction. The approach we at UMPERG have developed is, I think, considerably more radical. [4]

The Question Cycle

The core idea is that we structure class time around an interactive question cycle [5], iterated three times per hour, more or less. The question cycle serves as the primary vehicle, the primary engine, for instruction. It’s not an add-in; it’s the meat of the class.

We use the question cycle to:

* Reveal and explore students’ thinking,
* Introduce new ideas,
* Refine and extend students’ understanding, and
* Develop students’ analysis and problem solving skills.

A classroom response system is not strictly necessary for this, but it sure helps.

Classroom Dynamic

There’s a lot more to our approach than the question cycle, but that would require a few more ten-minute talks.

What do we call it?

We’ve called the approach by various names at various times, depending on what aspect we’re focusing on.

* Question Driven Instruction (QDI)
* Technology Enhanced Formative Assessment (TEFA)
* Assessing to Learn (A2L)
* Agile Teaching (AT)

I’ll use the first one, “Question Driven Instruction” or “QDI”, tonght.

So back to the project. We want to make a really strong case for the efficacy of classroom response system coupled with the QDI pedagogical approach. That means doing a large-population study of student learning impacts, a so-called “scaling study”, rigorous enough for the What Works Clearinghouse.

But, we can’t evaluate learning impacts fairly without a cadre of teachers who are trying to do QDI, and doing it competently.

Scaling Study Preliminaries

So, we need two things first.

* We need a professional development program that can efficiently and reliably move teachers to a state of QDI competence.
* And we need measures of implementation fidelity that tell us when a teacher is, in fact, doing a reasonable job at it.

If we don’t have the first, we won’t have any QDI to study. If we don’t have the second, we won’t know whether negative results — that is, poor learning impacts — indicate that QDI doesn’t work, or that it just isn’t happening.

So the project we proposed to The National Science Foundation, and that we’re currently working on, is a preliminary study that sets up a scaling study.

TLT Project Goals

Our project is called Teacher Learning of Technology Enhanced Formative Assessment. We are studying teacher learning, not student learning. There will be no measuring of student learning gains.

The project has three general goals:

# To better understand how teachers get from novice to expert in the use of a classroom response system and QDI.
# To refine our methodology for teaching QDI to teachers, and “package” it so that others can do the professional development.
# To prepare the measures, instrumentation, design, and general know-how for a “scaling study” on student learning impacts.

TLT Design: Professional Development

Our plan is to work with the entire science department at a high school, so that teachers can learn collaboratively and support each other, and so that students get a consistent learning experience from class to class.

During the first treatment year, we’ll conduct an intensive two-semester professional development course. The course will focus on the skills that go into successful QDI, including:

* Using classroom response system technology,
* Designing effective questions,
* Navigating the question cycle,
* Moderating classroom discourse, and
* Integrating QDI with curriculum goals and external constraints.

[Pause.] Change doesn’t happen overnight. For all three treatment years, we’ll facilitate a collaborative action research program for the teachers. This will:

* Support the ongoing evolution of their teaching practice, and
* Provide a forum where the teachers can set the agenda and come to terms with QDI.

TLT Design: Data Acquisition

We’ll collect longitudinal data on teacher change over the three treatment years, plus baseline measurements. We’ll gather data from classroom observations, and from interviews and surveys, in order to track changes in:

* What teachers do in the classroom;
* How they approach lesson planning;
* How they perceive knowledge, learning, and teaching;
* What aspects of teaching occupy their attention most;
* What difficulties they wrestle with;
* What supports and assistance they find helpful;
* How they perceive their own QDI skills; and
* How their students perceive the classroom environment.

We’re developing most of the research instruments ourselves, since existing instruments don’t really address the variables we want to track. We are, however, incorporating pieces of established instruments to aid comparison with other research.

TLT Timeline

We’ll be working with two schools, staggered by one year so that we’re taking baseline data in school 2 while teaching the PD course in school 1.

Our first school is fairly small, with eleven participating teachers. This includes most of the high school science faculty, as well as some of the math and junior high science teachers. We’re planning on a larger cohort for school 2, with over 20 participants.

Where are we now? We’re just wrapping up baseline data collection for school 1. Next week, we’ll kick off the PD course with a three-day workshop. We don’t have any results yet, but come back next year for some fascinating preliminary findings!

Links & Credits

If you want to learn more about the project, come chat with me and see our poster during tomorrow morning’s poster session.

Or, visit our web page about the project. (It’s a little sketchy right now, but I’ll be augmenting it soon.)

If you fell asleep in the middle of this talk and want to see what you missed, I’ll post the narrative and slides to my personal weblog, hopefully tomorrow.

And by the way, our graduate student Colin Fredericks is giving a talk at ten tonight, right here, about a different but also interesting project. So please stick around a little longer.

Thanks for your time. Any questions?

¹ Ian D. Beatty (2004) Transforming Student Learning with Classroom Communication Systems. Educause Center for Applied Research (ECAR) Research Bulletin ERB0403.

² Robert J. Dufresne, William J. Gerace, William J. Leonard, Jose P. Mestre, Laura Wenk (1996). Classtalk: A Classroom Communication System for Active Learning. Journal of Computing in Higher Education 7, 3-47.

³ Mazur, Eric (1997) Peer Instruction: A User’s Manual (Prentice Hall, Upper Saddle River, NJ).

⁴ Ian D. Beatty, William J. Leonard, William J. Gerace, Robert J. Dufresne (2006). Question Driven Instruction: Teaching science (well) with an audience response system. In David A. Banks (Ed.), Audience Response Systems in Higher Education: Applications and Cases (Idea Group Inc., Hershey, PA).

⁵ Robert J. Dufresne, William J. Gerace, Jose P. Mestre, William J. Leonard (2000). ASK-IT/A2L: Assessing Student Knowledge with Instructional Technology. UMass Scientific Reasoning Research Institute technical report Dufresne-2000ask.

So why was I surprised to find that Uganda is different?

The geographically erudite reader will note that the African countries I’ve previously visited are all in the southernmost portion of Africa, more or less between 17 and 34 degrees south latitude. Uganda is smack on the equator. My southern African countries all have a history of colonial rule, and the one I’m most familiar with — South Africa — is still recovering from the brutality of Apartheid. Uganda was a British protectorate, never a colony. Uganda has different ethnic groups, different languages.

Uganda is different. Duh!

If I stopped there, this essay would be a simple self-smacking of the forehead. In the “Think Twice” spirit, however, I’m going to dig a little deeper. Is there a moral here, aside from the always-apt “beware the trap of hubris”?

I think there is, and it has to do with the nature of knowledge, and the many kinds of knowing. I “knew” in an abstract, conceptual, and logical sense that Africa is variegated, but not in a deep enough way to affect my unexamined expectations. Perhaps this parallels the distinction between “passive” and “active” vocabulary. (A person’s “passive” vocabulary with a language is all the words she understands when she hears or reads them. Her “active” vocabulary is all those that come to mind, unprompted, for use when speaking or writing.)

Educational researchers know (heh) that there are many kinds and degrees of knowing, and that we don’t fully understand all that’s involved in the thing we blithely call “knowing” [Redish-2003tfp]. It’s complicated. For useful knowledge, we need to have the right “mental resources” in our heads, and we also need to have the right associations and triggers in place so that those resources are “activated” in the appropriate contexts.

When do I understand the concept of “force”? When I can spell it? When I can quote a definition? When I can recognize the presence of one in simple and familiar situations, or in subtle and novel situations? When I can use the concept as a tool to reason with in familiar contexts? In unfamiliar ones? I doubt there’s a person on the planet who can do all of these things in all possible cases, infallibly, so does anybody really understand “force”?

If that doesn’t hurt your brain enough, consider metacognitive knowledge: knowledge about your own knowledge. To quote my colleague and former dissertation advisor, Bill Gerace: “Sometimes you know something. Sometimes you know you know something. And sometimes you know you knew something, but don’t any more.” What’s going on there?

And, as Uganda has reminded me, sometimes you only think you know something.

Evidence of myth: How often the verb “I taught” is used interchangeably with “I presented to my students”.

Fact: “What the instructor covers” differs from “what the students learn”, often drastically.

Evidence for fact: Stop teaching and check for understanding. If you really pay attention, you can’t miss it.

* * *

Why is this so difficult for people to get?

I can’t tell you how many times I’ve tried to explain this. Usually, I’m extolling instruction that focuses on developing students’ understanding and ability to reason with ideas, rather than simply rote-learning and regurgitating facts and procedures. I’m getting into the importance of earnestly probing to see what students actually understand, and not proceeding until they “get it”. Of taking the time to build a solid foundation, rather than building on sand. And then, almost inevitably, a teacher says “But in reality, that’s too slow. We have a certain amount of material to cover. If we don’t finish the syllabus, the students won’t be prepared for the [exam/next course/whatever]“.

What good, I ask, does it do to “cover” something if students don’t actually learn it? I usually get dead silence at this point. The person I’m talking to doesn’t have an answer, but doesn’t like the conclusion either.

I think the reason this is a difficult idea for many instructors to swallow is that swallowing it leads to a very uncomfortable line of thought, one that challenges their very role as a teacher.

If what my students learn cannot reasonably be assumed to match what I present, and my responsibility is to teach (i.e., cause or facilitate learning), then I have to stop focusing on syllabi and lesson plans and lecture notes and beautiful PowerPoint slides, and start focusing on my students’ knowledge and learning and difficulties and pre-existing conceptions. I have to get out of my own head and into my students’ heads. And that requires a whole new set of skills, a whole new role.

It’s much less threatening to think my responsibility stops at summarizing the contents of a textbook, assigning homework, and setting exams. And if the students aren’t meeting my expectations, then obviously they’re either lazy (their fault), stupid (nobody’s fault), or underprepared (their previous teachers’ fault or “the system’s” fault).

* * *

Coverage does not necessarily imply learing, and therefore does not necessarily imply teaching. I keep “covering” this point, and yet, somehow, my listeners don’t learn it.

If you want to find out whether someone understands something — a concept, a procedure, a perspective — don’t just ask whether they understand. The chances of a false positive are too high. Why?

1. The learner may think they understand, when they don’t, or at least not fully. The more complex the topic, the more likely this is. “Yes” can often mean “I think I get a little something of what you just said” (the optimistic approach).
2. The learner may know they don’t understand, but have framed the interaction as “trying to give the instructor the answers he/she wants.” This is distressingly common in authoritarian educational systems that stress drill-and-practice, which includes much of the world. Especially anyplace the British have been.
3. The learner may be saying whatever it takes to get you to leave them alone and go away, or at least pick on someone else in the class. The more completely lost they are, the more likely this response becomes.
4. With multiple learners, you’ll hear from the few bravest, most confident students. Even if they get it, that doesn’t mean the rest do.

In some cultures and contexts, you’re virtually certain to be told “yes” no matter what is going on in the learner’s head.

So how do you ascertain understanding? Ask the learner to use the idea, procedure, perspective, or whatever it is. Not just parrot back what you’ve said, not just mimic something you’ve demonstrated; actually apply the thing in a way that demands actually “getting it.”

A side benefit is that using knowledge will help the learner get it, and get it better, and keep it.