Information and Educational Assessment I

In a letter to the NYT about an article on radiation overdoses, George Lantos writes:

My stroke neurologists and I have decided that if treatment does not yet depend on the results, these tests should not be done outside the context of a clinical trial, no matter how beautiful and informative the images are. At our center, we have therefore not jumped on the bandwagon of routine CT perfusion tests in the setting of acute stroke, possibly sparing our patients the complications mentioned.

This raises an important, if nearly banal, point: if you don't have an action decision that depends on a piece of information, don't spend resources (or run risks) to obtain the information.  The exception, as he suggests, is when you are doing basic science of some sort.

Now consider, for a moment, the practice of "assessment" in contemporary higher education.  An industry has built up around the idea of measuring educational outcomes in which a phenomenal amount of energy (and grief) is invested to produce information that is (1) of dubious validity and (2) does not, in general, have a well articulated relationship to decisions.

Now the folks who work in the assessment industry are all about "evidence based change," but they naively expect that they can, a priori, figure out what information will be useful for this purpose.

They fetishize the idea of "closing the loop" -- bringing assessment information to bear on curriculum decisions and practices -- but they confuse the means and the ends.  To show that we are really doing assessment we have to find a decision that can be based on the information that has been collected.  Not quite the "garbage can model of decision-making," but close.

Perhaps a better approach (and one that would demonstrate an appreciation of basic critical thinking skills) to improving higher education would be to START by identifying opportunities for making decisions about how things are done and THEN figuring out what information would allow us to make the right decision and THEN how we would best collect said information.  Such an approach would involve actually understanding both the educational process and the way educational organizations work.  My impression is that it is precisely a lack of understanding and interest in these things on the part of the assessment crowd that leads them to get the whole thing backwards.  Only time will tell whether these scientist-manqués manage to mediocritize higher education or not.