Assessment: A study of a study of a study
It’s a staple conclusion in any research report that ‘more research is required,’ but a United States Government Accountability Office study of a US Department of Defense study into the cost of Department of Defense studies has taken that idea to a new level. The whole idea of a study of a study of studies was meant to help the Department of Defense to reduce costs, but according to the Government Accountability Office study, ‘The Department of Defense has not evaluated and currently does not plan to evaluate the usefulness of its efforts to estimate the costs of selected reports and studies.’ It recommends that, ‘To determine whether its effort to estimate costs is having the desired effect of achieving greater transparency, reducing or eliminating reporting requirements, and raising cost awareness, the Secretary of Defense should take steps to evaluate the effort.’
If you’re not aware of a study of a study of studies occurring in your school that may be of some comfort, but this is not just a salutary tale about the dangers of very big bureaucracies. If you think about the ways you undertake assessments of your students, or evaluations of programs in your school, you typically start with your goals, and then design assessments to serve those goals. Back in 2010, then Secretary of Defense Robert Gates, reacting to the fact that the Department of Defense was ‘awash in taskings for reports and studies,’ called for the study, noting that there is little basis to determine whether the value gained from the studies is worth the considerable time and resources expended to generate them. What he asked for was an evaluation of the cost when the goal really should’ve been an evaluation of the benefit.
It’s a truism of assessment that we often assess what we know we can measure, not what we really want to know.
Read the United States Government Accountability Office’s Actions Needed to Evaluate the Impact of Efforts to Estimate Costs of Reports and Studies here.
As an assessment expert, how would you assist US Secretary of Defense Leon Panetta to take steps to evaluate the benefit of studies churned out by the US Department of Defense?
DID YOU KNOW?
Did you know that former US Secretary of Defense Donald Rumsfeld – he of the unknown unknowns – held the office twice, first in the presidency of Gerald Ford from 1975 to 1977 and then in the presidency of George W Bush from 2001 to 2006?
AND BY THE WAY
If you’re a teacher of philosophy or ethics, you might want to read the rather chilling transcript of Donald Rumsfeld on unknown unknowns here.
