questioning research

I'm reading some research papers in preparation for a presentation I'm due to give at ACEC in Cairns later this year. This paper was recommended by Tony Forster.

Questioning the Questions of Instructional Technology Research

The author, Thomas Reeves, starts from the premise that when research is done using taxpayers money that the research students should be counselled / pressured into chosing topics that are more socially important than what motivates people to take SCUBA diving courses.

There are important social issues in the world - adult illiteracy, attacks on public education, "at-risk" students, homelessness, AIDS etc.

He goes onto argue that the goal of educational research must be to improve education not just to understand what is happening. Institutionalised Education is not a natural phenomenon, it is "man-made" so there is a need to make it better.

This is an old but still important argument, the distinction between knowing the world and changing the world. Marx critiqued Feurbach along those lines too:

The philosophers have only interpreted the world, in various ways; the point is to change it.

I agree with Thomas Reeves on this basic premise and he follows this thread through in a practical sort of way.

He's tough on the issue of pseudo-science. He says that a lot of research is not up to scratch based on criteria. Figure 6 is the key link. The critique he is making of most educational research is along these lines:

• lack of linkage to a robust theory
• inadequate literature review
• hard to measure variables are not measured reliably
• outcomes of research have little or no relevance for the subjects in the study
• statistical analysis is fudged
• rambling, often incoherent discussion of results

He frankly states that there is an "incestuous relationship" between many researchers and peer reviewers.

This critique is frightening for our Game Making cluster IMO because most of us are not trained in such a rigorous research approach and are too busy, don't have the time / resources to read all the relevant literature, study up on statistics etc. We are practitioners who have dabbled in educational theory without reaching the rigorous standards demanded by Thomas Reeves.

However, his solution to this problem in a way lets us off the hook and steers around the above critique by suggesting a different sort of approach to educational research. He wants academics to work more closely with schools to strive to make a difference there, "where the real needs are so great." That the goal of educational research must be to make a difference, "in schools with real problems".

This is something members of our Game Maker cluster can relate to because that is what we are on about, that is our core business, and what brought us together in the first place. However, I don't really believe that we are off the hook entirely. What is our core theory? What literature have we read? Have we thought about those hard to measure variables? Statistics? We need to do some of this.

Reeves concludes with a summary of a couple of instances of developmental research that he does like.

One is Idit Harel's (1991) Instructional Software Design Project, which I have read and was so impressed by that I tried to emulate it a few years ago (yet to publish my paper on that).

The other is a study by Richard Lehrer(1993) where eighth graders used HyperAuthor to design their own lessons about the American Civil War. ... based on Perkins theory that "knowledge is a process of design and not something to be transmitted from teacher to students..." One interesting result of this study is that important differences were found a year later (but not immediately). The control group forget whereas the constructionist group remembered history as "a process of interpreting the past from different perspectives ..."

I agree with Reeves idea that theory is important and have been gradually reviewing my already established ideas about constructionism and thinking more about George Siemens new idea of connectivism. How do those theories relate to what I am doing with Game Making and blogging?

Reeves paper also influenced me in thinking about going beyond an empirical, quantitative approach and to make the effort to integrate the above theories with what I observe my students doing when they solve game problems and blog about that.

Tags:

• research