Schrum, L., Thompson, A., Sprague, D., Maddux, C., McAnear, A., Bell, L., & Bull, G. (2005). Advancing the field: Considering acceptable evidence in educational technology research. Contemporary Issues in Technology and Teacher Education [Online serial], 5(3/4). Available: http://www.citejournal.org/vol5/iss3/editorial/article1.cfm

Advancing the Field: Considering Acceptable Evidence in Educational Technology Research

Lynne Schrum, Editor
Journal of Research on Technology in Education

Ann Thompson, Editor
Journal of Computing in Teacher Education

Debra Sprague, Editor
Journal of Technology and Teacher Education

Cleborne Maddux, Research Editor
Computers in the Schools

Anita McAnear, Acquisitions Editor
Learning and Leading with Technology

Lynn Bell and Glen Bull, Editors
Contemporary Issues in Technology and Teacher Education

The Spring 2005 issue of the Journal of Research on Technology in Education (JRTE) included an unprecedented position paper collaboratively authored by a group of journal editors. The paper issued a call for a proactive approach to a research agenda in educational technology. The participating educational technology periodicals and journals included Learning and Leading with Technology, the Journal of Research on Technology in Education, the Journal of Computing in Teacher Education, Computers in the Schools, the Journal of Technology and Teacher Education, and Contemporary Issues in Technology and Teacher Education (CITE Journal).

This call for a proactive approach was prompted by widespread acknowledgment that a more organized and persuasive body of evidence on the benefits of digital technologies in schools is required. Addressing this need has two components: (a) identification of the need and (b) development of a systematic approach to resolution of the problem.

This editorial provides a follow-up report of the status of one facet of this effort, as promised in the initial call to action. The directions outlined are based on ongoing dialog among the task force of editors of the participating journals during the past year, culminating in two days of discussion at the seventh National Technology Leadership Summit (NTLS VII). This meeting, held at the Library of Congress in Washington, DC, also included the leaders of 10 educational associations, federal policy makers, and selected corporate partners. (Further information about the summit is available at www.NTLS.info)

A separate NTLS task force chaired by Gerald Knezek and Rhonda Christensen (chair and co-chair of the Society for Information Technology and Teacher Education, or SITE, research committee) is addressing the related issue of “Key Research Questions in the Core Content Areas.” They are working in concert with leaders from the technology committees of teacher educator associations representing the core content areas, and their report will be published in a subsequent article in CITE Journal.

Background

Four issues regarding educational technology research are currently converging. The potential for educational technology to revolutionize education has been described repeatedly, yet the promise has not been fulfilled (Conlon & Simpson, 2003; Cuban, 2001; Cuban, Kirkpatrick, & Peck, 2001; Sandholtz, 2001). Second, concerns have been raised that research on technology has failed to produce evidence that it makes a difference in the teaching and learning process (Lagrange, Artigue, Laborde, & Trouche, 2001; Pollard, 2004-05; Roblyer & Knezek, 2003; Strudler, 2003; U.S. Department of Education, 2004). Simultaneously, educational research, in general, has had an inauspicious history and has been challenged as being driven by multiple masters and as not providing strong evidence for any solutions to complex questions (Lagemann, 2000). Finally, the U.S. Department of Education challenged all educational researchers to consider carefully what constitutes scientifically based research (SBR).

The No Child Left Behind Act (NCLB) of 2001 specifically defines SBR as requiring empirical methods, randomized assignments, rigorous data analyses, and measurements or observational methods to obtain reliable and valid knowledge. According to this definition, research must be replicable and lead to findings that can be generalized and accepted by independent sources (Shavelson & Towne, 2002).

Each institution has its own research standards and policies such as those reflected in unique requirements by individual Institutional Review Boards and in the interests and expertise of its faculty members and graduate students. However, there are many commonalities of good practice shared by those who conduct excellent research. Ensuring that research is grounded in theory, or that prior knowledge is understood and incorporated are only two examples of many such universals. Therefore, we should begin to define the types of evidence that will be seen by teachers, policy makers, other researchers, and the public at large as serious, as useful, and as contributing to theory, understanding, and practice.

Acceptable Evidence: Endorsing a Platinum Standard

The Department of Education has judged that the “gold standard” for scientific research always involves randomized experimental designs with a treatment and a control group. However, it is difficult to achieve the gold standard in authentic school settings.

During the past year the authors and their colleagues have held a series of discussion panels at SITE05, the annual conference of the American Educational Research Association (AERA) , the National Educational Computing Conference (NECC), and other venues, meeting with dozens of doctoral students and novice researchers beginning their careers. These discussions revealed a typical pattern of behavior by beginning researchers.

This pattern is logical and rational in the context of their perception and understanding, but may not always lead to the highest quality of research. It goes without saying that a critical objective of a faculty member in this position is to survive by securing tenure and that tenure depends largely on publication productivity. It is considerably easier to carry out a randomized experimental design in nonschool settings than in school settings. Consequently, some researchers are responding by undertaking rigorous experimental designs only in nonschool settings. At a conference panel session on this topic in 2005, one assistant professor expressed her perceived dilemma in the following way:

I want to help the doctoral student identify research questions relevant to schools, and address questions important to policy makers. But if I help the doctoral student do that – the work of the school and the policy makers – I may not get published. Unless a study is implemented with the rigorous methodologies that many journals require, chances are that it won’t be accepted. And those rigorous methods aren’t always possible in schools. I am on the promotion and tenure track. So I want to throw this back to the journal editors on the panel and the reviewers. I want to do applied, evidence-based research but it won’t help me in my career.

In fact, all educational technology journals and periodicals encourage articles addressing actual practices in schools. The editors participating in this panel hastened to provide assurances that submissions in this area are both desired and needed.

However, the assistant professor and other audience members stated a belief that they can complete several publications involving research in nonschool settings in the time required to complete one study in an authentic classroom setting. They feel constrained by the publication requirements of promotion and tenure committees even though they might personally prefer to undertake fewer (but arguably more authentic) studies connected to teaching and learning in school settings.

This is a dilemma that will not be easily resolved, but it needs to be addressed for the sake of the profession as a whole. The dilemma is that actions benefiting individual researchers sometimes do not advance the best interests of the profession.

Journal editors are aware of the tension inherent in conducting research consistent with scientific principles that also takes into consideration the classroom context. Real schools and classrooms are messy and complex, and myriad factors contribute to each experience of a particular classroom, including individual attributes of the educator and learners and the subculture of any particular school. Experimental designs are often isolated from classroom realities, and results do not fit neatly into authentic teaching situations.

Consequently, we endorse a "platinum standard" for school research. The platinum standard requires rigorous research in authentic school settings that approaches idealized designs as nearly as possible given the constraints of schools and real-world learning environments. This term is suggested to illustrate that journal editors seek authentic research in authentic learning situations and recognize that research in these settings involves a number of complex design decisions and compromises.

Advancing the Field: Connecting Beliefs, Practice and Learning Outcomes

Much of the research in educational technology (and in the field of education as a whole) has not been directly connected to schools or related to learning outcomes. We must create a strategy that simultaneously meets the requirements for evidence that technology can make a difference in classrooms and articulates what we understand to be essential in asking appropriate questions and designing authentic research.

Many surveys on teacher beliefs and self-reports of educational technology practice have been conducted in an isolated manner, without clear connections to theoretical frameworks, to each other, or to the larger literature base. There is a need to connect teacher beliefs, teacher practices, and subsequent student learning outcomes. To that end, we collectively endorse the following recommendations.

1. Teacher Beliefs about Technology

The educational technology literature is saturated with studies focused on teachers’ attitudes toward technology and their feelings of self-efficacy toward integrating technology in their instruction. We encourage a more thorough examination of teacher beliefs about technology. Useful studies could examine how teachers define educational technology, what teachers believe constitutes appropriate technology use in the classroom, and what is their rationale for using or not using technology. These types of questions obviously lend themselves more to open-ended questions as opposed to forced-choice questionnaires.

Our strategy should not end, however, with examining teacher beliefs more in depth, because we know that teachers’ beliefs and intentions do not always translate into practice.

2. Teacher Practice with Technology

We recommend more research on teacher practice with educational technologies. In order to progress, though, we need to move beyond self-reports. While self-reports of teacher practice can be helpful, direct evidence of practice is needed. Methods for securing such evidence include, but are not limited to, direct observations, video case studies, and samples of lesson plans and associated student work. When exemplary teaching is identified, it should be acknowledged and disseminated.

Unfortunately, even exemplary teacher practice does not automatically translate into better student achievement. We believe this is the area in which much more effort needs to be directed.

3. Student Learning Outcomes

The educational technology field is grounded in the belief that technology and innovation will ultimately lead to improved learning outcomes. The connection between teaching practice and learning outcomes must be demonstrated. This is the most challenging area because it is difficult to tease out the impact of technology on student learning among the other variables that may influence learning.

One concern has been that standardized tests may not capture full evidence of all the student learning that occurs. Researchers should cast a wider net, as a way of initiating constructive discussions on this topic, but also have a responsibility to examine the effects (if any) of practice on standardized scores.

Another concern has been the magnitude of the task required to affect the outcomes of high-stakes, end-of-year tests. If the effect of practice on discrete, individual topics is considered, the task may appear less daunting. Through a series of successive approximations building on experience with individual concepts and lessons, we may eventually be able to address larger learning objectives. Different technologies are designed to address different content areas, and even specific concepts within a given subject area. Studies aggregating generic technologies, such as the “effect of the Web on learning,” may obscure real differences occurring in more defined content areas.

Research examining connections between beliefs, practices, and learning outcomes is required if the profession is to advance. Much of the research to date has focused on teacher beliefs, in part because these are easier to investigate. A lesser amount has been grounded in actual teacher practice, and even less research has focused on actual learning outcomes. Replicable, generalizable studies that address student learning outcomes are the ultimate goal and are the reason for conducting studies on teacher beliefs and practice that can be causally related to these outcomes.

Facilitating Effective Research: Next Steps

Adoption of the premises outlined above is a constructive first step in articulating our current beliefs. This leads naturally to the question of how we can translate these beliefs into the practice of research.

Mentoring

Obviously, many aspects of these issues are beyond the scope of journal editors to influence. What we can do, however, is carry out a productive discussion about the aspects of these issues that we can most directly affect in our capacity as journal editors and teacher educators. We, therefore, have adopted the following proposition as an initial premise:

Our most pressing objective is to identify how we can assist the coming generation of young researchers in carrying out research that is needed, relevant, rigorous, and influential in the formulation of educational policies in schools.

We are collectively exploring how to move toward this objective and how to assist graduate students and new faculty members to participate. We believe that discussions are needed concerning what our graduate students need to know and be able to do to understand the research of others, to conduct their own research, and to disseminate research findings in a way that has a good chance of influencing teachers and educational policy makers.

There are a number of issues that affect the ability of both beginning and experienced researchers to conduct effective research in school settings. The complex and rigorous requirements of Institutional Review Boards (IRB) place increasingly onerous burdens not only upon researchers but upon school personnel who may collaborate in studies. Requirements of this kind have been interpreted and implemented in differing ways at different institutions.

Discussion of effective ways to protect the rights of subjects while simultaneously facilitating effective research in ways that are not burdensome to schools is needed. Identification of ways to facilitate school-university partnerships can create a climate conducive to needed research.

We have begun discussions about these issues in these joint editorials and in a series of related panel discussions at the SITE, NECC, and AERA conferences.

Mentoring is probably one of the more productive uses of our efforts. Mentoring may be done through personal contacts, joint editorials, fireside mentoring sessions at conferences, and online discussions. These activities can focus on challenges to new researchers, such as gaining IRB approval, identifying strategies so university researchers can work with local school systems, and promoting the conducting of rigorous research in intact classrooms (i.e., settings in which students are not randomly assigned, etc.). All of these activities are being scheduled at forthcoming meetings and professional conferences.

Communication and Dissemination

To effectively influence practice, the results of research must also be communicated to policy makers, school board members, administrators, and teachers. Both the focus and the quality of research are irrelevant if the results are unknown to members of these important groups. It is possible that journal editors can play a useful role in facilitating communications among members of the educational technology community.

Several possibilities have been suggested that could serve as a starting point for this dialog. For example, would research editors be willing to collaboratively construct a Web presence in which each issue, or even each research article published has a complementary practitioner article? Such a Web site could also contain lists of magazines and journals whose readership includes teachers and educational policymakers and suggestions for writing articles for such publications that summarize research results.

Editors of practitioner-oriented publications might be invited to publish their opinions about how educational research needs to be improved, and the views of teachers, school board members, and others could be featured on the Web site in interview format. Another possibility would be for editors to encourage or require that all research articles include a brief paragraph or two with suggestions for how results could be disseminated to critical stakeholders, as well as clearly identifying possible barriers to implementation in school settings. These ideas are merely suggestions, and we welcome input from all researchers interested in improvement of our craft.

Conclusions

We collectively bear responsibility for ensuring that our work as a profession leads to enhanced student learning. As long as research in schools and nonschool settings is valued equally, researchers beginning their careers are likely to undertake the work that advances their individual careers. Similarly, as long as the rewards are equal, a study that is relatively straightforward to conduct, such as collection of survey data, will be selected in preference to projects that are more difficult and complex.

A starting point for discussion involves recognition of this professional conundrum, followed by identification of steps that may adjust the equation. One possibility may be inclusion of these issues in review criteria that are used for evaluation of submissions. Another possibility could involve addressing this through the process of mentoring new academics and graduate students, yielding increased interest in conducting relevant school-based research, and especially generalizable research involving student learning outcomes. If beginning researchers understand that significant opportunities for professional advancement are associated with this type of research – both for themselves and for the profession as a whole – we could potentially see an increased focus on this area in the future.

Complex issues demand complex research – longitudinal with multiple aspects that build on each other and build toward development of new knowledge. This will not be easily accomplished – if it were easy, we would by now have seen greater gains in student learning related to effective use of educational technology. Continued discussion of this topic is scheduled throughout the coming year at the SITE conference, NECC, and other professional associations participating in the National Technology Leadership Coalition efforts.

In the meantime, we welcome commentary and thoughts on how we might best address these important issues that we believe are crucial to the future of the profession. Thoughtful reflections that we receive will be published in future issues as we continue deliberations on this topic.

References

Conlon, T., & Simpson, M. (2003). Silicon Valley versus Silicon Glen: The impact of computers upon teaching and learning: A comparative study. British Journal of Educational Technology, 34(2), 137-150.

Cradler, J., Cradler, R., & Clarke, R. (2003). What does research mean to you? Learning and Leading with Technology, 30(8), 50-59.

Cuban, L. (2001). Oversold and underused: Reforming schools through technology 1980-2000. Cambridge, MA: Harvard University Press.

Cuban, L., Kirkpatrick, H., & Peck, C. (2001). High access and low use of technologies in high school classrooms: Explaining an apparent paradox. American Educational Research Journal, 38(4), 813-834.

Lagemann, E. (2000). An elusive science: The troubling history of educational research. Cambridge, MA: Harvard University Press.

Lagrange, J. B., Artigue, M., Laborde, C., & Trouche, L. (2001). A meta-study on IC technologies in education: Towards a multidimensional framework to tackle their integration. Paper presented at the Proceedings of the 25th Conference of the International Group for the Psychology of Mathematics Education. ERIC Document Reproduction Service No. ED 466 950.

Pollard, C., & Pollard, R. (2004/2005). Research priorities in educational technology: A Delphi study. Journal of Research on Technology in Education, 37(2), 145-160.

Roblyer, M. D., & Knezek, G. A. (2003). New millennium research for educational technology: A call for a national research agenda. Journal of Research on Technology in Education, 36(1), 60-71.

Sandholtz, J. H. (2001). Learning to teach with technology: A comparison of teacher development programs. Journal of Technology and 'Teacher Education, 9(3), 349-374.

Shavelson, R. J., & Towne, L. (2002). Scientific research in education. Washington, DC: Committee on Scientific Principles for Education Research, The National Research Council.

Strudler, N. (2003). Answering the call: A response to Roblyer and Knezek. Journal of Research on Technology in Education, 36(1), 72-76.

U. S. Department of Education. (2004). Toward a new Golden Age in American education: How the Internet, the law and today's students are revolutionizing expectations. Washington DC: Author.