Despite increased access to technology in the classroom, teaching with technology remains an instructional challenge (Aldunate & Nussbaum, 2013; Voogt, Erstad, Dede, & Mishra, 2013). Technology use in the classroom often remains “technocentrist” (Papert, 1990); that is, technology is used without an explicit instructional purpose, typically resulting in poor student outcomes (Cuban, 2013; Warschauer, Cotten, & Ames, 2011).
Recent findings suggest the availability and use of technology in the classroom has not been associated with improvement in student outcomes (Organisation for Economic Co-operation and Development, 2015). Thus, to improve classroom instruction with technology, researchers, policy makers, and practitioners need to move beyond focusing on technology itself and focus on teaching with technology (Zinger, Tate, & Warschauer, 2017).
Teacher instruction is central to the ways technology is integrated into classrooms and student learning opportunities (Aldunate & Nussbaum, 2013; Ertmer, Ottenbreit-Leftwich, & York, 2007). Furthermore, teacher professional development (PD) is a central factor of ongoing teacher education in preparing and supporting teachers as they introduce and use technological resources in their classrooms (Matzen & Edmunds, 2007).
Though extensive research has been undertaken on effective characteristics of PD, the ways teachers participate in and learn from their own education has been understudied (Kazemi & Hubbard, 2008). Thus, better understanding of how teachers engage in and learn from PD can help improve PD design and implementation. Improved PD should lead to improved teacher knowledge and teacher instruction (Antoniou & Kyriakides, 2013; Martin et al., 2010).
Recent studies have also suggested that using a design-based research (DBR) approach, which affords teachers greater input and agency in the design of PD and their own educational experience, may be a useful alternative to evaluative approaches to improvement of PD (Anderson & Shattuck, 2012). Design-based research may provide a deeper understanding of the ways PD supports teacher learning (Dede, Jass Ketelhut, Whitehouse, Breit, & McCloskey, 2008), as well as improving teacher learning outcomes from PD. DBR has further shown promise in attending to teacher instructional contexts, where curriculum designers have worked together with preservice teachers to design technology-rich lessons (Angeli & Valanides, 2009).
We took this lens as we examined a yearlong PD program designed to engage teachers in using online digital museum resources instructionally. Our goals were to provide insights on the impact of a DBR approach to PD design and implementation and improve teacher instructional knowledge. We hypothesized that the PD would improve teacher knowledge. The following questions guided our inquiry:
- What were teacher learning opportunities and how did they change over four PD sessions of a yearlong professional development series introducing an online resource, the Smithsonian Learning Lab?
- How did teacher feedback, as part of a design-based research approach, iteratively change the PD and teacher educational opportunities?
- Did teacher technological, pedagogical, and content knowledge improve through the PD program?
We looked at the affordances of the PD through the framework of the technological, pedagogical, and content knowledge (TPACK; Mishra & Koehler, 2006) with special attention to teacher learning and teaching context (Angeli & Valanides, 2009). In addition to exploring how DBR might improve technology-based PD, we attended to the understudied contextual component of TPACK (Rosenberg & Koehler, 2013).
The backdrop of our study was the Smithsonian Learning Lab (SLL), an online platform designed to provide educators with tools and resources to explore (see Figure 1) and design collections (see Figure 2) of digital museum resources to promote student learning. Recently, museums have increasingly focused on digitizing physical resources and sharing those digital resources with the public in ways that have not been previously available (Marty, 2008a). Indeed, museums are now positioned to democratize and provide unprecedented digital public access to museum-related content (Parry, 2007). Consequently, many museums are now exploring and developing digital resources to engage with the public (Marty, 2008b). Museums have also worked to develop online educational platforms that allow teachers to create their own collections of museum resources (Marty, 2011).
Review of the Literature
We broadly view PD as an intact activity system (Greeno, 2011), where teachers interact with content, instructors, facilitators, and peers in the context of their own instructional practice. Within PD, teacher learning opportunities are shaped by affordances of design and implementation of the PD. We view PD design as an ongoing, iterative process to improve teacher opportunities to learn and as an integral part of ongoing teacher education.
The initial PD design in this study was guided by many of the features of effective PD design, including duration, collective participation, content focus, and active learning opportunities (recommendations of Desimone, 2009). Extending these principles by taking a DBR approach may support improvement of PD from session to session, as well as across entire cycles of PD. The review of literature was guided by the potential and affordances of using DBR in PD, TPACK as a framework for teacher knowledge and learning through PD, and challenges in current approaches to technology-focused PD.
To address teacher instructional TPACK in context, we adopted a DBR approach, in which teachers, PD facilitators, and researchers work together to inform PD design (Anderson & Shattuck, 2012). DBR offers a useful approach for studying complex learning environments such as PDs (Dede et al., 2008).
A design-based approach, first proposed by Brown (1992) and Collins (1992) is comprised of a number of important features. At its core, DBR uses formative research to refine design and promote both practical and theoretical research outcomes (Collins, Josep, & Bielaczye, 2004). Components of DBR include its interventionist nature, taking place in naturalistic environments, and continuous iterative approach, where changes are made, assessed, and refined to improve design (Barab & Squire, 2004). Additionally, participant (in our case, teacher) feedback is integral to the changes in design (MacDonald, 2008).
Prior studies in education employing DBR approaches have primarily focused on classroom instruction (Anderson & Shattuck, 2012), yet DBR can hold promise in the design and implementation of teacher PD (Dede et al., 2008). Studies employing DBR approaches in PD have demonstrated the benefit of using this approach. Findings from these studies have shown the promising outcomes in designing and refining teacher communities of practice (MacDonald, 2008), as well as in refining PD design and improving teacher instruction and student outcomes with technology (Wang, Hsu, Reeves, & Coster, 2014). An important area for exploration of the integration of DBR and PD is the potential impact of the approach on teacher development of TPACK (Annetta et al., 2013).
Teaching and TPACK
To characterize teacher PD learning opportunities and development we used the TPACK framework (see Figure 3; Mishra & Koehler, 2006). TPACK is based on the pedagogical content knowledge framework (PCK) proposed by Shulman (1987). PCK conceptualized teacher professional knowledge at the intersection of teacher content knowledge (CK), such as how to add two numbers in mathematics, and pedagogical knowledge (PK), such as how to create assignments for students.
Shulman proposed a more specialized domain-specific knowledge that combined CK and PK as PCK, for example, how to create math word problems. In TPACK, technological knowledge is added as an additional dimension along with content and pedagogical knowledge.
TPACK highlights the unique challenges raised by digital and analog technologies and how their introduction complicates the already complex context of teaching (Koehler, Mishra, & Cain, 2013). Beyond the domains and intersections of TPACK, the context within which teachers instruct also plays an important role and has increasingly been a focus of research (Angeli & Valanides, 2009; Rosenberg & Koehler, 2013). Characteristics of the teacher’s context, such as availability of technology and school culture, could have profound impacts on the type of instruction in a classroom. TPACK self-report measures have been used widely in research (Koehler, Shin, & Mishra, 2012), and a small number have been validated (e.g., Schmidt et al., 2009).
PD and PD With Technology
PD can play a central role in teacher education through changing teacher beliefs, knowledge, and classroom practices using technology (Brinkerhoff, 2005; Schrum & Levin, 2013). Extensive prior research has established key features of PD that are likely to be effective (Desimone, 2009; Garet, Porter, Desimone, Birman, & Yoon, 2001). These features have further been situated within the realm of technology PD. Key identified features include extended duration of PD, access to technology, opportunities for the teacher to engage actively in activities in a student role, time to address individual teachers’ contextual factors, a clear vision, and time to collaborate with peers (Lawless & Pellegrino, 2007; Zinger, Tate, & Warschauer, 2017). Nonetheless, a great deal is unknown about the link between what teachers experience in PD and what they learn from it (Lawless & Pellegrino, 2007), that is. linking their experiences and what they take away from the PD.
Designing and implementing PD that promotes teacher learning is a challenging endeavor. These challenges include the delicate balance between teaching teachers the use of tools, content, and pedagogy and teacher ownership and agency that is gained through teacher practice and collaboration during PDs (Polly, 2011).
Additionally, teachers who participate in PD bring with them different skills and knowledge, both in technology and instruction (Mouza, 2009), highlighting the varied nature of backgrounds and instructional needs of teachers that PD may need to meet. Furthermore, if PD is disconnected from teacher practice, it can lead to teacher frustration and disengagement (Lim & Khine, 2006). The intersection of technology-based PD and an iterative improvement-focused DBR approach may be one avenue to address these challenges.
The SLL is an online resource created by the Smithsonian Center for Learning and Digital Access (SCLDA). It provides teachers with online access to museum artifacts, artworks, and specimens from across the Smithsonian’s museums and research centers. The SLL’s database also includes a variety of digital media, including video interviews with experts, podcasts, magazine articles and interactives. On the SLL, teachers can find existing lessons and instructional collections of Smithsonian resources, and create, organize and manage these digital resources for use in their classrooms. Furthermore, the SLL includes functionality for users to add annotations to discussion prompts, quizzes, hotspots on resources, and other collections to enhance instruction and student engagement.
The present study was based on a larger PD project designed to support teachers in the use of the SLL. The project is located in and around a large city in the eastern United States and is a collaboration between the Smithsonian, the local county department of education, 16 middle schools, a local history museum, and a large West Coast research university.
The first year of the project included teams of middle school social studies teachers, and the second year included teams of high school social studies teachers. The current study encompasses the first year of the program. The middle school social studies teachers were recruited through the local county department of education. The department of education sought teachers with a variety of experience using technology in the classroom, resulting in a wide variation in school and classroom settings. Variations between schools included availability of technology, teacher experience, and school demographics. To promote collaboration and effective PD design, teachers were recruited in teams from the various school sites. Participating teachers joined the program in June 2015, the summer prior to the PDs and implementation of the SLL.
The first year’s PD series was comprised of four daylong 8-hour face-to-face sessions at the local history museum, which is a member of the Smithsonian affiliate network. The purpose of the PD was to provide instruction to teachers on how to use the SLL to teach social studies. PD sessions took place in November and December 2015 and in February and May 2016. The PD instruction included SCLDA staff, museum curators, educators and archivists, researchers, two instructional coaches, and participating teachers. In addition to instruction, the PDs included time for collaboration among teachers within schools, as well as across schools by subject area and content. PD days included between four and seven separate sessions or segments that included direct instruction, hands-on activities and practice, time for collaboration on curriculum design, and in later PDs, breakout sessions in which teachers could choose the content and type of activities they would engage in. These sessions provided teachers with supports across the TPACK domains.
Participants and Settings
Thirty-seven teachers from 16 schools were recruited for participation in the program. Three teachers, comprising the participating population of one school, left the program after the second PD session, and one additional teacher departed before the completion of the program. A cohort of 33 teachers (21 female and 12 male) from 16 schools, thus, completed the full PD series. Participating teachers’ instructional experience ranged from 2 years of teaching to over 20, with approximately half the teachers having between 11 and 20 years of classroom experience.
Teacher experience with using technology in their instruction ranged widely. All teachers used some form of technology or online resources in the classroom, and almost all of the teachers used a wide variety of technology resources and tools. Teacher backgrounds and ongoing PD with technology also ranged widely, from little formal preparation or PD to courses dedicated to technology use and over 40 hours of technology-based PD the prior year. All but one of the teachers had access to one-to-one computers for their students, either through technology in the classroom or availability of a computer lab. The other teacher had six student computers in her classroom, where students would need to share computers.
The PD design team represented a diverse group in terms of background and expertise. The SCLDA team included one primary investigator with extensive experience in the integration of museum resources in the classroom, two education staff members who organized and oversaw the PD project, and two local instructional coaches who observed teachers’ classrooms (approximately four times during the school year per teacher) and provided further teacher support as needed. Additional support from the coaches included building model teaching collections, demonstrating or coteaching lessons, and providing constructive feedback after observing lessons that included the use of the SLL.
The museum team was led by its education manager, who had extensive experience in working with museum school partnerships. She was supported by museum curators and staff. The curriculum and instruction coordinator of the county department of education offered context for the scope and sequence required of the teachers by the state. The university research team was comprised of the other primary investigator (besides one from the SCLDA team), a research faculty member with extensive experience in instructional technology integration and learning through technology, a university history project director expert, and a graduate student with extensive experience in teacher education and PD design and implementation.
This study was primarily qualitative in methodology. Data sources included PD agendas, teacher feedback from post-PD surveys, pre- and post-PD planning and debriefing meeting notes. Quantitative measures were used for teacher TPACK surveys, as well as Likert-like teacher post-PD survey responses about the overall quality of the PD.
Characterizing Teacher Learning Opportunities From PDs. The agendas from each of the four PDs were collected to identify TPACK domain learning opportunities for teachers. Each PD had between four and seven activity segments that were coded to identify the TPACK components they addressed. Additionally, due to the central nature of context to teacher practice and TPACK, we coded each PD segment as either generic or decontextualized, contextualized to the local area of the teachers, or individually contextualized to address individual instructional needs of teachers. A total of 24 activity segments were co coded by the first and second authors. There was a disagreement on one segment’s code, and the disagreement was resolved through discussion.
Determining How Teacher Feedback Informed PD Design. After each PD, teachers were asked to complete an anonymous PD evaluation and feedback form. Teachers’ post-PD surveys were used to determine teachers’ perspectives on what was most helpful and least helpful in supporting their implementation of the SLL. Over the course of four PDs, 149 individual surveys were collected and analyzed. Qualitative responses were initially coded structurally and subsequently axially using a primary code and then a subcode (as in Saldaña, 2016).
Each response was first broken down into individual idea units (if multiple ideas were presented, they were segmented into individual units for analysis). For example, a response on what was most useful that included “overview and practice time” was broken down to “overview” and “practice time.” A total of 259 individual responses were coded.
Individual responses were then coded axially, generating a primary code and a subcode. For example, “instruction” emerged as a primary code, whereas “site use,” “building a collection,” and “using primary sources” emerged as subcodes to instruction. Each segment was then coded by the first and third authors. Any disagreements in the codes were reconciled through meetings and discussion until both coders agreed with the coding. These responses were also used to determine the types of responsiveness of the PD design team to teacher feedback. Additionally, quantitative Likert-like post-PD teacher survey responses to the overall value of the PD were collected as a second reference point for teacher overall satisfaction with the PD.
These data were collected and informed post-PD debrief meetings by the design team following the PD. Qualitative data were organized thematically by the research team and presented to the collective PD design team immediately after each PD. PD designers, PD instructors, instructional coaches, and researchers then met to analyze the PD feedback and begin subsequent PD planning.
Teacher feedback, along with feedback and perceptions of the design team, were noted. A preliminary subsequent PD agenda was generated. Before the subsequent PD, the same group would meet to formalize PD plans and agenda. Field notes were generated from the post-PD debrief meetings to determine how teacher feedback was interpreted and how it was used to plan the subsequent PD.
How Did Teachers’ Perceptions of Their TPACK Change Through the PD? To determine if and across which TPACK domains teachers improved over the course of the PD, we administered an initial TPACK survey just prior to the initial PD in November and a final TPACK survey at the end of the program in May. The survey was based on a previously validated social studies-based TPACK survey, which asks questions on a 5-point Likert-like scale (Schmidt et al., 2009).
The survey was designed to assess TPACK of preservice teachers about their instruction, as well as teacher preparation experience. It asks teachers to report their perceived knowledge across the seven TPACK domains. For example, for technological knowledge (TK) the item is worded, “I can learn technology easily.”
Our program focused on in-service teachers, so some of the questions from the original survey were removed because they were not relevant to our setting. A total of 47 item addressed teacher knowledge across the TPACK domains. Some of the items addressed preparation experience, such as “My teacher education program has caused me to think more deeply about how technology could influence the teaching approaches I use in my classroom.” Additionally, CK questions included four different subjects.
As teachers in this study were already in the classroom and came primarily from the social sciences, we removed questions that were not relevant to our teacher populations, including those relating to teacher preparations and nonsocial-science questions. The survey as administered to the teachers included 24 question from the original TPACK survey. PK and technological content knowledge (TCK) included one question each, CK included two questions, and technological pedagogical knowledge (TPK) and PCK four each, TK included five, and TPACK included seven. A total of 26 of the 33 teachers who participated in the program completed both pre- and postsurveys.
Data Analysis Approach
In the analysis, we sought first to characterize the learning opportunities for teachers during each PD and then examine the changes in learning opportunities between each PD in the context of TPACK. We then analyzed how teacher feedback informed changes in the PDs by linking teacher feedback through post-PD debriefs and pre-PD planning meetings to the subsequent PD. Our DBR framed analysis approach is illustrated in Figure 4.
We then analyzed teacher pre- and post-TPACK scores from the beginning of the PD sequence to the end to determine (a) if collective growth occurred in specific areas of TPACK over time and (b) if so, in which knowledge areas.
PD Learning Opportunities and Changes. Teacher TPACK was coded across TK, PK, CK, TCK, TPK, PCK, and TPACK. We then coded contextual affordances by each PD segment. That is, we determined if the segment addressed teachers at their own practical classroom level, at a local, regional, or community level, or neither. Changes in teacher learning opportunities from each PD were then analyzed. The analysis of individual PDs focused on shifts in TPACK components over the four PDs and changes in levels of contexts across the four PDs.
How Teacher Feedback Informed Changes to the PD. Codes generated from teacher feedback were aligned with TPACK affordances of PD segments. For example, in PD 1 the code “collection creation” was identified as the most useful component of the PD (by 12 respondents). This code was then aligned with the small group collection creation activity. Collection creation involved assembling individual SLL resources in a group, which reflected TCK as well as the opportunity for small group work, in which teachers had an opportunity to create collections.
The same was done with teacher-reported least useful components of the PD; however, in some cases, the least useful components extended beyond the PD. For example, the least beneficial item identified in the first PD was SLL user experience, which is reflective of the SLL site itself rather than the PD. It should be noted that the SLL was officially launched in beta form in early November 2015, coinciding with the first PD. Continual improvements to the SLL’s user experience were made based on teacher feedback. This information was used by the PD design team to inform future PDs and improve the teacher educational experience.
How Teacher TPACK Changed Through the PD. To determine if and across which TPACK domains teachers improved over the course of the PD, we examined teacher results descriptively to identify changes in individual teacher pre and post scores. We next conducted a one-tailed t-test on teacher aggregate scores across all TPACK domains based on their pre- and postsurveys. That is, we combined the survey questions for each domain and averaged them across the cohort.
We then conducted a one-tailed t-test across each of the seven TPACK domains between the first and second administration of the survey for all participants with a pre- and posttest (N = 26). We conducted a one-tailed test, as we initially hypothesized that the PD would improve teacher knowledge. Domains that showed significant growth were then identified.
Findings are presented chronologically, as the PD program developed iteratively over the course of the four PD sessions. As the PD developed, individual sessions moved from focusing on single dimensions of TPACK in decontextualized ways to addressing learners’ needs in the more complex intersectional dimensions of TPACK in more contextualized ways (see Table 1). Teacher-reported barriers to implementation and multiple sources of data from the PD design team emerged as significant factors in changing the design of the PD. Teachers’ perceptions of their TPACK competence improved across five of the seven dimensions from the beginning to end of the program.
PD Teacher TPACK Learning Affordances
In the first PD, the project and purpose of the SLL were introduced. Teachers initially engaged in a locally contextualized activity, in which they had to select a single resource from a collection of 40 physical, analog resource cards connected to their city. Teachers then shared why they selected a resource and how it connected to them locally and individually.
Each of the other activities in the first PD were generic and primarily focused on use of the SLL. For example, teachers engaged in a “nightstand” activity, inspired by a local artist’s work, where they searched the SLL to find symbolic objects that might be on their own nightstand as a short biographical exercise. This activity was designed to help teachers understand how to add resources to a collection and how to use other features of the SLL but did not connect directly to their instruction or teaching context.
Beginning in the first PD and persisting through all PDs, time was provided for teachers to independently develop their own collections, with SLL and PD design team members available for support and brainstorming. In the first PD, this task was generic. Teachers were asked to brainstorm ideas for collections with little specificity or focus.
PD 1 to PD 2 Iteration
We identified two key areas to be addressed based on teacher feedback after the first PD: greater individualization of the PD and more contextualized learning experiences. The teacher post-PD surveys indicated mixed responses to the utility of activities in this PD. Initially, teacher feedback appeared contradictory. For example, 12 teachers found collection creation and use most beneficial, and seven teachers found it least beneficial. That is, whereas some teachers found certain instructional approaches and content helpful, others found it less beneficial, highlighting the need to meet teacher learning needs in more targeted ways. This issue was acknowledged by one of the coaches who noted a “wide range of teacher characteristics, some ready to do a unit, some not quite ready” during the debrief meeting.
The generic and decontextualized nature of activities also emerged as an issue from this feedback, as well as the need for greater individualization and specialization of sessions to meet the varied needs of teachers. The PD design group saw the need for greater individualization as a challenge to be addressed, so we considered breakout sessions for more individualized and targeted instruction in the second PD.
A finer grained analysis of teacher feedback highlighted that the decontextualized, generic nature of the activities was the crux of many of the teacher critiques. As one noted, they did not like “the prescribed creation of an example that can’t be used in the classroom.” Other teachers also noted the importance of connecting activities to their classroom practice: “The most beneficial part of today was the opening exercise where every teacher explained how they might use the images with their students.” This feedback led the PD design team to ask questions in consideration for the subsequent PD, including the integration of the SLL and instruction, and finding more contextualized ways of implementing the SLL. Consideration was given to tighter integration of the SLL as a pedagogical tool and a source of content.
Teacher feedback about the functionality of the SLL itself had implications for the PD as well. The primary critique of the PD reported by 10 teachers focused on SLL issues of searching for resources and site functionality. Though these tool-centered issues could not be directly addressed by the PD design team, the group acknowledged their importance as well as the value of reiterating to the teachers that the SLL was a beta version and that their feedback was helping inform its design. This issue continued to be raised by teachers but did not directly reflect on the PD.
Of note is the short 1-month time gap between PD 1 and PD 2, which limited the adjustments that could be made between them. A 2-month gap existed between the other PDs that allowed for more teacher implementation of the SLL and feedback from coaches’ observations.
In the second PD breakout sessions, in which teachers selected a topic or area of interest and attended smaller sessions, were implemented. This change signified a major shift from the first PD. In breakout sessions, participating teachers could attend one of five sessions that covered topics ranging from the analysis of portraiture, to civil war diaries, to artifacts from ancient civilizations. Nonetheless, all sessions focused on the use of primary sources for instruction. Thus, breakout sessions offered teachers content choices and more individualization than in the first PD but not pedagogical choices, and the sessions did not attend to classroom contexts.
In the second PD, time for teacher planning became more focused (through a prompt on examining primary sources) but remained decontextualized from teachers’ own classrooms and subject matter.
Two of the sessions in the second PD were dedicated to TK, teaching how to use different features and functions of the SLL, including searching for resources and how to create student rosters (see Table 1). The PD, however, also shifted to more complex components of TPACK, such as PCK, TCK, and TPACK, but all sessions remained generic and decontextualized from teachers’ own classrooms. For example, the opening activity of the day, coded TPACK generic, had teachers work in subject area groups creating collections that reflected innovations. The activity engaged teachers in building teaching collections and considering pedagogy and content along with technology but did not target or address instructional needs of teachers in their own classrooms.
PD 2 to PD 3 Iteration
The iteration from the second to third PDs encompassed changes and adaptation between December and February. During this time, coaches had an opportunity to observe participants’ classrooms and gather data on how teachers were using the SLL, which helped bring additional insight into teacher needs and practice to the post-PD discussion. Positive teacher feedback on the utility of the second PD fell into two primary categories: instruction on using different functions of the SLL, reported by 10 teachers, and instruction on refining searches and using search reports, reported by seven teachers.
The instruction on searching was implemented in direct response to teacher feedback from the prior PD on their challenges. Nonetheless, teachers also reported that the utility of the search and some of the tools on the SLL remained a barrier to use, as reported by 12 teachers.
Although the PD helped teachers better understand site functionality, technical challenges remained. As one teacher pointed out in the post-PD survey, “I really shouldn’t need a two-page document to know how to search the SLL,” which was a critique of the limitations of the search feature rather than the PD session on searching tips. The teacher went on to point out that “the lack of relevant search results when executing a basic search is going to turn off educators pretty quickly.”
This feedback helped the PD design team better understand that the challenges around search were not simply a function of learning how to use the search feature, but were either the lack of, or overwhelming number of, results that searches produced. It also helped the PD team better understand the limitations and affordances they could provide for teachers in the context of the PD. That is, the PD team understood its limited control over the SLL’s functionality itself beyond providing feedback to the SLL web design team. The PD team could, however, support teachers by acknowledging limitations of the SLL and providing solutions to the challenges raised by teachers from an instructional perspective.
The first two PDs primarily focused on function and use of the SLL from a more technical perspective. Of the 13 segments in the first two PDs, four were coded as TK. After the second PD, the design team acknowledged that the cohort of teachers had developed a good understanding of the SLL and its functionality. Consequently, we agreed that additional PD focused on technology itself would not be useful. From teacher feedback, the importance of time to work on collections with experts and their peers emerged as other important features (reported by a total of five teachers). Nonetheless, teacher feedback continued to be varied and at times appeared to be contradictory, highlighting the importance of contextualization, and focused on more individualized, targeted PD sessions to address both content area interests as well as pedagogical practices.
The two key takeaways for the PD design team from teacher feedback in the second PD were the importance of contextualizing PD sessions to meet the local and individual classroom needs of teachers and the importance of more targeted session options.
The PD design team focused on two ways to address these teacher needs. First, more breakout sessions covering a wide range of pedagogies and content areas would be introduced in the subsequent PD. Second, time for teachers to conceptualize and create collections, as well as providing teachers an opportunity to share with the group their successes, failures, and challenges would be incorporated. The importance of providing teachers with structures to focus their collection design in ways that were relevant to their classroom became more central to the design team moving beyond the second PD.
The third PD marked a significant shift in the content and context of the PDs. Whereas, in the first two PDs only one of 13 segments was coded as contextualized, in February five of the seven segments were contextualized. The breakout sessions in the third PD focused on using digital objects or practices in the specific instructional contexts of each teacher. The sessions in the third PD focused less on the technology of the SLL and more on thoughtful planning of instruction to promote student historical thinking, evidenced by multiple PCK-coded segments along with more targeted and focused breakout sessions. The February session also represented the first time teachers shared their own created collection, successes, and challenges in the implementation of the SLL in their classrooms, another contextualized activity.
In the third PD, for the first time in one of the breakout sessions, a museum archivist presented a collection of documents located in the museum and connected to the local community. Seventeen teachers identified this activity as most beneficial. This session involved both a presentation, examination of historical documents in the role of the student and time for teachers to work on how they could use these resources in instruction.
The documents entailed the escape from Nazi Germany by a Jewish family and their journey and settling in the local community. Documents investigated included ship boarding passes, mail correspondence, immigration documents, and a narrative of their curation. Teachers were struck by the fact that the house that the family moved to still stood in their community. Teachers enjoyed the experience of working with archival documents connected to their local community. They also saw the collection and related activities as something they could use in their own classrooms, as one said: “I feel like an adapted version of this would be able to be used in my classroom.”
The other breakout session of the day focused on using a single museum resource to base an entire collection on that teachers would build for their classroom. Though both of these sessions were coded as PCK local (both used locally connected museum resources), they presented different pedagogical experiences for teachers. One breakout focused on building an archival narrative, and the other sessions focused on building on teacher instructional use of documents. This differentiation presented a step forward in the degree of individualization of breakout sessions beyond content.
PD 3 to PD 4 Iteration
The third PD represented a significant shift in teacher responses as well as the mindset of the PD design team. The shift in teacher feedback represented a move in the PD’s design away from the technical to greater focus on instruction, as well as consensus on the utility of the PD. This feedback led the PD design team to view future changes to the PD as refinements rather than major changes, paying greater attention to finer details rather than making large-scale changes. This shift was reinforced by the quantitative feedback from teachers, which was also the most positive of any of the PDs. Teachers reported their overall satisfaction with the PD to be 4.37 on a 5-point scale. This shift also aligns with theory on the use of design-based principles and evaluative studies (Supovitz, 2013), where PD transitions from a more broad-based refinement to a finer-grained refinement as it is iteratively developed. In its third iteration, the PD program had reached the point of transition from significant iterative change to more refined change.
Teachers identified a wide range of sessions as most useful, likely reflecting that individual breakout sessions better addressed their needs than larger, more generic sessions. Furthermore, this was the first PD where a large proportion of teachers (18 out of 38) either explicitly stated that they did not find any of the PD components “least beneficial” or stated that they liked the PD in its entirety. One teacher reported, “I think everything was useful today. I have no constructive criticism.” Another stated, “Today was helpful all around.” Critiques from teachers continued to focus on the technical functionality of the SLL, though fewer of these critiques appeared than in previous PDs. Teacher feedback from the third PD also saw a significant shift toward more instruction-focused challenges and needs from technical ones.
The PD design team took the positive feedback as reinforcement for the increasingly targeted breakout sessions and contextualization of the third PD. Additionally, sessions providing teachers with time to work within school teams, and teacher presentations of SLL implementation worked to contextualize use of the SLL and localize it closer to their classes. Based on the feedback, the PD design team decided that continuing these types of activities was important as was building on structures from the third PD that linked the SLL to specific instructional goals and activities. Indeed, the team decided to refine the targeted breakout sessions by extending them to a second local museum (with a focus on ancient Egyptian civilization for non-U.S. history teachers) to provide a more experiential learning environment that aligned with teachers’ interests.
Additionally, to better capture the utility of each individual breakout session, the post-PD evaluation form for the fourth PD was also refined to identify which breakout session teachers attended. These data would provide a more accurate picture of any potential differences in the utility of individual breakout sessions.
The final PD began with demonstrating some of the new support systems built into the SLL and soliciting feedback. This activity was decontextualized and focused on TK, but the support systems built were in response to issues that had been raised by the participating teachers. The remaining sessions of the day were contextualized and coded as PCK focused.
On the final PD, the breakout sessions were further focused and targeted, with teachers given the option of staying at the museum and engaging in activities with archival documents pertaining to U.S. history or traveling to a nearby affiliate museum to explore an exhibition on Ancient Egypt. Both options centered on the formulation of a guiding question that would then lead to building a lesson in a subsequent session. T
hese breakout sessions were coded as PCK and individualized, as teachers worked on creating lessons integrating content and pedagogy for their own classrooms. Focus groups took place in the middle of the day to elicit insight into teacher use of the SLL and on the supports provided in their implementation of the SLL. The day concluded with teachers sharing their classroom activities as well as addressing design and lesson development considerations.
Post-PD 4 and Next Steps
After the final PD, the design team reconvened to debrief on PD 4 and the entire PD series and make initial plans for the subsequent year’s PD (a new cohort of teachers was planned for the PD for the following year). The final PD had the lowest rate of teacher PD feedback (22 of 33 or 67%). The PD design team attributed this result to the PD being close to the end of the school year and the PD being the last one of the year. Nonetheless, qualitative feedback from the final PD was similar to that of the third PD.
Three categories emerged as most useful for teachers, where seven reported the collaboration with peers as most useful, six reported the opportunity to work with museum artifacts, and five reported learning of news ways to use the SLL as most useful. Twelve respondents (over half who responded) either did not identify any element of the PD as least beneficial, or posted a positive comment about the PD. Two respondents wished that more time had been available to work, and most critiques focused on limitations of the SLL, such as limited resources on some topics.
The PD design team used findings from the first year’s PD to inform the design of the subsequent year’s PD. The need to contextualize learning for teachers and to provide multiple breakout sessions to address the diverse needs of teachers were central to the design. A focus on historical thinking emerged as a mode by which the following year’s PD would become more contextualized, as well as attending to participant PCK to a greater degree initially.
Some of the breakout sessions from the first year’s PD were adapted for the second year and others were to be reworked or changed. The overall format of the PD was also changed, with the first two PDs moving from November and December to August and November. This change both positioned teachers to plan instruction with the SLL prior to the beginning of the school year and allowed coaches and researchers more time to observe teachers using the SLL earlier in the school year.
Teacher Change in TPACK
Overall average teacher-reported TPACK scores increased from 4.23 to 4.61 (nearly 1 full standard deviation) on a scale from 1 (strongly disagree) to 5 (strongly agree). Individual overall teacher change scores ranged from -1.04 to +1.64. Five teachers’ scores decreased from pre- to postsurveys (three had averaged a score of five across the domains on the presurvey), three teachers were unchanged (two of these averaged a score of 5 in both the pre- and post-surveys), and 18 teachers’ scores increased
Within the TPACK domains, on average, teachers improved significantly across five of the seven domains (see Table 3). The largest changes of .53 and .58, were experienced in the PCK and TCK domains, respectively. TK increased, and its intersection with primary knowledge domains (TCK, TPK, TPACK) all increased. That is, on average, teachers reported that they improved their social studies domain-specific pedagogies, as well as technological integration and instruction. Both of these were central foci of the SLL PDs, namely, using the SLL as a means for finding and organizing digital resources for instruction and improving teacher instruction in the use of historical primary sources. The overall teacher CK and PK did not increase significantly. That is, teachers’ social studies content and general pedagogical knowledge did not improve over the course of the year. These scores also started as the highest on the presurvey.
TPACK Domain Self-Report Results
|TPACK Domains||Pre Mean||Pre SD||Post Mean||Post SD||p-value|
|Note: Significance assessed using 1 tailed t-test. PK and TCK domains are composed of a single question.
* p < .05, ** p < .01, *** p < .001
In this study, we examined the instructional affordances for teachers participating in a PD and how their feedback, as part of a DBR approach, helped evolve the PD program over four PD sessions and ultimately resulted in teacher TPACK gains. Direct teacher feedback from the PDs and indirect feedback through observation by coaches and researchers played a central role in informing and prompting changes to the PD design and implementation. This study was small scale and largely qualitative, so we cannot directly link teacher increase in TPACK to the PD. We did find, however, that as the PD format and content changed in response to their feedback, teachers’ satisfaction with the PD increased as well.
Two implications for teacher educators arose from this study. First, the DBR approach to PD design helped calibrate the transition from instruction focused on TK in decontextualized ways to specialized instruction and more contextualized learning experiences focused on PCK, refining teacher educational opportunities. This approach helped negotiate the balance and transition between instruction and teacher ownership (as in Polly, 2011), where teacher feedback helped the design team increase teacher involvement beginning with the third PD.
Feedback and changes made by the PD design team also allowed for rapid iterative changes along the intervention development curve (as noted also in Supovitz, 2013). That is, the first two PDs led to significant large-scale changes in the third PD, where there was a shift to more refined data gathering and program refinement. Indeed, had we not taken this iterative approach, changes could not likely have been devised and implemented as quickly, and teacher engagement and learning likely would have been lower. Had the first two PDs been spaced an additional month apart, a more rapid evolution of the PD series may have been possible. A 2-month window between PDs better served data gathering, analysis, and change implementation of the PDs.
Second, PD breakout sessions helped address some of the challenges and critiques of technology-based PD that tends to be “one size fits all” (Schrum, 1999). Additionally, the use of specialized hands-on instruction through the breakout sessions helped promote teacher engagement and learning in ways that were more contextualized to their own teaching environments, which has been associated with improved instruction and student outcomes (Schrum & Levin, 2013). This instruction was primarily accomplished by identifying teacher needs and designing sessions to meet those needs. Though breakout sessions have been used in prior studies on PD (e.g., Tuttle et al., 2016), our study highlights how teacher feedback as part of a DBR approach can help establish, identify content for, and refine breakout sessions to improve teacher learning.
Breakout sessions provided PD designers with opportunities to integrate the use of SLL-based tools that the PD was promoting in ways that made the tools more contextualized, relevant, and accessible to the teachers. Teachers selected sessions to attend where the SLL was integrated in different ways that afforded greater alignment with teachers’ existing instructional practices and learning goals.
The introduction of breakout sessions addressed some of the apparently contradictory feedback to larger decontextualized sessions. Breakout sessions also led to a refinement of our design and evaluation approach to the PD. Given the varied areas of expertise of the PD design and instruction team, developing specialized breakout sessions did not create significant additional strain or work for the group. Indeed, breakout sessions may be an effective and efficient way to meet the diverse needs of teachers in PD programs when informed by teacher needs.
Implications for Practice
A number of practical implications also arose from this study, building on the ideas of researcher-practitioner partnerships (Supovitz, 2013), moving beyond the researcher-PD instructor relationship. Identifying teacher needs prior to and during PD through surveys, as well through classroom observations when possible, can add additional important perspectives to the PD design. Refining teacher response-gathering tools through the PD program can also help better pinpoint teacher needs as more generic tools become less useful. We found this to be the case with our post-PD teacher survey. Developing an inclusive, routinized, and systematic way to debrief from and plan PDs provided consistency for the PD design team that promoted inclusion, diverse thinking, and data-based decision making. Taking up these practices should have positive outcomes on PD design and implementation.
An extension to Supovitz’s (2013) heuristic of PD change may also be conceptualized, whereby taking a productive DBR approach to PD can link the responsiveness of PD and its instructional affordances for participating teachers (see Figure 5). That is, in productive PD development, the PD design and instruction team can quickly move to change the PD to meet the needs of participants (in our case over two iterations) and then make finer grained changes.
In our case the third PD represented an improvement over the second, but the fourth was less effective in improving learning opportunities for participants. This continuing experimentation refines the PD design and, at times, may incrementally improve learning affordances. At other times, though, it may not. Nonetheless, this ongoing effort to continually improve promotes the ongoing high quality of the PD.
Our study also highlights how technology-focused teacher educators and PD designers would benefit from being mindful of the fine balance between building teacher capacity to use technology and to meet teacher classroom instructional needs. In our study quickly moving from teaching how to use the technology to teaching how to teach with the technology was important. The PD team learned that contextualizing the technology to improve student learning experiences in the classroom was critical to teacher engagement and use of the SLL. This finding also suggests that to engage teachers successfully in technology-focused PD and instruction, PD should address teacher classroom instructional needs in the content and pedagogical realms.
Limitations and Future Direction
This study encompassed the first year of a 2-year PD program and helped inform the design of the first year’s PD. A number of limitations exist in the design of this initial study. The study could have benefited from a comparative control group that would have allowed us to make more causal arguments about the impact of the PD on teacher learning. Additionally, our study does not link change in teacher knowledge to change in classroom practice, which could strengthen an argument for this PD design approach. Future studies that address these two limitations would shed additional light on the effectiveness of taking design-based approaches to PD.
We intend to address one of these limitations and extend this line of research by implementing the design changes from the first year’s PD in the second year of the PD program. This longitudinal approach will help examine what a second cycle of PD design with a DBR approach affords in terms of design and teacher learning. Additionally, observations of a subset of teachers as well as teacher implementation logs will be used in Year 2 to identify potential changes in teacher practice as a result of the PD program. This approach has the potential of linking the PD to teacher practice and instructional affordances, in addition to prompting improvements in teachers’ TPACK.
Aldunate, R., & Nussbaum, M. (2013). Teacher adoption of technology. Computers in Human Behavior, 29(3), 519-524. doi: 10.1016/j.chb.2012.10.017
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16-25. doi: 10.3102/0013189X11428813
Angeli, C., & Valanides, N. (2009). Epistemological and methodological issues for the conceptualization, development, and assessment of ICT–TPCK: Advances in technological pedagogical content knowledge (TPCK). Computers & Education, 52, 154-168. doi: 10.1016/j.compedu.2008.07.006
Annetta, L. A., Frazier, W. M., Folta, E., Holmes, S., Lamb, R., & Cheng, M. T. (2013). Science teacher efficacy and extrinsic factors toward professional development using video games in a design-based research model: The next generation of STEM learning. Journal of Science Education and Technology, 22(1), 47-61. doi: 10.1007/s10956-012-9375-y
Antoniou, P., & Kyriakides, L. (2013). A dynamic integrated approach to teacher professional development: Impact and sustainability of the effects on improving teacher behaviour and student outcomes. Teaching and Teacher Education, 29(1), 1-12. doi: 10.1016/j.tate.2012.08.001
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. Journal of the Learning Sciences, 13(1), 1-14. doi: 10.1207/s15327809jls1301_1
Brinkerhoff, J. (2005). Effects of a long-duration, professional development academy on technology skills, computer self-efficacy, and technology integration beliefs and practices. Journal of Research on Technology in Education, 39(1), 22-43. doi: 10.1080/15391523.2006.10782471
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of Learning Sciences, 2, 141-178. doi: 10.1207/s15327809jls0202
Collins, A. (1992). Towards a design science of education. New Directions in Educational Technology, 15-22.
Collins, A., Josep, D., & Bielaczye, K. (2004). Design research: Theoretical and methodological issues. The Journal of the Learning Sciences, 13(1), 15-42.
Cuban, L. (2013, December 6). A second look at the iPad program at LAUSD (Weblog). Retrieved from https://larrycuban.wordpress.com/2013/12/06/a-second-look-at-ipads-in-los-angeles
Dede, C., Jass Ketelhut, D., Whitehouse, P., Breit, L., & McCloskey, E. (2008). A research agenda for online teacher professional development. Journal of Teacher Education, 60(1), 8-19. doi: 10.1177/0022487108327554
Desimone, L. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181-199. doi: 10.3102/0013189X08331140
Ertmer, P. A., Ottenbreit-Leftwich, A., & York, C. S. (2007). Exemplary technology-using teachers: Perceptions of factors influencing success. Journal of Computing in Teacher Education, 23(2), 55-61.
Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development efective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915-945. doi: 10.3102/00028312038004915
Greeno, J. (2011). A situative perspective on cognition and learning in interaction. In T. Koschmann (Ed.), Theories of learning and studies of instructional practice (pp. 359-383). New York, NY: Springer.
Kazemi, E., & Hubbard, A. (2008). New directions for the design and study of professional development: Attending to the coevolution of teachers’ participation across contexts. Journal of Teacher Education, 59, 428-441. doi: 10.1177/0022487108324330
Koehler, M. J., Mishra, P., & Cain, W. (2013). What is technological pedagogical content knowledge (TPACK)? The Journal of Education, 193(3), 13–19. Retrieved from http://www.jstor.org/stable/24636917
Koehler, M. J., Shin, T. S., & Mishra, P. (2012). How do we measure TPACK? Let me count the ways. In R. N. Ronau, C. R. Rakes, & M. L. Niess (Eds.), Educational technology, teacher knowledge, and cassroom impact: A research handbook on frameworks and approaches (pp. 16-31). Hershey, PA: IGI Global.
Lawless, K. A., & Pellegrino, J. W. (2007). Professional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of Educational Research, 77(4), 575-614. doi: 10.3102/0034654307309921
Lim, C. P., & Khine, M. (2006). Managing teachers’ barriers to ICT integration in Singapore schools. Journal of Technology and Teacher Education, 14(1), 97-125.
MacDonald, R. (2008). Professional development for information communication technology integration. Journal of Research on Technology in Education, 40(4), 429-445. doi: 10.1080/15391523.2008.10782515
Martin, W., Strother, S., Beglau, M., Bates, L., Reitzes, T., & Culp, K. M. M. (2010). Connecting instructional technology professional development to teacher and student outcomes. Journal of Research on Technology in Education, 43(1), 53-74. doi: 10.1080/15391523.2010.10782561
Marty, P. F. (2008a). An introduction to digital convergence: Libraries, archives, and museums in the information age. Archival Science, 8(4), 247-250. doi: 10.1007/s10502-009-9094-1
Marty, P. F. (2008b). Museum websites and museum visitors: Digital museum resources and their use. Museum Management and Curatorship, 23(1), 81-99. doi: 10.1080/09647770701865410
Marty, P. F. (2011). My lost museum: User expectations and motivations for creating personal digital collections on museum websites. Library and Information Science Research, 33(3), 211-219. doi: 10.1016/j.lisr.2010.11.003
Matzen, N. J., & Edmunds, J. A. (2007). Technology as a catalyst for change: The role of professional development. Journal of Research on Technology in Education, 39(4), 417-430.
Mishra, P., & Koehler, M. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017-1054. doi: 10.1111/j.1467-9620.2006.00684.x
Mouza, C. (2009). Does research-based professional development make a difference? A longitudinal investigation of teacher learning in technology integration. The Teachers College Record, 111(5), 1195-1241. Retrieved from http://www.tcrecord.org/content.asp?contentid=15479
Organisation for Economic Co-operation and Development. (2015). Students, Computers and Learning. Paris, FR: OECD Publishing. doi: 10.1787/9789264239555-en
Papert, S. (1990). A critique of technocentrism in thinking about the school of the future. Retrieved from http://www.papert.org/articles/ACritiqueofTechnocentrism.html
Parry, R. (2007). Recording the museum digital heritage and the technologies of change. New York, NY: Routledge.
Polly, D. (2011). Teachers’ learning while constructing technology-based instructional resources. British Journal of Educational Technology, 42(6), 950-961. doi: 10.1111/j.1467-8535.2010.01161.x
Rosenberg, J. M., & Koehler, M. J. (2013). Context and technological pedagogical content knowledge: A content analysis. Journal of Research on Technology in Education, 47(3), 186–210. doi: 10.1080/15391523.2015.1052663
Saldaña, J. (2016). The coding manual for qualitative researchers (3rd ed.). London. UK: SAGE.
Schmidt, D. a, Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009). Technological pedagogical content knowledge (TPACK): The development and validation of an assessment instrument for preservice teachers. Journal of Research on Technology in Education, 42(2), 123-149. doi: 10.1007/978-1-60761-303-9
Schrum, L. (1999). Technology professional development for teachers. Educational Technology Research and Development, 47(4), 83-90. doi: 10.1007/BF02299599
Schrum, L., & Levin, B. B. (2013). Teachers’ technology professional development: Lessons learned from exemplary schools. TechTrends, 57(1), 38-42. doi: 10.1007/s11528-012-0629-6
Shulman, L. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1-23.
Supovitz, J. A. (2013). Situated research design and methodological choices in formative program evaluation. National Society for the Study of Education, 112(2), 372-399.
Tuttle, N., Kaderavek, J. N., Molitor, S., Czerniak, C. M., Johnson-Whitt, E., Bloomquist, D., … Wilson, G. (2016). Investigating the impact of NGSS-aligned professional development on preK-3 teachers’ science content knowledge and pedagogy. Journal of Science Teacher Education, 27(7), 717–745. doi: 10.1007/s10972-016-9484-1
Voogt, J., Erstad, O., Dede, C., & Mishra, P. (2013). Challenges to learning and schooling in the digital networked world of the 21st century. Journal of Computer Assisted Learning, 29(5), 403-413. doi: 10.1111/jcal.12029
Wang, S. K., Hsu, H. Y., Reeves, T. C., & Coster, D. C. (2014). Professional development to enhance teachers’ practices in using information and communication technologies (ICTs) as cognitive tools: Lessons learned from a design-based research study. Computers and Education, 79, 101-115. doi: 10.1016/j.compedu.2014.07.006
Warschauer, M., Cotten, S. R., & Ames, M. G. (2011). One laptop per child Birmingham: Case study of a radical experiment. International Journal of Learning, 3(2), 61-76. doi: 10.1162/IJLM
Zinger, D., Tate, T., & Warschauer, M. (2017). Learning and teaching with digital media: Technological pedagogy and classroom practice. In J. Clandinin & J. Husu (Eds.), The SAGE handbook of research on teacher education (577-593). London, UK: Sage.