Adams, S., Burns, P., & Martin-Hansen, L. (2018). Youth views of science and engineering in a program for preparing teachers to use educational technology in STEM education. Contemporary Issues in Technology and Teacher Education, 18(3). Retrieved from

Youth Views of Science and Engineering in a Program for Preparing Teachers to Use Educational Technology in STEM Education

by Stephen Adams, California State University, Long Beach; Paul Burns, California State University, Long Beach; & Lisa Martin-Hansen, California State University, Long Beach


As part of a graduate course for supporting K-12 teachers’ use of technology in teaching science, technology, engineering, and mathematics (STEM) subjects, teachers worked in teams to create workshops for youth at a Boys & Girls Club site. Teachers used curriculum kits from the Engineering is Elementary project of the Museum of Science, Boston, together with technological resources including iPads, to plan and conduct workshops with four sessions of 8 hours each. A mixed-methods evaluation examined perceptions of 36 youth regarding science and engineering. The youth (Grades 2 to 8) self-identified as 47% African-American, 33% Hispanic/Latino, 3% Asian, and 17% as other/Caucasian/mixed ethnicity. After the workshops, boys and girls more strongly agreed with an engineering-related question, that they liked thinking of new and better ways of doing things, and they agreed more strongly that they knew what scientists did for their jobs. Also after the workshops, girls more strongly agreed they knew what engineers did for their jobs, reaching a similar level as boys, whose responses did not change significantly.  Focus group data aligned with the survey responses for most questions.  Overall, the study suggested benefits of the program to participating youth, an indicator supporting this teacher preparation model.

Teachers are being asked to give more attention to teaching engineering, integrated with science and other disciplines (National Research Council, 2013, Appendix A, p. 4). They are also expected to productively integrate educational technologies into their teaching (International Society for Technology in Education [ISTE], 2017a). In addition, calls are increasing for instructional approaches that help students better understand the nature of work in science, technology, engineering, and mathematics (STEM) professions (Capobianco, Diefes-Dux, Mena, & Weller, 2011; Oware, Capobianco, & Diefes-Dux, 2007).

This paper reports a promising approach for teacher development that addresses these goals — an approach that additionally aims to serve youth from diverse communities.  As part of a course for supporting teachers’ use of technology in STEM subjects, preservice and in-service teachers developed and implemented STEM workshops that emphasized engineering for youth at a Boys & Girls Club site.

Too often, approaches to teach ways to integrate educational technologies in instruction are technocentric, focusing on software and tools rather than curricula (Harris, Mishra, & Koehler, 2009).  In the program we describe here, technologies were not introduced in isolation or as an end in and of themselves, but as part of an overall program of STEM instruction that uses project-based learning (PBL). PBL has useful attributes for supporting STEM instruction.  It can provide authentic contexts with real-world and cognitively complex problems, help students link current and new knowledge, and provide a motivating social context. Further, educational technologies align well with PBL: They can be used as a tool for analyzing problems and constructing and sharing artifacts (Blumenfeld et al., 1991). Although teachers are encouraged to use PBL and to teach STEM subjects, they may have limited preparation to do this.

The educational program we describe, Technology and Engineering for Students and Teachers (TEST), offers teachers a setting to experience using PBL with youth, using technology as a support. Previous research on the TEST program found that the approach was beneficial to participating teachers (Adams & Bernal, 2016; Adams, Bernal, Cole-Jackson, & Martin-Hansen, 2015). In particular, teachers showed gains in developing their technology, pedagogy, and content knowledge (TPACK; Schmidt et al., 2009).

We hypothesized that, in addition to benefiting the teachers, the program would benefit the youth. With engaging instruction in science in informal settings, gains can be made in science knowledge as well as in attitudes about science (Brossard, Lewenstein, & Bonney, 2005).  Informal science programs may help to stimulate participants’ interests in science and enhance their views of science-related career options (Bell, Lewenstein, Shouse, & Feder, 2009). Moreover, STEM programs can help support positive attitudes toward STEM disciplines (Christensen, Knezek, & Tyler-Wood, 2014; Hayden, Ouyang, Scinski, Olszewski, & Bielefeldt, 2011; Lou, Shih, Diez, & Tseng, 2011).

Another goal of the TEST program is to offer participating youth further time with STEM instruction. In the US, time for science instruction can be limited. A survey of over 7,000 mathematics and science instructors in the US found that only 35% of students in Grades 4-6 receive science instruction on all or most days (Baniflower et al., 2013, p. 58).

The issue of limited time for science instruction can be exacerbated for underserved students, whose teachers may feel pressure to spend more minutes on literacy and mathematics. Lack of time spent in science learning translates into lack of opportunity to learn science. Increasing dedicated time and student-centered engagement in science learning is one step toward promoting greater equity (Tate, 2001).

This paper relates to both of the major goals of the TEST project: educating teachers and serving community youth at a Boys & Girls Club. The paper describes the program for teachers and reports on a study of attitudes toward STEM subjects and STEM careers among youth participating in the project. The study informs whether teachers’ teaching, in fact, translates to observable benefits to the youth.  Following is a description of the background and theoretical perspectives supporting the design of the TEST program, as well as perspectives pertinent to the study of youth. Next, the program for teachers is described followed by the study of youth participants.


The goal of the TEST project is for teachers to gain experience using technology in project-based learning in STEM education with an emphasis on engineering, while also supporting STEM learning for youth in a Boys & Girls Club. The following sections describe considerations and design principles that guided the development of the educational program for teachers as well as the study of outcomes for youth.

Support Active and Deep Learning

ISTE’s (2017c) Standards for Educators advocate using technology to support “active, deep learning,” a goal with roots in Dewey’s earlier advocacy of active learning (Dewey & Dewey, 1915).  ISTE standards further advocate “leveraging digital tools and resources so students can gain mastery of content area knowledge while also gaining vital competencies, including problem solving, critical thinking, effective communication, [and] collaboration…” (ISTE, 2017c NEED A LOCATION FOR THIS QUOTATION).   In this vein, the TEST program engaged teachers in facilitating engineering activities that used technology and required the active participation of youth.

Support Attitudes Useful for Learning in STEM Subjects

In connection with active learning, the ISTE Standards for Educators advocate that teachers should foster students’ “self-direction and belief in their ability to grow and improve with hard work and perseverance” (ISTE, 2017c NEED A LOCATION FOR THIS QUOTATION).  This aspect of the ISTE Standards for Educators relates to self-efficacy (Bandura, 1994) as well as the concept of “grit,” involving perseverance and passion (Duckworth, Peterson, Matthews, & Kelly, 2007).

In a large study of students in Grades 4-8, Rojas, Reser, Usher, & Toland (2012) found that girls had statistically higher grit scores than boys. Bottomley (2015) investigated first-year university engineering students regarding grit, and found higher scores among women than men and some differences in ethnic groups.  Simpson and Maltese (2017) studied failures of STEM professionals, and found that their participants viewed persistence to be a singularly important trait and viewed their current success as STEM professionals to be rooted in part in previous experiences with failure.

In addition to perseverance, self-confidence can be important to learning STEM subjects. For example, an analysis of achievement of students participating in the 2007 Trends in International Mathematics and Science Study found that self-confidence in learning science strongly predicted achievement in science (Mohammadpour, Shekarchizadeh, & Kalantarrashidi, 2015).  In connection with these considerations, teachers in the TEST program were engaged in facilitating activities for youth that had the potential to build confidence by encouraging youth to persevere through challenges. The study assessed youth attitudes toward STEM subjects and their perceived difficulty.  The study also assessed grit as a descriptive measure, but probing for changes in grit was beyond its scope.

Support an Understanding of the Work of Engineers

One goal of the TEST program is supporting teachers in teaching about the work of engineers. NGSS (2013) standards conveyed expectations for teaching about science and engineering practices. In particular, they stated, “Engaging in the practices of engineering likewise helps students understand the work of engineers, as well as the links between engineering and science.” (Appendix F UNLESS THIS IS A SINGLE-PAGE APPENDIX, PROVIDE LOCATION FOR THIS QUOTATION).

Also, ISTE’s (2017b) Standards for Educators included an item for “authentic learning,” which was defined to include “career-/workforce-related projects” and “design projects and processes.” [PROVIDE LOCATIONS FOR THESE QUOTATIONS] In addition, a major curricular movement in California, Linked Learning, advocated helping students understand applications of academic learning to the workplace (Warner et al., 2016).  In alignment with considerations like these, the TEST program engaged teachers in teaching about the work of engineers.  The study included some assessments of youth perceptions of what engineers (as well as scientists) do for their jobs.

Support Learning for All Students

A further consideration of the TEST program was to support learning for all students. The program was designed to use cooperative learning and other strategies that have been recommended as beneficial for a broad range of learners (Saravia-Shore & Garcia, 2008).  The program gave teachers direct experience working with diverse learners at the Boys & Girls Club. In concert, the study investigated outcomes with these students. To monitor for any possible gender differences, the study compared responses of boys and girls.

For further information, a website about the project is available at  As discussed in the next section, the program used curriculum kits designed to serve all learners from the Engineering Is Elementary (EiE) project.

Curricula of the Engineering Is Elementary Project

Curricula from the EiE project were selected for the TEST program, as they aligned with the overall goals just described. Curricula of the EiE project have a four-part structure that includes (a) a story, set in a culture around the world, to provide the context for an engineering problem; (b) information about a particular engineering field relevant to the problem, to help provide a better understanding of what engineers do; (c) activities in which participants collect scientific data to inform engineering designs; and (d) engineering design challenges for participants to find a solution (Cunningham, 2009).

Units of the EiE program were specifically designed to support learning for all students, including boys and girls, students from traditionally underrepresented populations, and students from low socioeconomic status backgrounds. They provide culturally relevant issues and problems, as well as a culturally responsive pedagogy (Ladson-Billings, 1994) that engages students in connecting the content to real-world situations.

Cunningham and Lachapelle (2016) articulated principles that guided the design of the units, in four overall categories: (a) “set learning in a real-word context,” (b) “present design challenges that are authentic to engineering practice,” (c) “scaffold student work,” and (d) “demonstrate that ‘everyone can engineer.’” [PROVIDE LOCATION FOR THESE QUOTATIONS] Evaluation studies of EiE curricula (e.g., Lachapelle et al., 2011) found that the materials are in fact successful with diverse learners. (For an executive summary of EiE research, including the effectiveness of the materials with diverse learners, see Engineering Is Elementary, 2016)

A core element of EiE units is the project’s formulation of an Engineering Design Process with five elements: Ask, Imagine, Plan, Create and Improve (Cunningham, 2009). The process is cyclical, as shown by Figure 1, and is intended to teach children to learn to improve engineering designs. The failure of a design provides students formative feedback on that design, not summative feedback about students’ capabilities. The NGSS (2013) included elements that align with the EiE Engineering Design Process, including asking questions and designing solutions.

Figure 1. The EiE Engineering Design Process. Source:


Teachers are expected to know, understand, and be able to implement scientific and engineering processes in their classrooms (NGSS, 2013).  Curricula of the EiE project were relevant to this goal and also aligned with the goal of helping youth better understand the work of engineers. A study of EiE curricula found that they helped students gain a better understanding of what engineers did for their jobs (Lachapelle, Phadnis, Jocz, & Cunningham, 2012).

In addition to helping students understand linkages between science and engineering, units of the EiE project were also designed to help students better understand technology. Children may tend to think of technology as necessarily involving electrical components. EiE units have material that introduces them to a broader definition of technology, involving objects and processes to help people solve problems (Jocz & Lachapelle, 2012). In our course, teachers planned lessons using curriculum kits from the EiE project.

Overall Design of Course for Teachers

Elementary teachers may have limited confidence teaching STEM subjects (Appleton, 1995; Baniflower et al., 2013). The course for teachers gave them an opportunity to teach lessons related to engineering, a subject that most elementary teachers are not confident in teaching. A US survey of K-6 teachers found that 73% felt they were not adequately prepared to teach engineering and only 4% felt very well prepared to teach engineering (Baniflower et al., 2013).

In the current program, as part of a three-unit graduate level university course, 22 teachers working in teams developed STEM workshops for youth. Teachers were provided curriculum kits from the EiE ( ) project (Cunningham, 2009) to use as the basis of their workshops. These kits use PBL. In their original form, the kits used low-cost materials and tended to be low-tech. However, in the project described herein, the kits were used as a starting point for teachers to integrate educational technologies. In this way, teachers used a range of technologies together.

As they prepared to teach their workshops, teachers were provided iPad minis and encouraged to find ways to support the curriculum using them, including using apps for notetaking and measurement, Internet resources, videos, and video production software. In short, teachers were tasked with developing methods for integrating digital technologies into engineering instruction, while being scaffolded with a well-tested and well-regarded engineering curriculum.

Five groups were formed of three to six teachers, and each group selected an EiE curriculum kit to use. (A sixth group, which piloted a curriculum based on robotics that was not part of the EiE curriculum, was excluded from analysis.) The participating teachers had different backgrounds. Teams of teachers were formed to have teachers with differing strengths on each team. Each team included at least one member with five or more years of teaching experience, one member with STEM content experience (such as a B.A. in a STEM field), and one member with educational technology experience (e.g., current or past participation in a master’s program in educational technology).

The teams had not previously worked with one another as a group. The average number of years of teaching experience was 6.5 years. Of this group, 26% had 0 to 2 years of prior teaching experience, 35% had 3 to 8 years of prior teaching experience, and 39% had 9 to 15 years of teaching experience. Approximately 25% of the group were in an M.A. program in educational technology, 25% were in an M.A. program in educational administration, 10% were in an M.S. in science education, 10% were in an M.A. in curriculum and instruction, and 5% were in an M.A. in mathematics education. The remaining 25% were not currently in a graduate program, but had completed a master’s degree, either in educational technology, educational administration, or science education.

The course was offered for 12 weeks in summer 2015 via California State University, Long Beach. It included five 3-hour face-to-face meetings (15 hours total); 20 hours of online activities, including five scheduled 2-hour sessions and 10 further hours of online activities at times arranged by participants; and five 2-hour sessions at a site of the Boys & Girls Clubs of Long Beach, including an orientation session and four workshop sessions (10 hours total).

Teachers were given an overview of the EiE project and encouraged to review the set of kits available from the project. The second week of class, teams met face-to-face to decide on a kit to order based on the interests of group members. Costs of the kits, which were in the $300-$500 range, were covered with grant support.

The EiE project offered two collections of kits especially designed for use in out-of-school-time settings: the Engineering Everywhere series, designed for Grades 6-8, and the Engineering Adventures series, designed for Grades 3-5. The teachers selected the following kits from these series:

Each kit was used by one group, except for the first kit, Go Green: Engineering Recycled Racers, which was used by two groups.  The selected kits spanned a range of engineering topics. The EiE project units typically contain eight modules of 45 minutes to an hour each.  Altogether, teams of teachers adapted these to create 8-hour workshops that involved four sessions of 2 hours each. As there were five teams of teachers, this arrangement translated to five 8-hour workshops.

In addition to covering the costs of the curriculum materials, grant support funded both the costs of instruction for the teachers. The STEM workshops were offered at no cost to the participating youth.  In addition to using EiE units, teachers were asked to integrate technology into their workshops; they were also asked to make explicit connections between the content of their workshops and the work of professional engineers.

Integration of Educational Technology

One goal of the project was to support teachers in integrating educational technology in engineering instruction. In this way, the project aimed to support expectations articulated by ISTE Standards for Educators, which stated that teachers should use technology to create, adapt and personalize learning experiences that foster independent learning and accommodate learner differences and needs” (ISTE, 2017a, italics added [ADD LOCATION OF THE QUOTATION]).

Teacher teams were provided a set of 10 iPad minis to use in their workshops.  They were encouraged to integrate them into the EiE-based units in ways that supported their overall instructional goals. iPads were chosen for the project for several reasons. First, they were relatively portable, so teachers could experiment with them in a class setting and then use them in the Boys & Girls Club. Second, school districts in the geographic area of the project have established iPad initiatives, creating a demand for teachers to learn more about them. Also, staff at the Boys & Girls Club site felt that youth would be interested in the iPads.

Teachers were introduced to concepts about integrating technologies into teaching, include the SAMR framework (Puentadura, 2013) and material regarding using technology to help students integrate new and existing knowledge (Linn & Eylon, 2011).  Each group of teachers was tasked with formulating uses of digital technologies to support their units. Each teacher group was free to come up with its own methods; groups were also encouraged to share ideas.

Horton (2012) delineated three major categories for using technologies in learning: “Absorb,” “Do,” and “Connect.”  Although teachers were not explicitly introduced to this particular framework, their use of technology aligned with these categories.  In Horton’s “Absorb” activities, participants obtain basic information that can, in turn, support active learning. For example, teachers identified videos on the internet with foundational information related to the units and included them in activities. In Horton’s “Do” activities, participants apply their knowledge, such as hands-on activities with technology. For example, teachers added material in which students used iPads as tools to assist with measurement, to record data, or to record notes about their designs. Last, in Horton’s “Connect” activities, learners make connections to real-life situations.  In this category, teachers added activities in which students created a retrospective presentation connecting their engineering work to a real-world situation.

Teachers’ use of digital technology in the EiE units in the study spanned these three kinds of activities articulated by Horton (2012) and emphasized the use of iPad apps. Table 1 summarizes functions of apps used in the various EiE project units with reference to Horton’s three types of activities.

Table 1
Functions of Apps Used in Engineering is Elementary Project Units

Function App Author/Publisher
Apps with General Applications to Multiple Units
Introduce engineering challenge Engineering Adventures – Messages from the Duo Museum of Science (2016)
Learn about the definition of technology (“Absorb” activities) Technology Flashcards Museum of Science (2015)
Document engineering designs (“Do” activities);
Create digital content explaining engineering designs and connecting them to real-world situations (“Connect” activities)
Adobe Voice
Adobe (2014)
Luminant (2017)
Movenote (2015)
Apple (2017)
VoiceThread (2017)
Apps Specific to Individual Units
Research brain before designing helmets (“Absorb” activity for “safety helmet” unit) 3DBrain Cold Spring Harbor Laboratory (2016)
Research cane toads prior to designing traps (“Absorb” activity for “invasive species” unit) Cane Toad University of West Australia (2016)
Measure speed when testing race car designs (“Do” activity for “race car” unit) Speedclock Kaiser (2017)


We now turn to describing specific uses of digital technology, organized by the various units from the EiE project that were used. In the unit to create race cars out of recycled materials, teachers used the AudioNote app (Luminant, 2017), which allows photos and both written and audio notes, to record observations about their work. They incorporated an app, Speedclock (Kaiser, 2017), to measure the speed of their recycled racers. They also incorporated a Google-based spreadsheet to record distance and time traveled by the recycled racers.  In the unit to design safety helmets, teachers incorporated an app for notetaking, Movenote (which is no longer available). They also used an app about the brain, 3D Brain (Cold Spring Harbor Laboratory, 2016), and resources on the Internet for students to learn more about the brain and concussions.

In the unit on engineering ice cream, teachers incorporated VoiceThread (2017) as a tool for documenting their designs and had students make multimedia presentations about their designs using iMovie (Apple, 2017). In the unit to engineer a trap for invasive cane toads, teachers incorporated an app created by EiE, “Engineering Adventures – Messages from the Duo,” that has narrated messages from a fictional brother and sister, India and Jacob, regarding the engineering problem to be solved (Museum of Science, 2016). They also incorporated an app, “Technology Flashcards,” created by EiE for use in a preparatory activity concerning the nature of technology (Museum of Science, 2015).

In addition, they showed videos from the Internet pertaining to cane toads and had students learn more about cane toads using an app about them (University of West Australia, 2016). Over the course of the workshops, this group had students use the camera app of the iPads to document their designs and used the Adobe Voice app (Adobe, 2014) for a culminating activity to create public service announcements about cane toads.

Connecting Activities to Engineering and Science Careers

EiE project materials include explicit linkages to NGSS (2013) standards, including the science and engineering practices. Teachers were asked to be sure to include material in their workshops aimed to help students make connections between activities in the workshops and engineering and scientific professions. Teachers were asked to ensure they were explicit about such connections, as participating youth may not necessarily make the connections themselves.

Research Questions for Study of Youth Outcomes

The study was set in the context of a university graduate course designed to increase teachers’ abilities to implement educational technology in science and engineering instruction. The role of technology was to support overall curricular goals related to STEM, particularly engineering education. The study focused on the experiences of youth participating in the workshops and explored overall impacts, focusing on the youth participants’ perceptions of STEM subjects and STEM careers before and after the workshops. In particular, it examined the following questions:

  1. Prior to the workshops, how do youth respond to questions about “grit”? Are there differences by gender?
  2. Do youth attitudes about STEM change after participation in the workshops? Are there differences by gender?
  3. Do the perceptions of youth regarding the jobs of engineers and scientists change after participation in the workshops?  Are there differences by gender?
  4. What are the overall perceptions of youth about the workshops, including processes of engineering?

In the TEST project, children were immersed in engineering presented at their level as they participated in the EiE curricula.  We hypothesized that as youth were actively engaged in activities that parallel those in engineering, such as working collaboratively and designing solutions to engineering challenges, it could impact their attitudes toward STEM (and engineering in particular) as they began to see themselves in roles such as that of an engineer or scientist.  Also, as the EiE curriculum gives information about the roles of engineers (and to an extent, scientists), we anticipated the potential for students to report a better understanding of these roles after engaging in the workshops.

The instructional model provided a set of experiences for youth to participate in engineering activities that encouraged them to creatively solve engineering problems. We hypothesized that the experience would positively affect youths’ attitudes toward STEM subjects and STEM careers. The instructional activities for youth were relatively brief, taking place over four 2-hour sessions.  Promoting or measuring long-term change in youths’ attitudes was beyond the scope of the project.  Our approach involved comparing attitudes before and after these activities.

Identifying positive impacts on youth attitudes after the workshops hardly seemed assured based on the conditions of the study. The workshops were conducted at a Boys & Girls Club site under realistic, but challenging circumstances, with the potential to create distractions. The rooms in which the workshops were held included intercom speakers giving announcements of activities going on at the site.


We studied youth participating in the TEST program using a mixed-methods approach that combined questionnaires and focus groups. Institutional Review Board certification was obtained.


Youth participants came from a site of the Boys & Girls Clubs of Long Beach. Fifty-nine youth, aged 8-13, participated regularly in the STEM workshops.  Of these, 36 students, or 61%, provided informed consent forms to participate in the study. Of the participants, 20 identified as girls (56%), 15 identified as boys (42%), and one declined to state a sex (3%). (The total percentages exceed 100% due to rounding.) Also, 17 participants self-identified as African-American (47%), 12 self-identified as Hispanic/Latino (33%), one identified as Asian (3%), and six as other/Caucasian/mixed ethnicity (17%). Of the participants, 69% reported they were offered the opportunity to do science projects or experiments in school only once a month or less.

A summary of data sources and analyses for each question is shown in Table 2.

Table 2
Research Questions and Analyses

Construct Research Question Data Analysis
Grit Q1. Prior to the workshops, how do youth respond to questions about “grit”?  Are there differences by gender? Preworkshop survey items Mean responses to survey items. Comparison of mean    responses of boys and girls.
Attitudes regarding STEM RQ2. Do youth attitudes about STEM change after    participation in the workshops? Are there differences by gender? Pre- and postworkshop survey items Comparison of mean responses on pre- and postsurvey questions. Comparison of responses of boys and girls.
Perceptions of the work of engineers and scientists RQ3. Do the perceptions of youth regarding the jobs of    engineers and scientists change after participation in the workshops? Are    there differences by gender? Pre- and postworkshop survey items Comparison of mean responses on pre- and postsurvey questions. Comparison of responses of boys and girls.
Perceptions of Workshops and Content RQ4. What are the overall perceptions of youth about the    workshops, including processes of engineering? Postworkshop survey items. Focus groups. Mean responses to survey items. Qualitative analysis of focus groups.



To assess youth attitudes toward engineering and STEM, two surveys were created. The surveys were administered to youth at two points in time, immediately before and immediately after the workshops. In the interest of brevity, they were limited to 10 items each. The items used a 5-point Likert scale. Eight items were common to both surveys, and two items were unique to each. A goal in the design of the surveys was to keep them short, while addressing the research questions. Most survey items were selected from previously published surveys. Appendix A provides a listing, with questions in four overall categories corresponding to the four research questions:

  1. grit (two items),
  2. attitudes toward STEM (six items including two items related to self-efficacy),
  3. knowledge of the jobs of engineers and scientists (two items), and
  4. attitudes toward the STEM workshops (two items).

The items from the first category, regarding grit, came from a survey created by Duckworth et al. (2007). These items were used as a kind of descriptive information about the sample and facilitated comparisons with other studies that have used these items. The items were not included on the postassessment, as they were not designed for use in testing before and after an intervention.

Items from the next two categories—attitudes toward STEM and STEM careers—were on both the pre-assessment and the postassessment and formed the bulk of the survey. Three of these items came from a prior youth engineering and science attitudes assessment (EiE, 2010), with questions that were derived from a study of engineering attitudes and knowledge (Gibbons, Hirsch, Kimmel, Rockland, & Bloom, 2004). Using these questions facilitated comparisons with previous research of the EiE project.

Four questions of the survey came from the Trends in International Mathematics and Science Study Student 4th Grade Questionnaire (National Center for Education Statistics [NCES], 2011), facilitating comparisons with prior research using these items. An eighth question, concerning technology, was new. Last, question items related to the fourth category concerned youths’ attitudes toward the STEM workshops. Two items were created in this category and were included only on the post-assessment.

Focus Groups

Focus groups provided a source of data for the fourth research question, “What are the overall perceptions of youth about the workshops, including processes of engineering?”After the workshops, six focus groups were conducted of participating youth using semistructured prepared questions, with follow-up questions asked as needed for clarity.  These sessions were audio recorded, had 5-10 students each, and lasted 25 to 45 minutes.  A protocol for the focus groups was adapted from one used in a prior study of an informal education program (Taylor-Powell & Calvert, 2006). It included questions about youths’ reactions to the workshops, such as what they learned, liked, or found challenging. The modifications included a question about the EiE Engineering Design Process that was featured in the units. See Appendix B for a complete listing.


The responses to the Likert survey questions were coded on a scale of 1-5, with a higher number indicating stronger agreement. For example, strongly disagree was coded as a 1, not sure was coded as a 3, and strongly agree was coded as a 5. Descriptive statistics were computed of the demographic data and of items that appeared on the preworkshop survey only and the postworkshop survey only. To test for significant changes on questions that appeared on both the pre-and postsurveys, paired t tests were completed with an alpha of .05.

Because these tests indicate the presence of any significant changes but not their magnitude, effect sizes were computed using Cohen’s d and reported with the convention small ≥ 0.20, medium ≥ 0.50, large ≥ 0.80 (Cohen, 1992). For both Cohen’s d, and t tests, we report absolute values.

Two researchers coded all focus group transcripts using a constant-comparative data analysis with emergent codes (not a priori), placing each unit of data (a phrase, sentence, or paragraph that stood on its own) into categories. The NVivo qualitative analysis software was used for this purpose. Researchers observed that elements of the EiE Engineering Design Process were prominent in responses to focus group questions even in sections of interviews not specifically concerning this topic. Key emerging codes corresponded to each of the five elements of the EiE Engineering Design Process. Coders resolved any differences by consensus. The focus group transcripts were also reviewed for information pertinent to each survey question.


The discussion of the results is organized by the research questions. Results from focus groups are discussed as applicable as they pertain to the research questions.


The preworkshop survey included two items related to grit as a kind of descriptive data. Girls more strongly agreed with questions on the pre-assessment related to grit than did boys. On the item, “I am a hard worker,” the mean response of boys was 3.3.  On the other hand, the mean response of girls was 4.2, or slightly over agree.  A two-sample t-test (that did not assume equal variances) found this difference was significant [t (24) = 2.09, p < .05]. Similarly, on the item, “I am hard working and careful,” the mean response was 3.3 for boys, but higher for girls, whose mean response was 4.4. A two-sample t-test (that did not assume equal variances) was likewise significant for this item [t (22) = 2.87, p < .001].


Table 3 presents a summary of data regarding participants’ attitudes toward STEM.  First, we discuss the items that had significant changes.

Table 3
Youth Responses to Items Regarding Attitudes Toward STEM Before and After Workshops

Question Group Mean
N Paired t-Test
Paired t-Test
Cohen’s d Cohen’s d size
l like thinking of new and better ways of doing things.[b] All 3.94 4.46 35 2.40 .02* 0.52 medium
Boys 3.73 4.40 15 1.58 .14 0.53 medium
Girls 4.21 4.47 19 1.56 .14 0.40 small
I enjoy learning science.[c] All 4.11 3.75 36 1.62 .11 0.31 small
Boys 4.00 4.13 15 0.62 .55 0.11
Girls 4.15 3.45 20 2.01 .06 0.60 medium
I enjoy learning math.[c] All 4.25 4.11 36 0.62 .54 0.12
Boys 4.47 3.93 15 1.95 .07 0.40 small
Girls 4.05 4.20 20 0.45 .66 0.15
I like using technology. All 4.46 4.71 35 1.36 .18 0.27 small
Boys 4.14 4.50 14 0.86 .40 0.27 small
Girls 4.75 4.85 20 0.70 .49 0.19
Science is harder for me than for many of my classmates.[c] All 2.92 2.72 36 0.89 .38 0.13
Boys 3.53 2.87 15 2.87 .01* 0.44 small
Girls 2.55 2.65 20 0.30 .77 0.07
Math is harder for me than for many of my classmates.[c] All 2.14 2.75 36 2.51 .02* 0.40 small
Boys 2.27 2.93 15 1.63 .13 0.39 small
Girls 2.10 2.70 20 1.88 .08 0.42 small
*p < .05
[a]Absolute value  reported with degrees of freedom = n-1
[b]Source of item  (Engineering is Elementary, 2010)
[c]Source of item  (NCES, 2011)

“I like thinking of new and better ways of doing things.” Mean responses to this item were 3.94 before the workshops, rising to 4.46 after the workshops, a significant increase [t (35) = -2.40, p = .02] and a medium effect size (d = 0.52). In focus groups, youth made statements that aligned with the survey question. For example, youth comments included, “We learned how to improve things, well like, if you get something wrong you can improve it, and work it out better” and “Whenever we make something, each time we come back, we always make it better.”

“Science is harder for me than for many of my classmates.” On this item the level of agreement of boys and girls combined was similar before and after the workshops: the mean was 2.92 before the workshops, and 2.72 after the workshops, a difference that was not significant [t(36) = 0.89, p = .38] and a negligible effect size (d = 0.13). Also, the mean responses of girls were 2.55 before the workshops and 2.65 after the workshops, a change that was not significant [t(20) = 0.30, p = .77].  On the other hand, there was a change for boys.  The mean response for boys was 3.53 before the workshops, but 2.87 after the workshops, a significant decrease [t(15) = 2.87, p < .01] and a small effect size (d = 0.44). In other words, after the workshops, boys agreed less strongly with the statement.

“Math is harder for me than for many of my classmates.” Overall, youth participants disagreed with this statement from the Trends in International Mathematics and Science Study (TIMSS; NCES, 2011). Mean responses for both boys and girls were under 3 before and after the workshops. An unanticipated outcome was that youth more strongly agreed with the negative statement after the workshops. The mean response was 2.14 before the workshops, but 2.75 after the workshops, a significant decrease [t(36) = 2.51, p = .02] and a small effect size (d = 0.40). These changes were unexpected given that math was not emphasized in the workshops. We are not certain of an explanation, as the focus group data does not support or illustrate a new dislike for mathematics, and mathematics was not a large focus of youths’ work in the EiE investigations. We also learned that math tutoring was simultaneously occurring at the youth club site. It is possible that this or other outside factors were at play.

We did not find evidence in the focus groups to corroborate that students viewed math less favorably because of the workshops. Rather, when youth participants commented on math, they indicated that they viewed it more favorably in the workshops, as seen in the following examples:

At first, I thought that math was really boring. When I did it, I’d get stuck and it just irritated me a lot, but when I started doing this, I actually noticed that math is actually well, fun.

I really hated math. I always struggled doing math. You just have to make it fun. And then when I did engineering, which was very easy, I started to like math.

On three further questions, comparing responses before and after the workshops, there were no statistically significant changes (see Table 3).  Two of these items came from the TIMSS survey (NCES, 2011): “I enjoy learning science and “I enjoy learning math.” A third item that did not change significantly was, “I like using technology.”  In other words, the question items that did not change significantly involved either enjoying learning math or science or liking using technology. These items that did not have significant changes complement the items that did have significant changes. In sum, there were no changes on simply liking or enjoying STEM subjects, but there were (a) pronounced changes on an item related to engineering and (b) changes related to the perceived difficulty of other STEM areas.


Table 4 presents findings regarding youth perceptions of the jobs of engineers and scientists. There were statistically significant changes on one item (for girls) and another item (for boys and girls combined).

Table 4
Youth Responses to Items Regarding the Jobs of Engineers and Scientists Before and After Workshops

Question Group Mean
N Paired
Cohen’s d Cohen’s size
I think I know what engineers do for their jobs.[b] All 3.83 4.11 36 1.62 .12 0.27 small
Boys 4.27 4.13 15 0.56 .58 0.14
Girls 3.60 4.15 20 2.34 .03* 0.53 medium
I think I know what scientists do for their jobs.[c] All 3.75 4.17 36 2.16 .04* 0.38 small
Boys 4.00 4.40 15 1.57 .14 0.40 small
Girls 3.50 3.95 20 1.53 .14 0.39 small
[a]Absolute value reported with  degrees of freedom = n-1
[b]Source of item  (Engineering is Elementary, 2010)
[c] ??

“I think I know what engineers do for their jobs.” When the responses of boys and girls were combined, there were no statistically significant changes overall to this item. The overall mean response was 3.83 before the workshops and 4.12 after the workshops, which was not a significant increase [t(36) = 1.62, p = .12], but yielded a small effect size (d = 0.27). However, the responses for this item differed when data for boys and girls were disaggregated. For boys, the mean response was 4.13 before the workshops and 4.06 after the workshops, a difference that was not significant [t(15) = .56, p = .79] and a negligible effect size (d =  06). However, for girls, the mean response was 3.60 before the workshops and 4.15 after the workshops, a significant increase [t(20) = 2.34, p = .03] and a medium effect size (d = 0.53).

Figure 2 shows the change in response for each participant, computed by subtracting pre-assessment responses from postassessment responses. For example, a student whose response was 3 on the pre-assessment and 4 on the postassessment would be coded as a 1 in the graph. Ten girls more strongly agreed with the statement after the STEM workshops, while two less strongly agreed.  On the other hand, four boys less strongly agreed while two more strongly disagreed.  It is unclear why the two boys more strongly disagreed; that information did not emerge from the interview data.

Figure 2
Figure 2. Difference between pre- and postworkshop agreement ratings for each participant to the statement, “I think I know what engineers do for their jobs.”


“I think I know what scientists do for their jobs.” On this question, the mean response was 3.75 before the workshops, which rose to 4.17 after the workshops, a significant increase [t(36) = 2.16, (p = .04] and a small effect size (d = 0.38). Student comments in focus groups suggested a shift in the way students perceived science.  Rather than seeing science as a kind of classroom activity (“Science, I thought it was just like you put like a liquid in another liquid and have something happen”) they perceived connections to engineering (“I saw that science was also important because engineers need it a lot”). In addition, some student comments reflected students seeing themselves as scientists:

Most of all I liked the experience because it’s like, well when I go home sometimes I pretend like I’m a scientist and learning about cane toads and my mom is like, did you have a good day with the cane toads and stuff? And I’m like, “yeah.” So, it’s a really good experience.


The postassessment included a survey item that asked students to indicate their level of agreement with the statement, “I would be interested to take another workshop like the one I just took but on a different topic.” The percentage of students agreeing with this statement was 50%, while 39% were neutral and 11% disagreed. The postassessment also included an item that simply asked students to indicate their agreement with the statement, “I learned about engineering in the workshop.”  On this item, 92% of the students agreed, 3% were neutral, and 6% disagreed. (The sum of the percentages exceeds 100% due to rounding).  The next section discusses students’ responses from the focus groups, which give a further source of information regarding their experiences in the STEM workshops.  These results highlight aspects relating to RQ2 and RQ3 regarding students’ knowledge of about engineering – as well as understanding the engineering design process.

The EiE Engineering Design Process was central to all STEM workshops, as it is in all EiE curricula. In the focus group interviews, sources of data about the Engineering Design Process came from two sources.  One source was a direct question about it. Students were shown a diagram of the engineering design process, like the one shown in Figure 1 from the EiE project, but with the words removed. They were asked if they remembered the diagram and could explain it and its steps.  Researchers did not name specific steps of the Engineering Design Process. In focus group discussions, students indicated they recognized the diagram, as shown in the example below:

Interviewer:     Can you explain this diagram?

Student:          It’s the engineering design process.

Youth also recalled steps of the process, as indicated in the example below:

Interviewer     So, do you remember this diagram?

Student 1:       Yes.

Student 2:       It was Imagine.

Student 3:       Oh. Imagine, Ask, Create.

Student 4:       Plan…Build.

Student 2:       It starts with an I. Engine…?

Student 3:       Improve! Improve!

While these responses indicated a recognition of the Engineering Design Process, we found that references to the Engineering Design Process appeared in students’ responses to other questions. That is, in answering other questions, students made comments that aligned with steps of the Engineering Design Process. For example, the following quote was coded as a reference to the Improve step, and occurred in a question about which week of the workshop the students liked best:

The week that I liked the best was 3, and 4, because, when there was a week one, then I got to improve, we got to improve on the car, car that we made, and check the miles and how far it goes. And how fast it goes.

Altogether, in sections of the interview that were not specifically about the Engineering Design Process, coders identified 156 instances in which youth made comments corresponding to a step of the Engineering Design Process. Each of the five steps were represented, but with differing frequencies. The most common steps were Create and Improve, which were each referred to 59 times. In other words, the Create and Improve steps were discussed most, collectively accounting for 118 instances, or 76% (of 156) of the references to the Engineering Design Process.  The Plan step was referred to 20 times, the Imagine step was referred to 12 times, and the Ask step was referred to six times. Possibly, the Create and Improve steps were referred to more than other steps as an indicator of the extent to which these steps consumed students’ time and attention.

We also analyzed the number of occasions in which participants referred to these steps concurrently – in the same sentence or general statement during the focus group. The steps most commonly mentioned concurrently were Improving and Creating (15 instances), followed by Improving and Planning (10 instances), Creating and Planning (5 instances), and Asking and Imagining (five instances).  Steps less frequently mentioned concurrently were Planning and Asking (four instances), Planning and Imagining (four instances), Creating and Asking (one instance) and Asking and Improving (one instance).

Other observations from the focus groups include the following: Youth reported pride in working individually without help; for example, “I think the best week was the one where we got to make our cars and test them, because my car, the one that I had, I had no help with at all.”  At times, youth participants indicated that it was difficult to integrate ideas from other youth: “Teamwork is the hardest because, like, we wouldn’t cooperate and we wouldn’t sort of listen and we wanted to do other stuff.”

Discussion and Conclusion

The program describe here for supporting teachers to use digital technology in engineering instruction provided a backdrop for the study of youth in this project. Teachers of differing backgrounds who had not worked together before collaborated in teams to plan and conduct STEM workshops.  Using EiE project materials as a starting point, teachers developed ways to use educational technologies in their workshops. Youth participating in the TEST program were predominantly from underserved populations. Of our youth participants, 69% indicated they had opportunities at school for doing science projects once per month or less, and 80% identified as either African American or Hispanic/Latino.  With these conditions, the study of youth shows evidence of beneficial outcomes, which is also an indicator of teachers’ success in conducting the workshops. A larger sample would have afforded more statistical power, but even so, several of the survey questions given to youth saw statistically significant changes. Survey results showed some signs of increased self-efficacy on two items related to this construct, in that after the STEM workshops:

  • Boys and girls more strongly agreed that they like thinking of new and better ways of doing things.
  • Boys (compared to girls) less strongly agreed that science is “harder for me than for many of my classmates.”

Supporting or measuring long-term changes in youth grit and self-efficacy was beyond the scope of this project. The study findings are encouraging, though, considering the instructional activities for youth were created by teachers who were just learning about these educational approaches as part of a university course.

The survey item with the strongest effect size for boys and girls combined was “I like thinking of new and better ways of doing things.” This result is an indicator of a more positive attitude toward engineering after the workshops. By way of comparison, a study of EiE curricula in classroom settings did not find a change on this item, either for all students as a group or for different demographic groups (Lachapelle et al. 2012).  Possibly, students in our sample were more influenced by the EiE activities in this regard.

In our study, there were no significant changes on items related to student attitudes toward STEM disciplines other than engineering; that is, “I enjoy learning math,” “I enjoy learning science,” or “I like using technology” (see Table 3). We observed survey results could reflect ceiling effects on these items and that the instructional intervention emphasized engineering and was relatively brief.

Nonetheless, the workshop activities appeared to heighten awareness of activities in STEM-related careers, as seen in other summer STEM programs (Duran, Hoft, Lawson, Medjahed, & Orady, 2014). Survey questions showed evidence that after the workshops youth reported increased familiarity with the jobs of engineers and scientists:

  • Boys and girls more strongly agreed that they knew what scientists did for their jobs
  • Girls more strongly agreed they knew what engineers do for their jobs.

These findings differed somewhat from a previous study of EiE curricula, which found that both boys and girls reported greater knowledge of the work of engineers, but not the work of scientists (Lachapelle et al., 2012). On the item, “I think I know what engineers do for their jobs,” only girls in our study showed an increase. Boys in our study tended to agree with the question on the pre-assessment; hence, there was little room for them to increase. Our results differed from this prior EiE study on the item, “I think I know what scientists do for their jobs.”

Unlike this previous study, our study found that youth reported a greater understanding of what scientists do for their jobs. In our implementation, youth used technological tools for data collection and reporting purposes, and teachers were encouraged to explicitly discuss STEM careers. Further research that uses EiE project units as we did could investigate whether, after instruction, youth report a greater knowledge of what scientists do for their jobs, and if so, why.

In multiple instances, participation in the STEM workshops tended to equalize responses for boys and girls. After the workshops, girls more strongly agreed with the question about knowing what engineers do for their jobs than they had before the workshops. In the process, their responses became more like the responses of boys. Also after the workshops, boys less strongly agreed with the negative item, “Science is harder for me than for many of my classmates,” than they had before the workshops; in this way, their responses became more like those of girls.

These findings align with a previous evaluation of EiE curricula. Lachapelle, Jocz, and Phadnis (2011) found that, in a control group, girls’ interest in becoming an engineer was less strong than that of boys, but the interests of boys and girls equalized after participation in EiE curricula. Also in this previous study, in a control group, boys showed less interest than girls in being a scientist, but the interests of boys and girls tended to equalize after participating in EiE curricula.

While the results of this prior study were similar in these respects, the contexts differed. In the prior study, the EiE units were conducted in K-12 schools in Grades 3, 4, and 5 and cotaught by teachers who had participated in an EiE training program and by science specialists from the Science Museum of Minnesota. The present study contributes findings based on workshops conducted in an out-of-school-time setting, by teachers who were learning about using educational technology in engineering education via a university course.

In the questions about grit collected only on the pretest, we found that the mean scores of girls were higher than those of boys. This gender difference is similar to Bottomley’s (2015) finding in a study of first-year engineering students in which women had higher scores on grit items than did men. It was beyond the scope of the study to make any claims about whether the construct of grit per se is predictive of or influences students’ learning in engineering education. Further work could specifically explore connections between constructs related to grit and youth learning about engineering, including the Engineering Design Process that encourages refining designs and learning from failures. Related to this topic, Lottero-Perdue (2016) suggested that the process of responding to failures and revising engineering designs in EiE units could support the development of a growth mindset (Dweck, 2006).

Whereas the grit items related to being a “hard” worker, two question items we used from TIMMS related to math or science being perceived as “hard.” Although the TIMMS questions may well be related to grit, they are not the same construct, and we consider the TIMMS questions to be more closely related to self-efficacy.

We did not undertake a comparison of responses to grit items before and after the workshops, but did compare responses to questions related to self-efficacy before and after the workshops. Before the STEM workshops, boys in our study were more likely than girls to agree with a TIMMS item, “Science is harder for me than for many of my classmates.”  However, boys’ mean agreement with this statement decreased after the workshops, becoming more like that of girls. Although the study did not investigate whether any changes in attitudes were long lasting, perceived self-efficacy in science can be linked to achievement in science.

In a study focusing on fourth graders in Korea, lower agreement with this specific item was found to be linked to higher scores on science tests (House & Telese, 2017).  Although the survey results from the present study related to this item about youth’s perceived difficulty of science are limited, subsequent research could use interviews to explore whether youth perceive these kinds of engineering activities as influencing their views of science and their own self-efficacy.

Responses to a survey item from TIMMS about math, Math is harder for me than for many of my classmates were unexpected, in that students agreed with this statement more strongly at the end of the workshops. However, this result was not corroborated by focus group data. In focus groups, students made positive comments about both math and science. Mathematics had not been an emphasis of the workshops teachers created. Additionally, other impacts upon our mathematics measures could be due to a mathematics project taking place at the Boys & Girls Club site at the time. We observed (in field notes) that this other program, in which some students participated, included a didactic pedagogy with worksheets and math drills.  Possibly, this other program impacted our youth participants, confounding assessment of their attitudes toward mathematics.

In the focus groups, youth made comments reflecting experience with the EiE Engineering Design Process. Collectively, they referred to each step of the Engineering Design Process in sections of the interview not specifically about the Engineering Design Process. It does seem noteworthy that the Improve step was prominent in what youth volunteered to talk about, a possible indication of its salience.  Their interest in discussing matters related to the Improve step in focus groups is consistent with findings from the survey data about the question, “I like thinking of new and better ways of doing things.”  Youth more strongly agreed with this item that after the workshops, and it had the highest effect size for the group as a whole.

Limitations and Recommendations for Future Work

Although it was not practical in this initial study to include a control group, it could be done in future studies and would serve to better assess impacts of the workshops, themselves, on student attitudes. Also, although EiE materials were central to the workshops, teachers modified them by integrating educational technologies and by incorporating further activities to emphasize connections between workshop activities and STEM careers. Investigating possible impacts that may have resulted from these alterations is beyond the scope of the present study, but could be specifically investigated in future research.

Although formal training in the EiE unit was not included in the program, it could be included in future projects like this. Persons using EiE curricula may not necessarily have access to formal training in it; the conditions of the present study were one example of a realistic situation.

Our questionnaires relied on self-reports, which have limitations and are subject to reference bias (Duckworth & Yeager, 2015). Also at this initial stage, we had limited opportunities for data collection and wished to keep the surveys brief while exploring a range of topics. Further studies could include multiple question items on a narrower set of topics. Also, we would either drop our survey question on mathematics or include specific questions about youths’ views related to this question in our focus group interview.

The use of a short Likert survey enabled us to collect some data from our youth participants in a short amount of time, without impacting the core instructional purpose of the STEM workshops.  That said, Likert surveys have inherent limitations for evaluating student attitudes for younger students. In future research, systematic prompting in interviews or focus groups could help compensate for limitations in survey questions. It would be interesting to use such methods to probe for connections between specific workshop activities and attitudes toward STEM subjects and STEM careers.

EiE curricula were developed initially for use in elementary school settings and have been studied most extensively in a school context (Engineering is Elementary, 2016; Lottero-Perdue & Parry, 2016; Oh, Lachapelle, Shams, Hertel, & Cunningham, 2016).  The project subsequently branched out to develop kits for use in out-of-school-time settings via the Engineering Everywhere and Engineering Adventures series that were used in the current study.

This study contributes to the base of research in uses of these curricula in out-of-school settings (Higgins, Hertel, Lachapelle, & Cunningham, 2013; Higgins, Hertel, Shams, Lachapelle, & Cunningham, 2015; Higgins, San Antonio et al., 2015; Higgins, San Antonio et al., 2013).  Further work could study these units in out-of-school settings as well as their application to teacher preparation programs.

We have described one way to support teacher professional development using EiE curricula in informal settings, but other strategies are possible in this direction. For example, as part of another project of some of the authors (Kisiel, Martin-Hansen, and Adams, 2017), preservice teachers in a university teaching methods course used kits of the EiE project with youth at another Boys & Girls Club site. This approach did not aim to engage teachers in integrating technology into kits of the EiE project, as was done in the current program, but it did include a component in which teacher candidates taught, reflected on their experiences, and then retaught the material. The relative advantages of variations of these instructional strategies, tailored for different purposes, is a topic ripe for further investigation.


In the approach we report here, as part of a university course, teachers developed ways of using educational technologies with EiE materials and then conducted STEM workshops at a Boys & Girls Club.  The study provides evidence of benefits to youth from diverse backgrounds from participating in the workshops. This evidence, in turn, indicates that, although the participating teachers had limited experience with teaching this kind of project-based curriculum, they were nonetheless successful. Overall, this work contributes a promising model for teacher development to use educational technology in engineering education that uses project-based learning.

Author Note

Paul Burns is now at Intellectual Virtues Academy, Long Beach.

This project was supported by Fluor, Chevron, and by a grant awarded by the CSU Collaborative for the Advancement of Linked Learning with funding from the James Irvine Foundation. We would like to acknowledge assistance from the following persons for which we are also grateful. The efforts of Joan Bissell were integral to making the project possible. Christine Cunningham provided useful information about the Engineering is Elementary project and suggestions. Elaine Bernal provided assistance with the qualitative data analysis. We received helpful suggestions on previous drafts from James Kisiel, Lesley Farmer, Donna Ross, Daniel Soodjinda and anonymous reviewers.


Adams, S., & Bernal, E. (2016). Evaluating a program of teacher training in educational technology and STEM using two measures of TPACK. In G. Chamblee & L. Langub (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2016 (pp. 2783-2789). Chesapeake, VA: Association for the Advancement of Computing in Education.

Adams, S., Bernal, E., Cole-Jackson, M., & Martin-Hansen, L. (2015). Training teachers to use educational technologies in STEM using field experience at a community based organization. In D. Rutledge & D. Slykhuis (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2015 (pp. 3021-3025). Chesapeake, VA: Association for the Advancement of Computing in Education.

Adobe. (2014). Adobe Voice [software]. Available from:

Apple. (2017). iMovie [software]. Available from

Appleton, K. (1995). Student teachers’ confidence to teach science: Is more science knowledge necessary to improve self‐confidence? International Journal of Science Education, 17(3), 357-369.

Bagiati, A., Yoon, S. Y., Evangelou, D., & Ngambeki, I. (2010). Engineering curricula in early education: Describing the landscape of open resources. Early Childhood Research & Practice, 12(2).  PAGE NUMBERS?

Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning. Educational psychologist, 28(2), 117-148.

Bandura, A. (1994). Self-efficacy. In V. S. Ramachaudran (Ed.), Encyclopedia of human behavior (Vol. 4, pp. 71-81). New York, NY: Academic Press. (Reprinted in H. Friedman [Ed.], Encyclopedia of mental health. San Diego, CA: Academic Press, 1998).

Baniflower, E. R., Smith, P. S., Weiss, I. R., Malzahn, K. A., Campbell, K. M., & Weis, A. M. (2013). Report of the 2012 National Survey of Science and Mathematics Education. Chapel Hill, NC: Horizon Research, Inc.

Bell, P., Lewenstein, B., Shouse, A., & Feder, M.A. (Eds.). (2009). Learning science in informal environments: People, places and pursuits. Washington, DC: National Academies Press.

Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting the learning. Educational Psychologist26(3-4), 369-398.

Bottomley, L. (2015, June). Assessing GRIT of incoming engineering students. Paper presented at the conference of the American Society for Engineering Education, Seattle, WA.

Brossard, D., Lewenstein, B., & Bonney, R. (2005). Scientific knowledge and attitude change: The impact of a citizen science project. International Journal of Science, 27(9), 1099-1121.

Bybee, R. W., & Fuchs, B. (2006). Preparing the 21st century workforce: A new reform in science and technology education. Journal of Research in Science Teaching, 43(4), 349-352.

Capobianco, B. M., Diefes-Dux, H. A., Mena, I., & Weller, J. (2011). What is an Engineer? Implications of elementary school student conceptions for engineering education. Journal of Engineering Education, 100(2), 304-328.

Christensen, R., Knezek, G., & Tyler-Wood, T. (2014). Student perceptions of science, technology, engineering and mathematics (STEM) content and careers. Computers in Human Behavior, 34, 173-186.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155.

Cold Spring Harbor Laboratory (2016). 3DBrain [software].  Available from:

Cunningham, C. M. (2009). Engineering is elementary. The Bridge, 30(3), 11–17.

Cunningham, C. M., & Lachapelle, C. P. (2016). Designing engineering experiences to engage all students. Educational Designer, 3(9). PAGE NUMBERS?

Dewey, J., & Dewey, E. (1915). Schools of to-morrow. New York, NY: Dutton.

Duckworth, A.L., Peterson, C., Matthews, M.D., & Kelly, D.R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 9, 1087-1101.

Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237-251.

Duran, M., Höft, M., Lawson, D., Medjahed, B., & Orady, E. (2014). Urban high school students’ IT/STEM learning: Findings from a collaborative inquiry- and design-based afterschool program. Journal of Science Education and Technology, 23(1), 116-137.

Dweck, C. S. (2006). Mindset: The new psychology of success. ADD CITY, STATE: Random House Incorporated.

Engineering is elementary. (2010). Engineering and Science Attitudes Assessment. Retrieved from

Engineering is elementary. (2016). Research and evaluation results for the Engineering is Elementary Project, 2004-2016: An executive summary. Retrieved from

Gibbons, S. J., Hirsch, L. S., Kimmel, H., Rockland, R., & Bloom, J. (2004, October). Middle school students’ attitudes to and knowledge about engineering. In International Conference on Engineering Education, [PROCEEDINGS?? PLEASE FORMAT IN APA STYLE] Gainesville, FL.

Harris, J., Mishra, P., & Koehler, M. (2009). Teachers’ technological pedagogical content knowledge and learning activity types. Journal of Research on Technology in Education, 41(4), 393-416. doi:10.1080/15391523.2009.10782536

Hayden, K., Ouyang, Y., Scinski, L., Olszewski, B., & Bielefeldt, T. (2011). Increasing student interest and attitudes in STEM: Professional development and activities to engage and inspire learners. Contemporary Issues in Technology and Teacher Education11(1). Retrieved from

Higgins, M., Hertel, J., Lachapelle, C. P., & Cunningham, C. M. (2013). Engineering Adventures curriculum development grant. Boston, MA: Museum of Science.

Higgins, M., Hertel, J. D., Shams, M. Lachapelle, C. P., & Cunningham, C. M. (2015). NASA mission grant: Engineering Adventures unit development. Boston, MA: Museum of Science.

Higgins, M., San Antonio, C., DiIeso, M., Meyer, N., Shams, M., Tauer, T., … & Cunningham, C.M. (2015). Engineering Everywhere curriculum development: Final Report. Boston, MA: Museum of Science.

Higgins, M., San Antonio, C., Hertel, J., Cunningham, C.M., & Lachapelle, C.P. (2013). i2 and Engineering Everywhere curriculum development grant: Annual report. Boston, MA: Museum of Science.

Horton, W. (2012). E-learning by design (2nd ed.). San Francisco, CA: Pfeiffer.

House, J. D., & Telese, J. A. (2017). Confidence in science and achievement outcomes of fourth-grade students in Korea: Results from the TIMSS 2011 assessment. Education, 137(4), 389-392.

International Society for Technology in Education. (2008). Standards for teachers. Retrieved from

International Society for Technology in Education. (2017a). Standards for educators. Retrieved from

International Society for Technology in Education. (2017b). Standards for educators. [pop-up window linked to “authentic learning” of Indicator 5b.] Retrieved from

International Society for Technology in Education (2017c). Standards for educators. [pop-up window linked to “active, deep learning” of Indicator 5b.] Retrieved from

Jocz, J., & Lachapelle, C. (2012). The impact of Engineering is Elementary (EiE) on students’ conceptions of technology.  Retrieved from

Kaiser, S. (2017). Speedclock [software]. Available from

Kisiel, J., Martin-Hansen, L., & Adams, S. (2017). Developing effective and engaging practices in STEM Education: Supporting learning via university-community collaborations. Unpublished evaluation report completed for the W.M. Keck Foundation Southern California Program.

Lachapelle, C. P., Cunningham, C. M., Jocz, J., Kay, A. E., Phadnis, P., Wertheimer, J., & Arteaga, R. (2011). Engineering is Elementary: An evaluation of years 4 through 6 field testing. Boston, MA: Museum of Science.

Lachapelle, C. P., Jocz, J., & Phadnis, P. (2011). An evaluation of the implementation of Engineering is Elementary in fourteen Minneapolis public schools. Retrieved from

Lachapelle, C. P., Phadnis, P., Jocz, J., & Cunningham, C.M. (2012). The impact of engineering curriculum units on students’ interest in engineering and science. Paper presented at the National Association for Research in Science Teaching Annual International Conference, Indianapolis, IN.

Ladson-Billings, G. (1994). The dreamkeepers: Successful teachers of African American children. San Francisco, CA: Jossey-Bass.

Linn, M. C., & Eylon, B. S. (2011). Science learning and instruction: Taking advantage of technology to promote knowledge integration. CITY, STATE: Routledge

Lottero-Perdue, P. S. (2016, MONTH?). Fifth graders’ perceptions about failure and mindsets before and after learning to engineer. Paper presented at the National Association for Research in Science Teaching International Conference, Baltimore, MD.

Lottero-Perdue, P. S., & Parry, E. A. (2016, June). Elementary teachers’ reflections on design failures and use of fail words after teaching for two years. Paper presented at the American Society of Engineering Education Annual Conference & Exposition, New Orleans, LA.

Lou, S. J., Shih, R. C., Diez, C. R., & Tseng, K. H. (2011). The impact of problem-based learning strategies on STEM knowledge integration and attitudes: An exploratory study among female Taiwanese senior high school students. International Journal of Technology and Design Education21(2), 195-215.

Luminant Software (2017). AudioNote [software]. Available from

Mohammadpour, E., Shekarchizadeh, A., & Kalantarrashidi, S. A. (2015). Multilevel modeling of science achievement in the TIMSS participating countries. The Journal of Educational Research, 108(6), 449-464.

Museum of Science. (2015). Technology flashcards [software]. Available from

Museum of Science. (2016). Engineering adventures – Messages from the duo [software]. Available from

National Center for Education Statistics (2011). Trends in International Mathematics and Science Study student grade four questionnaire.  Retrieved from

National Research Council. (2013). Next generation science standards: For states, by states. Washington, DC: The National Academies Press.

Oh, Y., Lachapelle, C. P., Shams, M. F., Hertel, J. D., & Cunningham, C. M. (2016, April). Evaluating the efficacy of Engineering is Elementary for student learning of engineering and science concepts. Papper presented at the annual meeting of the American Educational Research Association, Washington, D.C.

Oware, E., Capobianco, B., & Diefes-Dux, H. (2007, June). Gifted students’ perceptions of engineers? A study of students in a summer outreach program. Paper presented at the Annual American Society for Engineering Education Conference and Exposition, Honolulu, HI.

Puentadura, R (2013). SAMR: Getting to transformation. Retrieved from

Rojas, J. Reser, J., Usher, E., & Toland, M. (2012).  Psychometric properties of the academic grit scale. Retrieved from

Saravia-Shore, M., & Garcia, E. (2008). Diverse teaching strategies for diverse learners. In R. Cole (Ed.), Educating everybody’s children: Diverse teaching strategies for diverse learners. Revised and expanded 2nd edition (pp. 41-97). Alexandria, VA:  Association for Supervision and Curriculum Development.

Schmidt, D., Baran, E., Thompson, A., Mishra, P., Koehler, M., & Shin, T. S. (2009). Technological pedagogical content knowledge: The development and validation of an assessment instrument for preservice teachers. Journal of Research on Technology in Education, 42(2), 123.

Simpson, A., & Maltese, A. (2017). “Failure is a major component of learning anything”: The role of failure in the development of STEM professionals. Journal of Science Education and Technology26(2), 223-237.

Tate, W. (2001). Science education as a civil right: Urban schools and opportunity‐to‐learn considerations. Journal of Research in Science Teaching, 38(9), 1015-1028.

Taylor-Powell, E., & Calvert, M. (2006). Wisconsin 4-H youth development: Arts and communication program evaluation. Retrieved from  PLEASE PROVIDE A WORKING URL FOR THIS DOCUMENT

Torkzadeh, G., & Van Dyke, T. P. (2002). Effects of training on Internet self-efficacy and computer user attitudes. Computers in Human Behavior, 18(5), 479-494.

University of Pennsylvania. (n.d.). 8-item grit scale (adapted for children). Retrieved from

University of West Australia. (2016). Cane toad [software]. Available from

VoiceThread (2017). VoiceThread [software]. Available from

Warner, M., Caspary, K., Arshan, N., Stites, R., Padilla, C., Patel, D., … Adelman, N. (2016). Taking stock of the California Linked Learning District Initiative. Seventh-year evaluation report. Menlo Park, CA: SRI International.

Appendix A
Engineering and STEM Attitudes Survey

Grit (pre-assessment only)

I am a hard worker.[c]

I am hard working and careful. (Item simplified from original question, “I am diligent (hard working and careful).”[c]

Attitudes Toward STEM (pre-assessment and post-assessment)

I like thinking of new and better ways of doing things.[a]

I enjoy learning math.[b]

I enjoy learning science.[b]

I like using technology.

Math is harder for me than for many of my classmates.[b]

Science is harder for me than for many of my classmates.[b]

Views of the Jobs of Engineers and Scientists (pre-assessment and post-assessment)

I think I know what scientists do for their jobs.[a]

I think I know what engineers do for their jobs.[a]

Attitudes Toward the STEM Workshops (post-assessment only)

I would be interested to take another workshop like the one I just took but on a different topic.

I learned about engineering in the workshop.


[a]Source of item: Engineering is Elementary, 2010

[b]Source of item: NCES, 2011

[c]Source of item: University of Pennsylvania, n.d.

Appendix B
Focus Group Protocol

We will be talking about the math and science workshop. This includes ALL lessons/activities/projects/events.

Ground Rules

Everyone’s opinion is important; we want to hear from each of you.

Let everyone talk; but not at the same time.

This isn’t a test – no right or wrong answers; your opinions count.

We’ll listen respectfully.

We won’t interrupt or cut in but neither will we dominate the time.

We are taping the session just so we are sure to correctly capture what you say.

No names will be identified with any remarks.

At the end of the discussion you can stay and listen to the recording.

If after hearing the tape, you want to add something to the tape, you can.

Let’s make sure all cell phones, etc. are turned off.


  1. First, let’s get to know you a little bit… Why did you get involved in doing this summer workshop?
  2. Now, let’s talk about the workshop… How did it go?
  3. After participating in the workshop, did you learning anything new?
  4. After participating in the workshop, did your views of math or science change? (if yes, “can you say more about that?”).
  5. If you could change anything about the workshop, what would it be?
  6. Are the parts that we should be sure to keep?
  7. Now, let’s focus on the elements of the workshop project(s)…. What did you like about the projects?
  8. Which weeks did you like the best? – week one, two, three, or four? Why?
  9. Did you experience challenges or difficulties with the projects? Which weeks were the most difficult?
  10. Were you able to work around those challenges? How?
  11. Do you think math or science is difficult? Why/ Why not? Can you give an example from the workshop?
  12. (Show the Engineering Design Process diagram that is shown without labels for the stages.) Do you remember seeing or doing something like this? Can you explain this diagram? What are the different steps?
  13. Did you use iPads? How was that?
  14. Are you involved in other science, math, or technology activities? If so what are they? Where do you do them?
  15. Is there anything else you’d like to add?