Panel Session 502 to be held in 212 A
Sponsored by the Systems in Evaluation TIG and the Pre-K - 12 Educational Evaluation TIG
Chair(s):
Patricia Jessup, InSites
Discussant(s):
Alyssa Na'im, Education Development Center, Inc.
Presenters:
Ginger Fitzhugh, Evaluation & Research Associates
David Reider, Education Design Inc.
Karen Peterman, Karen Peterman Consulting Company

Abstract: Three evaluators discuss how they have begun to incorporate systems constructs and strategies into traditional evaluations of science, technology, engineering, and math (STEM) education programs. Systems thinking can assist in understanding the complexity of STEM programs, and enhance evaluations’ relevance and utility. Panelists will each present a short case study about how they have integrated systems thinking into their evaluations, describing specific systems concepts they have applied to their evaluation designs, data collection, data analysis, and reporting. The panelists are participants in a community of practice to develop evaluators’ capacity to use systems thinking (ECLIPS: Evaluation Community of Learning, Inquiry, and Practice, funded by the National Science Foundation), and will discuss the challenges and successes they have faced being systems novices. Following the presentations, the panel chair and discussant will facilitate a reflection amongst the presenters and session attendees about strategies and potential value for introducing systems thinking into evaluations.

Challenges and Successes Associated With Introducing Systems Concepts to an Existing Evaluation
Ginger Fitzhugh, Evaluation & Research Associates
Is it possible to introduce systems concepts to an existing evaluation? This presentation will describe the successes and challenges an experienced evaluator new to systems thinking faced when incorporating systems concepts into an evaluation of the national expansion of an underwater robotics program for middle and high school youth. The presenter will describe how she began incorporating systems concepts halfway through the five-year evaluation although faced with a limited evaluation budget and scope. In particular, the presentation will describe how systems concepts of boundaries, relationships and perspectives were incorporated into existing data collection, analysis and reporting structures. She will also describe how she introduced systems thinking concepts to the project team and coordinated with a second evaluation team who shared responsibility for evaluating the project.

Evaluating Scale-up Programs from a Systems Perspective
David Reider, Education Design Inc.
Innovative Technology in Science Inquiry (ITSI) brings inquiry-based science projects that use computational models and real-time data acquisition to grade 4-12 classrooms in four states. This presentation will focus on evaluating how the technology and science learning cultures, needs, practices, and policies of four different states influence the success and traction of the project using specific habits of systems thinking: (1) systems changes over time, (2) systems generating their own behaviors, (3) circular nature of cause and effect relationships, and (4) identifying systems leverage actions. This effort also involves modifying the evaluation plan partway into the five-year project to accommodate systems thinking structures. The presenter will discuss how he selected systems strategies, how he communicated the changes to the design team, and the impact the changes have made to the evaluation.

Blurring the Lines: The Process of Applying Systems Thinking to an Ongoing Evaluation
Karen Peterman, Karen Peterman Consulting Company
This presentation will focus on the ways that a seasoned evaluator has blurred the lines commonly associated with external evaluation in an effort to integrate systems perspectives into an ongoing evaluation plan. The presenter will focus on the rationale behind decisions to (1) share data collection responsibilities with the client in an effort to preserve the integrity of an existing system, and (2) extend the analysis plan beyond the scope of the original evaluation questions to try to understand the larger context of how constructs relate to one another and the project. The presenter will also describe how the addition of two new evaluations has provided the opportunity to think about how these programs relate to (a) each other, (b) the mission of the client institution, and (c) archetypal systems structures. The successes and challenges of blurring the lines to embrace systems thinking will be addressed.