VI. Evaluation and Research


Evaluation and research will be a critical component of CIRTL, embedded in all of its activities. Fairweather (MSU) will lead an experienced and nationally recognized multi-institutional Evaluation and Research Team of Austin (MSU), Baldwin (MSU), Cabrera (UW), Colbeck (PSU), S. Millar (UW), and Webb (UW). Our research program will be one of action research (National Research Council, 2000). The Evaluation and Research Team will be involved in all CIRTL activities, gaining important knowledge about changing graduate-through-faculty practices and beliefs toward learning through teaching. Indeed, the graduates-through-faculty will be key informants as they study their own teaching. Through this participation, we will learn about critical aspects of the change process difficult to study through surveys and observations. Equally important, the Evaluation and Research Team will be able to inform and guide CIRTL development and transfer activities.


VI.a. Evaluation

The CIRTL Evaluation and Research Team will bring together a group with extensive experience in conducting studies over the full range of educational experiences. Team members have conducted evaluations of numerous education programs, including ECSEL, GK-12 graduate student fellowship programs, education systems as part of NISE, teacher professional development programs, and K-12 curriculum programs.

Ongoing formative evaluation will ensure that CIRTL effectively achieves its goals in a timely way. Using outcome measures described below, the Evaluation and Research Team will provide an ongoing assessment of CIRTL’s interim progress that will enable the CIRTL co-directors and team leaders to make midcourse corrections and improvements. As part of a continuous improvement management model, we will develop an electronic monitoring system to enable each project to report its progress to CIRTL management. As problems are identified, the Evaluation and Research Team will use ethnographic interviews to examine programs, uncover reasons for any poor performance, and make recommendations for improvements. Because the CIRTL development activities and first implementations will be at UW, these formative evaluation efforts will be led by Webb and primarily executed by UW evaluators.


Ultimately, CIRTL’s success will be measured by achievement of its five goals (section I). We will evaluate successful achievement of these goals as follows:

Principal Objective for Goal 1 Develop tools and strategies that prepare and motivate STEM graduates-through-faculty to use and further improve best practices in STEM teaching and learning for all students. Throughout the CIRTL Network, we will measure and compare graduate-through-faculty instructional practice pre- and post-CIRTL participation, particularly with respect to best practices, effectiveness with diverse audiences, and engagement in effective teaching-as-research. We will also compare the attitudes and instructional practices of graduates-through-faculty who have had extensive CIRTL participation with those who have had limited or no direct exposure to CIRTL activities. A number of instruments exist for measuring classroom instructional practices, classroom climate, students’ perceptions of their abilities in STEM, and student performance, including those of Cabrera et al., (2001), the National Survey of Student Engagement (www.indiana.edu/~nsse/), and the College Student Experiences Questionnaire ( www.indiana.edu/~cseq/).

Principal Objective for Goal 2 Develop learning communities that recruit, support, and sustain STEM graduate-through-faculty reform in teaching and learning and lead to institutional change. We will measure the number of graduate-through-faculty participants in CIRTL activities as a function of gender, race/ethnicity, and discipline. We will also measure changes in participation over time at institutions throughout the CIRTL Network. We will document the spread of reforms from participating graduates-through-faculty to their colleagues. We particularly seek to document the role of the learning community in attracting graduates-through-faculty to CIRTL. We will measure the extent that CIRTL-developed activities have been incorporated and funded as regular operations throughout the CIRTL Network. Relevant indicators include the development of new organizational structures to carry out CIRTL or the integration of CIRTL into existing administrative units; increased institutional and departmental commitment to teaching as indicated by faculty rewards and incentives for faculty development in instruction; widespread use of CIRTL-recommended practices; and reallocation of resources for improved teaching and learning.

Principal Objective for Goal 3 Transfer and implement the CIRTL program throughout the CIRTL Network. We will examine the effectiveness of various transfer strategies between institutions in the CIRTL Network. The focus will be on measuring the success of adaptation and implementation of the CIRTL program via the tools and strategies produced by the CIRTL teams. We will describe the distinct institutional environments and contextual factors that impede or enhance the transfer of tools and strategies between institutions. We will use documentation and case study techniques to describe the effectiveness of the transfer of tools and strategies, taking into account the different institutional resources, governing structures, and prior efforts to improve STEM education. Measures of the effectiveness of these tools and strategies at the varied institutions will be attained through evaluating achievement in Goals 1 and 2.

Principal Objective for Goal 4 Demonstrate that participating STEM graduate students and postdoctoral researchers show a commitment to use of best instructional practices in future faculty positions, and to ongoing improvement of those practices through teaching-as-research. To determine the effects of the collective CIRTL experiences on STEM future faculty, we will conduct a longitudinal study of a cohort of UW STEM Ph.D. students and post-docs over 5 years. We will collect baseline data on the entering cohort and document the attitudes of this group about teaching and teaching-as-research at different points in time during their program. We will track their teaching practices for those who do teach during the 5 years. In the 5th year, we will again assess attitudes. Our assessments will include their views about student engagement in their own learning, knowledge of active and collaborative instruction, awareness of teaching to diverse student audiences, and the practice of teaching-as-research (Colbeck, Cabrera, & Marine, 2001). When possible, we will follow the Ph.D. recipients into their new positions, document their attitudes and teaching practices, and compare these practices to those encouraged by CIRTL. We anticipate that this longitudinal study will follow CIRTL alumni after the 5-year duration of the center.

Principal Objective for Goal 5 Lead a national conversation about STEM graduate education and early faculty career development. We will judge effectiveness by the following kinds of indicators: (a) presentations at disciplinary and professional conferences, (b) evidence that the CIRTL agenda has been recognized and used by national associations, (c) amount of use of Web-based and other dissemination tools, and (d) publications in major outlets. We will also look to our National Advisory Board (section VIII) to judge our progress in leading a national conversation on improving teaching and learning of STEM education at the undergraduate level.


VI.b. Research

The Evaluation and Research Team will produce knowledge on STEM professional development and reforms needed to achieve significant change in STEM higher education. The team will conduct a broad survey and analysis of existing STEM professional development and graduate preparation programs throughout the U.S. Team members will visit other higher education institutions to document the teaching needs and concerns of the full range of institutions that hire STEM graduates, including community colleges, historically black colleges and universities, liberal arts colleges, masters-level universities, and research universities. These studies will inform the design of the CIRTL program and will be important early deliverables.

A particular emphasis of the research program will be the transfer of successful programs between research universities. Relevant questions include: (a) What transfer and institutionalization approaches are most effective, and in what institutional contexts, taking account of different institutional resources and different levels of prior involvement in efforts to improve education in STEM? (b) What challenges and barriers occur in the use of various strategies for transfer and institutionalization? (c) What general lessons apply to non-CIRTL institutions wishing to use faculty development and graduate education as a lever for improvement in teaching and learning in STEM?