Data for the Villagers:
Using Survey Data for Change
Kathleen M. Morley and Randy L. Swing
April 20, 2006
Kathleen M. Morley completed her Ph.D. in Sociology in 2000 with a qualitative study on the social and academic integration of academically and racially/ethnically diverse first-year students at UMass-Amherst. Dr. Morley has had twelve years of institutional research experience at the University of Massachusetts Amherst. She has also served as Assistant to the Provost for Institutional Research at Franklin Pierce College in NH. In July 2005, Dr. Morley became Assistant Director of the Policy Center for the First Year of College. She has served as co-coordinator of the Self-Paced and Two-Year Institution cohorts participating in the Foundations of Excellence™ project.
Randy L. Swing is Co-Director & Senior Scholar for the Policy Center on the First Year of College. The Center’s signature project, Foundations of Excellence™, is an aspirational and measurement model for the first college year developed by over 200 colleges and universities. His recent research includes case studies of institutions of excellence in the first year and benchmarking first-year seminar outcomes. He is a fellow at the National Resource Center on The First-Year Experience where he produced two monographs on first-year assessment. Randy is a frequent speaker at national and international conferences. Until 1999, he held various leadership positions at Appalachian State University in assessment, advising, and first-year seminar. He holds a Ph.D in higher education from the University of Georgia.
.
Introduction
This essay is a follow-up to “It Takes a Village: A Task Force-Based Assessment Model for the First Year of College.” Its purpose is threefold: first, to address the use of survey data in developing an institutional perspective on the first year of college that promotes change; second, to introduce the Foundations of Excellence surveys; and third, to illustrate how these surveys can be used in the institutional change process. In the end, all assessment is about making improvements or confirming existing practices. As discussed in “It Takes a Village,” data frequently have not engaged the campus in the improvement process. This essay offers suggestions for increasing the likelihood that survey data will lead to institutional change for the first year of college.
Wouldn’t It Be Great If, in our efforts to use assessment to stimulate positive first year change on campus,
The real power in these principles is in their combined use.
Because the first-year experience is not limited to a specific program or series of initiatives, a comprehensive understanding of the first year of college requires that our assessment efforts focus on the entire first-year experience. Focusing on the first year as the unit of analysis allows an institution to consider both the full scope and all the interrelated components of the first year. From recruitment to achievement of sophomore status, a comprehensive analysis including the curriculum and co-curriculum can reveal areas of strength and weakness across the campus so that the institution can be strategic in making improvements.
An assessment that focuses on the entire first-year requires an institutional perspective of the first year. Current surveys on the first year tend to address student characteristics, levels of student engagement, and faculty interactions with students. The missing piece is data on what the institution does. The institution has control over organizational structures, policies, etc. that influence how faculty/staff and students approach the first year. By focusing on campus actions and decisions (i.e., what we can actually change) an institution can identify ways to influence behavior and, ultimately, outcomes.
An institutional perspective of the first year can be developed by collecting survey data from all the relevant stakeholders, including faculty, staff, administrators, and students. Contrasting perspectives allows the institution to identify the unique needs of certain subgroups and address those needs with targeted interventions. The variance in perspectives indicate where the institution could focus its change efforts. In addition, gathering multiple perspectives reveals the complexity of the first year, moving beyond a catalog description of the first year to “real-lived” experiences.
An institutional perspective of the first year can be further enhanced by engaging all the relevant stakeholders in the interpretation of survey data (treating the survey as a tool, not as a product). Unfortunately, data are often analyzed and summarized in isolation by institutional research or assessment staff. A full understanding of the first year of college requires intellectually substantive discussions that interpret survey results in the context of other forms of campus evidence and local knowledge. Allowing faculty, staff, administrators, and students to dig into the data, interpret results, develop meaning in relation to their own experiences, and come to consensus about the implications of the results for the overall campus produces a cohort of individuals with a deeper understanding of the complexity of the first year of college on their campus. A shared understanding by a cohort of “locals” constitutes a potentially powerful force for change.
Foundations of Excellence Surveys
Focusing on the entire first year, focusing on what we can actually change (i.e., campus actions and decisions), collecting multiple perspectives from all relevant stakeholders, and treating survey data as a tool rather than as a product are embedded in the Foundations of Excellence surveys and associated on-line analytical tools.
The Foundations of Excellence surveys are theory-based surveys, one for faculty/staff and one for first-year students (available in both two-year and four-year versions) which yield an institutional perspective on the first year. They are based on an aspirational model for excellence in the first year of college. Faculty, staff, and administrators are asked about the messages the institution communicates, the degree to which the organizational structure for the first year is effective, the assignment of resources, etc. First-year students are asked to what degree the institution has helped them connect with other new students, their advisors have discussed how college can help them achieve life goals, their instructors have communicated concepts clearly, etc.
The Foundations of Excellence surveys serve as a tool for discussion about first-year excellence rather than a product that lists survey results. The first year of college is too complex to capture all of the dynamics shaping it in one survey. Assessment of the first year at a particular institution calls for both professional judgment of campus constituents and local knowledge. The Foundation of Excellence surveys move beyond an average score to a complex understanding of dynamics shaping the first year by recording demographic items that allow campus members to examine differences within the campus.
For information about registering for the surveys see http://www.webebi.com/_AsmtServices/FYI/FOE.aspx. The surveys can be administered independently from the Foundations of Excellence project. Institutions that use the surveys independently will gain multiple perspectives about the effectiveness of institution’s first-year structures and policies. The on-line analytical tools allow an institution to track changes over time. The Foundations of Excellence surveys provide an holistic perspective on the first-year that confirm certain existing practices and can be used to generate serious discussions about the need for strategic improvements.
Conclusion
Numbers do not change institutions, people do. The collection of data on institutional practices is just a first step. By examining the ways in which an institution designs and delivers all components of the first year, assessment efforts can focus on issues the institution can control. To maximize change, data need to be examined for variance (among certain student populations) and put in the hands of “the campus villagers” so that rich, substantive discussions can inform improvement planning. Data-supported cross-functional discussions about the campus provide professional development, foster an institutional perspective, and engage the entire campus in the first year improvement process.
*****
The authors retain all rights to this essay. However, FYA-List subscribers may distribute the essay for non-commercial purposes.
Requested citation for redistribution or reference:
Morley, K. M. and Swing, R.L. (2006). Data for the Villagers: Using Survey Data for Change. Essay for the First-Year Assessment Listserv. Columbia, SC: University of South Carolina, National Resource Center for The First-Year Experience and Students in Transition. (http://www.sc.edu/fye/resources/assessment/essays/Morley-4.20.06.html)
*****
