Overall Effectiveness of UNIV 101 as Measured by the First-Year Seminar Assessment
The effectiveness of UNIV 101, as measured by the First-Year Seminar Assessment’s Overall Program Effectiveness factor, continues to improve from year to year (see Figure 1). The factor mean increased to 5.89 on a 7-point scale in fall 2017 (N=2,113).
The Overall Program Effectiveness factor assesses the degree to which students report the course:
- included interesting subject matter (M=5.79)
- contributed to their ability to succeed academically (M=5.84)
- contributed to their ability to adjust to the college social environment (M=5.91)
- covered topics important to the student (M=5.84)
- is one to recommend to other first-year students (M=5.6.08).
Overall program effectiveness, fall semesters 2008-2017. Data retrieved from the First-Year Seminar Assessment and presented on a 7-point scale.
Relationship of FYI Factors and Persistence
Research suggests that helping students establish a sense of belonging is the number one predictor of their decision to persist to their second year.
Overview & Methodology
The purpose of this study was to determine which factors on the First-Year Initiative survey* best predicted student persistence to the second year of college. The FYI is a nationally benchmarked instrument used to measure the effectiveness of a first-year seminar. The data were collected through a web-based survey at the end of the fall 2008 semester from students enrolled in University 101 at the University of South Carolina. 2,014 completed the survey, yielding a response rate of 72%. Responses from the FYI survey were matched with information from the student data file (persistence and first-year grade point average) to provide additional outcomes to measure student success. After eliminating missing data, 1,446 students remained in the analysis.
EBI**, who administered and disseminated the data, created 15 factors from a series of questions on the FYI survey. These 15 factors represent a number of vetted good practices and campus initiatives that are grounded in theoretical and empirical evidence. To predict the effectiveness of these factors on persistence into the second year, a series of logistic regressions were conducted. A number of control measures — gender, race and high school grades — were introduced into the model to isolate the effect of each factor on student persistence. Because of the high multicollinearity between the 15 factors, each factor was introduced into the model separately. When a regression model has high multicollinearity, this suggests that the independent variables (i.e. the 15 factors) are highly correlated with one another. The relationships between each factor and the dependent variable (i.e. persistence) become inflated and magnify the differences between coefficients. As such, each factor was introduced into the regression model separately to measure the unique relationship between the factor and persistence.
*In 2013, the First-Year Initiative survey was revised and renamed the First-Year Seminar Assessment.
**In 2015, EBI changed its name to Skyfactor.
The regression analyses yielded significant effects on two factors:
- sense of belonging and acceptance
- overall satisfaction with the university
A standard deviation increase in sense of belonging and acceptance increased the odds of persisting into the second year by 38% (p < 0.001), holding all other variables constant. On a scale from 1 to 7, the mean was 5.62 — with 95% of the students measuring between 5.55 and 5.69 — and a standard deviation of 1.33. Furthermore, a standard deviation increase in overall satisfaction with the university increased the odds of persisting into the second year by 94% (p < 0.001), holding all other variables constant. The mean for this factor was 5.74, with 95% of the students measuring between 5.68 and 5.80 and a standard deviation of 1.23.
It is important to note that the mean for each factor was relatively high. This suggests that students who participated in the survey had a high sense of belonging and acceptance and overall satisfaction with the university. Moreover, the standard deviations for each factor were low, indicating low variability between the scores on each factor. In other words, very few students indicate having a low sense of belonging and acceptance and a low overall satisfaction with the university. The extent to which we can surmise that the seminar is having a positive effect on these outcomes cannot be determined because we do not have complementary data for students who did not participate in the course.
However, this should not deter from the overwhelming significance that sense of belonging and acceptance and overall satisfaction with the university have on student persistence. Moreover, these data suggest that efforts targeted toward enhancing students’ sense of belonging and acceptance and overall satisfaction with the university will increase their likelihood of persisting into the second year.
Peer and Graduate Leader Contributions to the University 101 Student Experience
End-of-course evaluations from fall 2017 show the vast majority of UNIV 101 students agreed their peer or graduate leader made valuable and beneficial contributions to their UNIV 101 classroom and first-semester experience (see Table 1). End-of-course evaluations presented students with a series of six statements about their peer or graduate leader and asked them to indicate their level of agreement (strongly agree, agree, neutral, disagree or strongly disagree) with each item. Statements measured the peer and graduate leaders’ contributions in the classroom, their approachability and their stature as a role model and resource, as well as their role in helping students make a successful transition to college. Students also were asked to evaluate the peer and graduate leaders’ impact on their overall UNIV 101 experience.
|Evaluation Criteria||n||Mean||% Agreed|
|My peer/graduate leader made important contributions to the class.||3782||4.82||96.9%|
|My peer/graduate leader was approachable.||3780||4.87||97.8%|
|My peer/graduate leader was an appropriate role model.||3777||4.85||97.4%|
|My peer/graduate leader was a valuable part of my University 101 experience.||3776||4.81||96%|
|My peer/graduate leader helped me make a successful transition to the university.||3772||4.73||93.3%|
|My peer/graduate leader was a valuable resource.||3753||4.80||95.9%|
Note: Data from fall 2017 UNIV 101 end-of-course evaluation. Mean on 5-point scale. Percentage represents the number of students who strongly agreed or agreed with the statement.
Figures 1 and 2 present longitudinal data showing that students’ satisfaction with their peer and graduate leaders’ contributions is high each year, with satisfaction in four of the six areas increasing in fall 2017.
Student satisfaction with peer and graduate leaders’ role and contributions, fall semesters 2012-2017. Data retrieved from UNIV 101 end-of-course evaluations and presented on a 5-point scale. Note: Peer and graduate leaders’ approachability was not addressed on end-of-course evaluations prior to 2014.
Student satisfaction with peer and graduate leaders’ role and contributions, fall semesters 2012-2017 (cont.). Data were retrieved from UNIV 101 end-of-course evaluations and presented on a 5-point scale.
Impact of Matching Process on Peer and Graduate Leader Satisfaction
Research suggests that peer and graduate leaders who get matched with a co-instructor by the University 101 Programs office have the same satisfaction with their role and teaching partner as those who find a match on their own.
Data from the fall 2016 end-of-semester University 101 peer and graduate leader evaluations were analyzed to determine whether the process used to match students with co-instructors — either a student and co-instructor making a match on their own, or a match made by the University 101 Programs office — had any impact on the peer or graduate leader’s subsequent satisfaction with his or her role in the UNIV 101 classroom or relationship with his or her co-instructor.
Peer leaders indicated on their Peer Leader Program Evaluation whether or not they knew their co-instructor before being matched; graduate leaders indicated on their Graduate Leader Survey whether or not they selected their co-instructor on their own. Both the Peer Leader Program Evaluation and the Graduate Leader Survey asked respondents to report the degree to which they were satisfied with their role in the classroom. The wording of questions regarding the students’ relationships with their co-instructors differed between the surveys (see Tables 1 and 2), but each addressed the degree to which the students were satisfied with the relationship.
Data were analyzed to find differences in means on the selected questions for each group. Independent samples T-tests yielded no significant differences between peer leaders or graduate leaders who made a match on their own and those that had a match made by the University 101 Programs office.
Peer Leader Satisfaction with Class Role and Relationship with Co-Instructor
|Evaluation Criteria||Knew Co-Instructor
|Did Not Know
|I was satisfied with my role in the UNIV 101 classroom.||4.36 (36)||4.42 (104)||.695|
|I was satisfied with my relationship with my UNIV 101 co-instructor.||4.53 (36)||4.51 (104)||.911|
Note: Data from fall 2016 Peer Leader Program Evaluation
Graduate Leader Satisfaction with Class Role and Relationship with Co-Instructor
|Evaluation Criteria||Selected My Co-Instructor On My Own
|Paired by University 101
|I was satisfied with the role I had in my UNIV 101 class.||4.62 (13)||4.35 (17)||.338|
|If I had it to do over again, I would work with the same instructor again.||4.46 (13)||4.24 (17)||.604|
Note: Data from fall 2016 Peer Leader Program Evaluation
Impact of Peer and Graduate Leaders on Overall Program Effectiveness
Research suggests that first-year seminar courses co-taught by a peer or graduate leader are more effective than classes taught only by an instructor.
Research about peers has found that students can, and often do, have significant positive impacts on the development and learning of their peers. To that end, University 101 Programs utilizes peer and graduate leaders as resources, mentors and co-facilitators in the UNIV 101 course.
Data from the 2013 First-Year Seminar Assessment were analyzed to determine the impact of peer/graduate leaders on the UNIV 101 course, as measured by the FYSA's Overall Program Effectiveness factor. The overall program effectiveness factor assesses the degree to which students report the course:
- included interesting subject matter
- contributed to their ability to succeed academically
- contributed to their ability to adjust to the college social environment
- covered topics important to the student
- is one to recommend to other first-year students.
The FYSA is administered at the end of each fall semester to all students enrolled in UNIV 101. In 2013, the instrument was sent to 3,848 students and yielded a 59% response rate (n=2272). The dataset was coded to include a variable that indicated if a section was assigned a peer or graduate leader or had no teaching partner assigned. In fall 2013, 191 sections of UNIV 101 were assigned a peer or graduate leader while 22 sections had no teaching partner. The data were analyzed to find differences in means on overall program effectiveness between sections that had a teaching partner and those that did not. An independent samples T-test yielded significant differences in overall program effectiveness for sections of the course with a peer leader (M=5.49, SD=1.56) and sections without a peer leader (M=5.14, SD=1.78); t(2270)= 9.46, p=.01
Impact of Peer/Graduate Leader on Overall Program Effectiveness. Data were retrieved from the First-Year Seminar Assessment and presented on a 7-point scale.