Year-End Statistical Survey of CMU Students

Institute for Research and Analysis Presentation
Rebecca Nathan and the IRA employed different approaches in their studies of college life, but they were trying to answer the same questions. While I was certainly interested in seeing what the data showed from their end of the year survey (their biggest survey which they conduct only every 4 years), I found myself evaluating which method of research yields a richer and more accurate portrait of college life. According to their presentation, the survey was constructed and conducted to answer certain questions. These are the ones I thought were important: How have students experienced growth in their own skills? How satisfied have they been with their academics? What on/off campus extra/co-curricular things do students participate in? To measure students academic and non-academic engagement. What are students’ opinions on university values—how well does the school teach/follow its values, does the student follow them? Nathan’s ethnography also answered many of these questions.

In her book she mentioned the moves she made to make her study morally and academically valid. The one that comes to mind is how she hid her true identity from the students, something which anthropologists by rule do not do even though I know I wouldn’t talk candidly to her knowing she was a teacher from my own school. In this presentation, they explained at length the pre-survey work that goes into clarifying and removing ambiguity from questions, choosing their survey sample and weighing the results to reflect the population as a whole. Lets get back to my evaluation: which method answers their questions better? Surveys have a lot of flaws that aren’t always apparent. Ten people can interpret a survey question or survey answers ten different ways. Allowing a sample to represent a larger population gets even more complicated when certain groups decline the survey more than others. This survey well represented the student body in race, citizenship, class year, but not on gender or college, forcing the surveyors to weigh their data so no groups are under represented. But so many such adjustments must sometimes be made to a sample that by the time you interpret the results, its hard to remember exactly who the results fairly represent. The high esteem we grant culturally to science and math (which carries on to graphs) makes it harder to question and therefore understand data presented on a graph. Take the Inconvenient Truth for example: it uses graphs to create a visual (emotional) reaction, not a logical one. Ethnography on the other hand is much fairer. It doesn’t require meddling and adjusting. Of course it isn’t free of bias, but the bias is all up front. Ethnography relies on anecdotal evidence which we know doesn’t necessarily extend to the greater population. When you recognize that Nathan is able to include certain interviews or quotations and omit others, an ethical fault of someone trying to write an expose, it doesn’t change the reliability of the data because anecdotal evidence doesn’t speak for everyone. But in the end, you know that the individual views expressed are real and valuable while an individual survey response is only valuable if lots of other people agree with it.

Specific results from the presentation: more women report (greater or a higher rate) of maturation than men. Women, minorities, US students say they engage in discussion more than men, majorities, international students, respectively. Students off all demographics report spending more time on classes and homework than social, sports, club activities, and report spending even less time with teachers outside of class. On weeknights, the average student gets 6-7 hours of sleep (a minority gets 4- on one extreme or 9+ on the other extreme hours). Over the weekend however, it is more typical for a student to get nine hours of sleep, 4- hours being the least common.

Back to my rant on surveys. This survey tells us a lot of whats (forgetting our methodology issue), but no whys. For example, why do women report maturing more over college than men? Do men think they started out very mature and therefore reported less, or do men recognize that they have not matured much? What is the reason for this gendered difference? Why do students of certain demographics engage in class discussion more than others? Surveyors can come up with hypotheses to answer these questions, but the survey itself can never answer these ‘why’ questions. In order to do so they put together focus groups to get more detailed answers. The ethnography allows researchers to gain a greater depth of understanding. Why do some international students feel isolated? Amercan sayings such as “see you later” or “how are you” are misleadingly friendly.

I think in the end, although Nathan may have had an agenda, within the constraints of who it actually represents, her book gives us a much deeper understanding of those people’s experience in college. With survey data it is hard to show who it actually represents, and it still only gets us half way to answering the more complicated things we really want to know about students.