What Happened With My Open Laptop-Exam Class (Part One)

Background My plans for an open-laptop exam generated a fair amount of buzz when I announced them in the fall, so I figured you would be interested to learn more about how things played out. Annenberg PhD Student Adam Kahn, who helped design this curricular intervention/innovation, is still working through a massive amount of survey data collected about the process, so any observations I share now are provisional based primarily on what I saw from in front of the lecture room and on exit surveys students completed after turning in their final exams.  In general, I think the experiment was successful, even though, with any design process, there are many things I would change on the next iteration. And, as we will see, the experience had some critics among the students in the class.

To remind you, the basic set up was this: Students completed a series of individual assignments throughout the first part of the term, which counted for 50 percent of their total grades. In the second part, they were put onto teams, which worked together on every assignment, including a series of weekly problem sets conducted in the discussion section, contributions to class discussion, and the final exam. Students had to attend the discussion section in order to receive the team’s points for their contributions, but otherwise, participants received their grades based on collective rather than individual performance. We introduced this process into a 200 level lecture hall class on New Media Technologies and Culture, with a population of 110 students, mostly Communication majors, taking what was a required subject for their degree.  You can see the syllabus for the class, including the assignment structure, here.

Impact on Class Discussion

My first observation was that the emotional tone of the class shifted dramatically following the midterm as we placed students on teams. The teams sat together in the lecture hall; they chose a shared name, and they used that name to identify themselves when they participated in the class discussions. From the start, there was a strong sense of team identity for most of the groups. I’ve speculated that this approach might work especially well in the context of USC where there is such a strong sports culture.

From the start, I had placed a strong emphasis on class participation during lecture sections, trying to move towards a more Socratic approach to teaching the content. There had been push back early on when I relied too heavily on discussion, and so I had tried to find a balance between short lectures designed to introduce core concepts and then more open ended discussion to allow students to share their perspectives on core debates of the digital age. We struggled a bit with managing discussion in a large lecture hall context: students balked at the mechanics of passing around microphones, but some of the students had trouble being heard in the large space and were thus more reluctant to speak. Over the course of the term, the process started to feel more natural for both the teacher and the students, and we had some very engaged and informed conversations.

As with any discussion class, there were a number of students who were quick to raise their hands and engage, while there were others who were intimidated by the large size of the class. The most active participants continued to dominate discussion in the second half, but there were many others who made their first contributions during this period, either empowered by having teammates supporting them or by the sense of competitiveness that teams introduced into the mix. As one student explained, "I liked that we all sat together during lecture. This enabled us to whisper about the lecture content and, all together, come up with a question to pose or a comment to offer." More dramatically, team members were much more likely to anchor their statements to specific statements or information contained within the readings. Indeed, it was clear that a much higher percentage of the students had done the readings and done them closely knowing that they were dependent upon each other for the quality of information being transmitted to the group.

A highlight of the course came when we conducted a role playing activity in one of the lecture sessions focused around debates about digital piracy and the evolution of new business models for the music industry. Each team was assigned a specific role -- from new artists trying to break into the industry to recording studio executives, from fans to teachers and librarians, from religious performers to international musicians who are developing a following in the United States. The teams were assigned their parts in advance and encouraged to do a little home work so that they had thought through their assigned perspectives. Each group was asked to make an opening statement, which were surprisingly well informed, for the most part, and then, they were given time to negotiate across groups to see if they could identify common interests and propose new solutions to the issues. This was the only time in the term when we encourage activity across groups rather than within groups, and multiple students pointed to this activity as transformative in terms of their understanding of the value of the team process. It also resulted in a spectacular discussion which got students out of familiar debating points around issues of digital piracy and allowed them to develop a more systemic understanding of the issues. I would love a way to create more such experiences across the class the next time I teach it.

Working Within Teams

Students were placed randomly on teams, in the hopes of insuring greater diversity. On the one hand, we felt that if students self-selected teams, they would be more likely to choose people with whom they already shared many common interests, i.e. people who were like themselves. On the other hand, we also wanted to avoid the common pattern of consciously combining strong and weak students onto teams together, which tends to result in the stronger students being asked to carry the load by themselves.  In the exit surveys, students were sharply divided between those who felt that the random assignments insured that they met new contacts and brought more diverse knowledge together and those who felt that some of the logistical problems they encountered would have been minimized if students had been able to work with people they already knew.  Here, for example, was a student who valued being randomly assigned: "When my group worked, we worked efficiently because we didn't know each other at all, so there were few distractions. We were friendly, but didn't have a lot in common, which was conducive to learning the subject material." Yet this student also noted that their lack of familiarity with each other could sometimes result in a lack of accountability:" I didn't make it to class the first day and realized later that no one in my group had taken any initiative to do the necessary organization for future readings, in-class work, etc. No one was really a leader. We couldn't count on each other. There were no ground rules set, etc.”  Some students wanted better mechanisms for dealing with students who failed to contribute to the collective good: "“I think the students should either be able to choose their own groups or somehow get rid of the weakest link." The large scale of the lecture class makes it particularly likely to attract students who are not strongly motivated by the subject matter and who are likely to exploit the good will of their classmates.

Each team consisted of 3-5 students (with the unevenness a product of the uneven number of students who had registered for the different discussion sections which met at different days and times). It was clear from the start that the larger teams worked better, overall, with smaller teams more vulnerable to individual students who let down their team through under-performance.

Most of the teams became effective learning communities, but not all of them did. We had taken steps to insure shared expectations of members, asking each team to write a contract together so that they had a mutual understanding of their responsibilities to each other. We had built in one core check on group participation -- i.e. the students had to attend the discussion section and work on the problem set in order to gain credit for that assignment.  Otherwise, we relied on social mechanisms to insure that they held each other accountable. Through these weekly problem sets, students gained practice working together, learning each other’s strengths and vulnerabilities. We had felt using the discussion sections in this way would insure some regular face-to-face time between group members (as did having students sit in team during lecture).

Overall, attendance in discussion sections increased with the emergence of a team structure, though there were still many students who still did not attend class regularly, a manifestation of the “free loading” problem which often crops up when working within a commons. And for those teams which were struggling with the process, there was a perception that the instructors were not doing “anything” about it. We wanted to resist the temptation of shuffling the teams once the process began, since doing so would be likely to disrupt the coherence of those teams which were functioning well, since we wanted to encourage teams to find ways to work through their own problems seeing learning to self-correct their process as an important learning opportunity. In many cases, teams that did not gel at first did find their footing over time, part of the value of repeated experiences working in teams, while in some cases, teams that had worked well up until that point hit real friction when they turned their attention to dealing with the high stakes final exam. Here, for example, was a student who felt the group had gotten in the swing of things just in time for the exam: "My group members let me down on numerous occasions but our final went so well and so smoothly that I'm having a hard time deciding how I felt about the whole thing overall.”  TAs did give advice to team members who were having a frustrating time; we felt that there were penalties built into the system for those members who under-perfomed -- again, the fact that they did not get points for sections which they missed and the likelyhood that underperforming students had also underperformed during the individual portion of the class.  Next time, I want to provide much greater advice to the students about strategies for insuring team cohesion and meaningful interaction.

We struggled with the question of whether we should have introduced some self-evaluation process where team members could assess what each contributed to the process and so that we could adjust grading accordingly. We choose not to do so for several reasons: We feared that such a practice might further fracture teams which were struggling to survive, raising the tension level at the time when we wanted teams to be developing greater trust in each other, and as importantly, we felt that it would be inappropriate to change the rules of the game mid-process.  Next time we do this, I am going to weigh this question again more closely, since the lack of such formal mechanisms was the single most frequent complaint we heard about the group activities.

Designing Problems

Designing the problem sets for the discussion section proved challenging for a number of reasons. We wanted the questions to be sufficiently challenging so that students were motivated to put in the extra efforts and also be able to see that they could indeed do more collectively than they would have been able to do individually. We wanted the questions to be open-ended enough so that students could show what they knew, bring their individual and collective knowledge beyond the class into the process, and have a chance to dig deeper into their own passions and interests. We also wanted to have questions which relied on as many of the readings from the week as possible, since we were encouraging students to divide up the readings between them and then deploy what they needed in response to each problem. Early on, it was clear the teams needed more guidance on the best way to find the information they needed, and the challenges of working in a hour long discussion section (well, 50 minutes really) meant that we needed to simplify the options in order to allow students to get out of the gate quicker. Here, for example, is how one student described their team's frustrations:

"The assignments given in discussion sections were rather long and difficult for the amount of time allotted to students to complete them. The assignments also placed a large emphasis on the skill of being able to produce quick thoughts and responses to questions that students were not fully prepared to answer. If the questions were given prior to coming to class, it would have helped to allow students to come in more prepared and produce more thoughtful and engaging responses."

We streamlined the problems week by week, but students still complained that they did not have time to fully complete the assignments during the class period. (I am going to share with you the assignments in a follow up series of posts).  We had been reluctant to extend the time working on the problem sets because we were afraid the most anxious students would turn them into a much bigger project than intended and because extending them beyond the class time would increase the logistical challenges involved in working with teams.

While most of the students complained about the time constraints, some felt like we had achieved an ideal balance: “I think that the discussion section questions struck the perfect balance in that they pushed the students to produce a lot of quality work in a short amount of time, yet it was completely fair as our knowledge was collaborated from what we obtained throughout the week. I was always very satisfied and impressed with the work we were able to produce in such short periods of time.” Some students used the practice runs to rehearse strategies and refine skills in preparation for the final: “The activities done during discussion section were also beneficial because you could kind of gage what people's strengths and weaknesses in the course material were and how it can be applied to the final.”

Overall, we felt the quality of the problem set responses were strong, with most of the teams scoring in the A-B range, and with signs of general improvement over time, suggesting that, in most cases, the teams were learning to work better together each time they confronted a new problem.

(MORE TO COME)