Yes, You Can Use Your Laptop on This Exam...and Your Knowledge Community, Too!

In February 2011, I shared with my readers a pedagogical problem I was facing in a large undergraduate lecture class on new media and culture:

I made the announcement that the exams in the class would be open book, open note and that I was planning to distribute a list of potential questions in advance from which I would draw in constructing the exam, a practice I have used for more than 20 years without any great confusion....

No sooner did I announce this policy than I got a question I've never been asked before. A student wondered whether open book, open note, meant open laptop. I needed time to reflect on this and said I would answer in the next class period. Actually, it took me a few to get back to them with a response. Given this was a class on technology and culture, I decided to use this as a teachable moment.

So, I started by breaking down the computer into two elements. First, there is the computer as a stand alone word processing machine. I certainly would have had no great objections to students using the computer to write their answers or even to access their materials.... But, in an era of networked computing and wireless classrooms, allowing students to use a laptop during an exam suddenly would allow students to access any information anywhere on the web and more significantly would allow students to trade information with each other throughout the test in ways which would be extremely difficult to monitor.

As I thought about it, the challenges of designing a meaningful test under those circumstances intrigued me. What would it mean to create an exam which could be taken not by individual students but by networked groups of students -- either the class as a whole or a specifically designated study group? Could we enfold ideas of collective intelligence into the design of tests? Could we create challenges which demonstrated their mastery of the material through the search strategies they deployed and the knowledge they produced together? In theory, such an exam holds promise as more and more jobs require the capacity to pool knowledge and collaborate with a team of others to solve complex problems, and learning how to mobilize expertise under these conditions should be a key goal of our educational process.

I was not able to come up with an approach fast enough to implement it that semester. Readers to the original blog post made a series of suggestions, though most of them seemed to work better on the scale of the seminar or small group classroom rather than the lecture hall. But, my theoretical commitments meant this question was not going to go away. I also know that the question has haunted some of my readers, one of whom shared this interesting blog post with me via Twitter last week, which deals with other conceptions of what an open laptop exam might look like.

Ironically, having failed to create opportunities for collaboration inside the exam space, the students did what might easily have been predicted: they formed study groups outside class and worked through responses together. Many students had written entire answers to the provided questions in advance, and simply copied their answers into a blue book. In some cases, as many as thirty or forty students got the same question wrong and in the same way, suggesting just how expansive the study network (scarcely a study group) had become. The question may no longer be whether learning is going to be networked, but rather how much control faculty are able to exert over the networks where learning and studying take place.For me, this is part of the implication of the recent cheating scandal at Harvard.

Adam Kahn, a PhD student in the USC Annenberg School, read the blog post and reached out to me, suggesting that he would like to help me think through these challenges, since he is doing work on a body of research known as Transactive Memory, which is interested in the ways groups of people solve problems together. Kahn had been a student in my New Media Literacies class my first semester at USC. We've worked together off and on for the past year, developing a conceptual essay about the problem for a forthcoming book on higher education. There, we developed a blue print for how we might need to reinvent the lecture class process in order to support the emergence of knowledge-building and problem-solving communities of the kind required to successfully complete a networked exam. Here's part of what we said in that essay:

Because students add and drop classes for the first few weeks of the semester, it is important to not form student groups too early, as group turnover can hurt transactive memory (Moreland and Argote 2003). Thus, the semester will begin with core concepts and common texts the instructor feels everybody should know (Lévy’s shared knowledge). The course’s first midterm would be a traditional, non-collaborative exam, comprising a certain percentage of the grade. This diagnostic test can allow the instructors to gauge student’s relative abilities when putting together teams. Members should be assigned, with the goal of diversifying skills and knowledge.  Allowing students to self-select would almost certainly increase group homogeneity bound as membership would be to existing friendship ties, i.e. those who shared the same interests and activities outside the classroom.

After this exam, though, the group becomes important. At this point, the reading list will grow so that it is too much for any individual to read. Students will have to become mutually dependent to survive.

Because transactive memory forms around face-to-face communication, we must provide students a time to meet. Large lecture classes often have smaller discussion sections. Normally a teaching assistant facilitates a discussion and/or clarifies confusing points from lecture. However, this hour might be better spent allowing team members to meet to discuss the readings they divided amongst themselves and relate them to the common lecture themes of the week. The teaching assistant would also stress problem solving and coordination skills, helping groups refine strategies and learn from their mistakes, more like a coach than like our traditional model of a teacher. Although the ideal team would be self-regulating and self-guiding, the teaching assistant could also help them to assign roles or divide labor, if needed, to insure that each member pulls their own weight.

The teaching assistant can provide one question each week that would be representative of those on an exam, allowing group members to synthesize their different readings and learn how the other students think--their strategies for identifying the core stakes of a problem, mobilizing knowledge, testing data, assessing conclusions, and communicating results. These questions could require students to do online searches, tap into knowledge from other classes, or draw on their extracurricular expertise. Through these test runs, students would learn each other’s specialization, build trust, and coordinate their efforts on tasks similar to the group exam.

As they enter a collaborative test-taking process, students face the challenge of resolving conflict and committing to a shared answer, especially working under time constraints. Outside of the classroom, affinity groups develop norms, such as those surrounding contributions to Wikipedia, to which they can appeal to resolve such conflicts. So, for example, Wikipedia articles strive towards neutrality, which is often achieved through inclusion (that is, featuring all competing perspectives) rather than exclusion (arriving at a consensus response) (Lih 2009). Student’s experience of testing may be that there is a right answer the teacher is expecting and thus, they may be less receptive to test taking strategies which include a broader range of possible answers. Having multiple collaborative activities will allow each group to develop its own norms and protocols for resolving disputes and finding an answer students feel they can stand behind.

The final exam is designed to tap a range of different kinds of expertise. Think of the individual problems as possessing the sense of “meaningful ambiguity” which, McGonigal (2008, 214) argues, motivates the problem solving activity around alternate reality games: “by asking players to cooperate to make meaning out of an ambiguous system, the game-based hive mind celebrates individual perspective even as it embraces the larger, intricate intelligence that emerges only at the scale digital networks afford.” For such experiences to be compelling and satisfying, McGonigal (2003) argues, they have to introduce problems that seem within reach of the network of players. She notes that an empowered team often seeks to move beyond the game and tackle real world social problems, only to be disappointed that such problems may not, in fact, be resolvable given the group’s resources and capacities. Exam questions would need to be open ended enough to allow many different paths to a solution and yet ultimately something that participants can comprehend and resolve.

Assuming that the lecture meets twice a week, the questions are given out at the beginning of the first lecture and are due at the end of the second lecture. This will allow students to use the first lecture to start working on the answers and divide the labor. Then students can go home and seek more information on their own, and work more on the answers if they so choose. They coordinate efforts so that each student plays to her strengths and so that there is a robust system of checks and balances to identify and eradicate misinformation. Preparing for the exam may be much more like getting ready for a guild raid in World of Warcraft than like studying for a traditional test. The second lecture can be used to finalize answers. Also, by spanning two lectures affords groups at least two face-to-face opportunities to interact. Students can write their answers using an online tool, such as Google Documents, that allows them to write simultaneously in a single document. In this way, they can make changes to each other’s work (knowing who wrote what) and see changes being made to their own work. Changes can be tracked over time and reverted back to if needed....

Educational researcher Dan Hickey and his research team at Indiana University (Hickey, Honeyford, and McWilliams forthcoming) has been trying to explore what forms assessment needs to take within a participatory learning culture and concludes that assessment should “focus on reflections rather than artifacts.” His group has developed a range of activities that might follow a project or exam, asking students to reflect on what strategies they tried and why, rather than simply evaluating them based on what they produced. Of course, students will have different capacities to articulate their reflections. McGonigal (2008, 222) has similarly argued that working in large-scale teams to solve alternate reality games encourages “meta-level reflection on the skills and processes that players use to meet new challenges.” At the end of the day, the test might function as much as a probe to encourage students to continue to think about the process of their learning than as a simple assessment of what they, collectively and individually, know.

Well, this semester, we are going to be putting these ideas into action, as I teach my lecture hall subject a second time. I will  be sharing my syllabus next time.

I have ended up dividing the class into two parts: for the first part, students will be developing shared knowledge, that is, knowledge which will be required of every member on the team, and they will be performing as individuals, demonstrating their own mastery over the materials. In the second part, they will be sorted into teams which will work together on all future assignments: the discussion section times will be opportunities for the students to work on problem sets together with coaching from the Teaching Assistants, and the final week of class will be given over to a culminating activity which will require teams to work together to respond to the prompts.The midterm is approaching and soon we will be making the cognitive shift from individual to collective effort.

From the start, the class has emphasize new ways of learning in a networked culture, drawing heavily in the first few weeks on materials produced by MacArthur's Digital Media and Learning initiatives. I want to get them to take an inventory of their own skills and competencies as learners, the ways they use new media in the context of their lives, and to engage critically with the debates surrounding the so-called "digital natives" and their new media literacy skills. For this to really work, I am having to abandon the lecture as the primary mode of presentation in the class. Instead, I am moving towards something closer to the way Socratic Method works in Law Schools. On most days, we are reading essays which represent conflicting perspectives on core debates around digital media and culture, hoping to foster critical thinking and research skills.

As I developed this approach, I struggled with the issue of "freeloaders" -- that is, students who are willing to let the others do all the work and coast to a better grade. There's a limit to what I can do in terms of evaluating individual performance if we are going to really place such a strong emphasis on group performance, but I will be monitoring and evaluating individual attendance and participation in the discussion session and exam, and I will be asking participants to list everyone who contributed to a particular project (which can, in fact, include people who are not in their assigned group, as long as their participation is fully disclosed.)

There's a lot we still have to work through, so I would welcome feedback from readers about this approach and I would be especially interested to hear from anyone who has tried something like this process before. I promise to report back on how the class is going and share some of the problem sets we create later in the semester.