Yes, You Can Use Your Laptop on This Exam…and Your Knowledge Community, Too!

In February 2011, I shared with my readers a pedagogical problem I was facing in a large undergraduate lecture class on new media and culture:

I made the announcement that the exams in the class would be open book, open note and that I was planning to distribute a list of potential questions in advance from which I would draw in constructing the exam, a practice I have used for more than 20 years without any great confusion….

No sooner did I announce this policy than I got a question I’ve never been asked before. A student wondered whether open book, open note, meant open laptop. I needed time to reflect on this and said I would answer in the next class period. Actually, it took me a few to get back to them with a response. Given this was a class on technology and culture, I decided to use this as a teachable moment.

So, I started by breaking down the computer into two elements. First, there is the computer as a stand alone word processing machine. I certainly would have had no great objections to students using the computer to write their answers or even to access their materials…. But, in an era of networked computing and wireless classrooms, allowing students to use a laptop during an exam suddenly would allow students to access any information anywhere on the web and more significantly would allow students to trade information with each other throughout the test in ways which would be extremely difficult to monitor.

As I thought about it, the challenges of designing a meaningful test under those circumstances intrigued me. What would it mean to create an exam which could be taken not by individual students but by networked groups of students — either the class as a whole or a specifically designated study group? Could we enfold ideas of collective intelligence into the design of tests? Could we create challenges which demonstrated their mastery of the material through the search strategies they deployed and the knowledge they produced together? In theory, such an exam holds promise as more and more jobs require the capacity to pool knowledge and collaborate with a team of others to solve complex problems, and learning how to mobilize expertise under these conditions should be a key goal of our educational process.

I was not able to come up with an approach fast enough to implement it that semester. Readers to the original blog post made a series of suggestions, though most of them seemed to work better on the scale of the seminar or small group classroom rather than the lecture hall. But, my theoretical commitments meant this question was not going to go away. I also know that the question has haunted some of my readers, one of whom shared this interesting blog post with me via Twitter last week, which deals with other conceptions of what an open laptop exam might look like.

Ironically, having failed to create opportunities for collaboration inside the exam space, the students did what might easily have been predicted: they formed study groups outside class and worked through responses together. Many students had written entire answers to the provided questions in advance, and simply copied their answers into a blue book. In some cases, as many as thirty or forty students got the same question wrong and in the same way, suggesting just how expansive the study network (scarcely a study group) had become. The question may no longer be whether learning is going to be networked, but rather how much control faculty are able to exert over the networks where learning and studying take place.For me, this is part of the implication of the recent cheating scandal at Harvard.

Adam Kahn, a PhD student in the USC Annenberg School, read the blog post and reached out to me, suggesting that he would like to help me think through these challenges, since he is doing work on a body of research known as Transactive Memory, which is interested in the ways groups of people solve problems together. Kahn had been a student in my New Media Literacies class my first semester at USC. We’ve worked together off and on for the past year, developing a conceptual essay about the problem for a forthcoming book on higher education. There, we developed a blue print for how we might need to reinvent the lecture class process in order to support the emergence of knowledge-building and problem-solving communities of the kind required to successfully complete a networked exam. Here’s part of what we said in that essay:

Because students add and drop classes for the first few weeks of the semester, it is important to not form student groups too early, as group turnover can hurt transactive memory (Moreland and Argote 2003). Thus, the semester will begin with core concepts and common texts the instructor feels everybody should know (Lévy’s shared knowledge). The course’s first midterm would be a traditional, non-collaborative exam, comprising a certain percentage of the grade. This diagnostic test can allow the instructors to gauge student’s relative abilities when putting together teams. Members should be assigned, with the goal of diversifying skills and knowledge.  Allowing students to self-select would almost certainly increase group homogeneity bound as membership would be to existing friendship ties, i.e. those who shared the same interests and activities outside the classroom.

After this exam, though, the group becomes important. At this point, the reading list will grow so that it is too much for any individual to read. Students will have to become mutually dependent to survive.

Because transactive memory forms around face-to-face communication, we must provide students a time to meet. Large lecture classes often have smaller discussion sections. Normally a teaching assistant facilitates a discussion and/or clarifies confusing points from lecture. However, this hour might be better spent allowing team members to meet to discuss the readings they divided amongst themselves and relate them to the common lecture themes of the week. The teaching assistant would also stress problem solving and coordination skills, helping groups refine strategies and learn from their mistakes, more like a coach than like our traditional model of a teacher. Although the ideal team would be self-regulating and self-guiding, the teaching assistant could also help them to assign roles or divide labor, if needed, to insure that each member pulls their own weight.

The teaching assistant can provide one question each week that would be representative of those on an exam, allowing group members to synthesize their different readings and learn how the other students think–their strategies for identifying the core stakes of a problem, mobilizing knowledge, testing data, assessing conclusions, and communicating results. These questions could require students to do online searches, tap into knowledge from other classes, or draw on their extracurricular expertise. Through these test runs, students would learn each other’s specialization, build trust, and coordinate their efforts on tasks similar to the group exam.

As they enter a collaborative test-taking process, students face the challenge of resolving conflict and committing to a shared answer, especially working under time constraints. Outside of the classroom, affinity groups develop norms, such as those surrounding contributions to Wikipedia, to which they can appeal to resolve such conflicts. So, for example, Wikipedia articles strive towards neutrality, which is often achieved through inclusion (that is, featuring all competing perspectives) rather than exclusion (arriving at a consensus response) (Lih 2009). Student’s experience of testing may be that there is a right answer the teacher is expecting and thus, they may be less receptive to test taking strategies which include a broader range of possible answers. Having multiple collaborative activities will allow each group to develop its own norms and protocols for resolving disputes and finding an answer students feel they can stand behind.

The final exam is designed to tap a range of different kinds of expertise. Think of the individual problems as possessing the sense of “meaningful ambiguity” which, McGonigal (2008, 214) argues, motivates the problem solving activity around alternate reality games: “by asking players to cooperate to make meaning out of an ambiguous system, the game-based hive mind celebrates individual perspective even as it embraces the larger, intricate intelligence that emerges only at the scale digital networks afford.” For such experiences to be compelling and satisfying, McGonigal (2003) argues, they have to introduce problems that seem within reach of the network of players. She notes that an empowered team often seeks to move beyond the game and tackle real world social problems, only to be disappointed that such problems may not, in fact, be resolvable given the group’s resources and capacities. Exam questions would need to be open ended enough to allow many different paths to a solution and yet ultimately something that participants can comprehend and resolve.

Assuming that the lecture meets twice a week, the questions are given out at the beginning of the first lecture and are due at the end of the second lecture. This will allow students to use the first lecture to start working on the answers and divide the labor. Then students can go home and seek more information on their own, and work more on the answers if they so choose. They coordinate efforts so that each student plays to her strengths and so that there is a robust system of checks and balances to identify and eradicate misinformation. Preparing for the exam may be much more like getting ready for a guild raid in World of Warcraft than like studying for a traditional test. The second lecture can be used to finalize answers. Also, by spanning two lectures affords groups at least two face-to-face opportunities to interact. Students can write their answers using an online tool, such as Google Documents, that allows them to write simultaneously in a single document. In this way, they can make changes to each other’s work (knowing who wrote what) and see changes being made to their own work. Changes can be tracked over time and reverted back to if needed….

Educational researcher Dan Hickey and his research team at Indiana University (Hickey, Honeyford, and McWilliams forthcoming) has been trying to explore what forms assessment needs to take within a participatory learning culture and concludes that assessment should “focus on reflections rather than artifacts.” His group has developed a range of activities that might follow a project or exam, asking students to reflect on what strategies they tried and why, rather than simply evaluating them based on what they produced. Of course, students will have different capacities to articulate their reflections. McGonigal (2008, 222) has similarly argued that working in large-scale teams to solve alternate reality games encourages “meta-level reflection on the skills and processes that players use to meet new challenges.” At the end of the day, the test might function as much as a probe to encourage students to continue to think about the process of their learning than as a simple assessment of what they, collectively and individually, know.

 

Well, this semester, we are going to be putting these ideas into action, as I teach my lecture hall subject a second time. I will  be sharing my syllabus next time.

I have ended up dividing the class into two parts: for the first part, students will be developing shared knowledge, that is, knowledge which will be required of every member on the team, and they will be performing as individuals, demonstrating their own mastery over the materials. In the second part, they will be sorted into teams which will work together on all future assignments: the discussion section times will be opportunities for the students to work on problem sets together with coaching from the Teaching Assistants, and the final week of class will be given over to a culminating activity which will require teams to work together to respond to the prompts.The midterm is approaching and soon we will be making the cognitive shift from individual to collective effort.

From the start, the class has emphasize new ways of learning in a networked culture, drawing heavily in the first few weeks on materials produced by MacArthur’s Digital Media and Learning initiatives. I want to get them to take an inventory of their own skills and competencies as learners, the ways they use new media in the context of their lives, and to engage critically with the debates surrounding the so-called “digital natives” and their new media literacy skills.
For this to really work, I am having to abandon the lecture as the primary mode of presentation in the class. Instead, I am moving towards something closer to the way Socratic Method works in Law Schools. On most days, we are reading essays which represent conflicting perspectives on core debates around digital media and culture, hoping to foster critical thinking and research skills.

As I developed this approach, I struggled with the issue of “freeloaders” — that is, students who are willing to let the others do all the work and coast to a better grade. There’s a limit to what I can do in terms of evaluating individual performance if we are going to really place such a strong emphasis on group performance, but I will be monitoring and evaluating individual attendance and participation in the discussion session and exam, and I will be asking participants to list everyone who contributed to a particular project (which can, in fact, include people who are not in their assigned group, as long as their participation is fully disclosed.)

There’s a lot we still have to work through, so I would welcome feedback from readers about this approach and I would be especially interested to hear from anyone who has tried something like this process before. I promise to report back on how the class is going and share some of the problem sets we create later in the semester.

Comments

  1. Nikki Draper says:

    I haven’t gone through this specific process before, but depending on how the collective work is defined, you may find that it’s important to the students to find ways to recognize individuals within the group. In my experience, the challenge with freeloaders is not catching and punishing them; but how frustrating that situation can be for the other students in the team. It can be counter productive to the learning experience if students spend a lot of time being angry that X isn’t doing what s/he should and why isn’t some way to take this into account? Mechanisms for allowing individuals to be accountable in the group (or to the group) seem to help the students relax and embrace the task. I think it helps them trust that there will be fair treatment by the grader and that you recognize the practical problems they face in group work. (Caveat: I work in an educational system where grades are king.)

    Not knowing the nature of the collective work you want them to do, I guess I have to start by saying I’m not sure I buy that the grading of the collective work should put most or a lot of emphasis on the group’s performance. It seems to me that there are ways to think about how the individual members in the network function, how different individuals share and develop knowledge and the manner in which the choices they make and their optimization of the network pieces they have influence the group’s performance. We can tease out individual contributions with other complex collaborative efforts, so why not this, too? (otherwise, the Academy Awards would just be for Best Picture and that’s it! Everybody go home–which may arguably be a desirable outcome, I know.)

    Team-based learning might provide you with some interesting options. I sat in on a talk recently by the Dean of the Duke-NUS (Singapore) medical school and they use this approach. I thought it was interesting because they provided ways to track individual performance (regular tests that are problems the students need to come up with solutions for) and then for the students to work in teams to problem-solve the more difficult/controversial questions. And then those teams debate their choices with the other teams in the class. Instructors were there to answer questions and sometimes ask them, but they didn’t lecture, and no Socratic method either. The key to this, according to the Dean, was that the medical faculty sat down and worked together to write good, challenging problems which took a lot of work and that the students then needed to really commit to a solution (i.e. pick an answer).

    One thing I would highly recommend, create an evaluation / survey of the material in the course, teaching methods, the learning experience, the assignments and have the students complete it. I borrowed this from a colleague when I was revamping, revising course material and it’s extremely useful in terms of helping me see what the students found meaningful in the coursework, what wasn’t resonating, etc. I don’t always change things based on the feedback, but often I get extremely helpful and strategic feedback from it.

    Good luck with this. It’s an extremely interesting challenge. I’m very curious to see how it works and what you and the students learn from it all.

  2. Adam Kahn says:

    I’m Henry’s collaborator mentioned in the blog post.

    Nikki, this was a concern we had as well and had to think through carefully. The framework we developed drew on humanistic theories of collective intelligence and social scientific theories of transactive memory systems (and assuming we find that transactive memory actually develops in the groups, Henry may have his first publication with an inferential statistic :) ). A necessary condition for transactive memory is that group members are cognitively interdependent, and one way to develop cognitive interdependence is through the reward structure (in this case, grades). There is a body of social psychology and organizational behavior research on how to reduce freeloading, and we will be drawing on this as well when convening the testing groups.

  3. Andrew H says:

    I think you’ve come up with a very practical solution to a widespread problem. During my days as an undergrad at the University of Texas I saw the same study “networks” and they were out of control. Let us know how your strategy works out.