Measuring New Media Literacies: Towards the Development of a Comprehensive Assessment Tool (Part One)

Last fall, I spread a message to my Twitter followers, asking for their participation in an online survey we were conducting, trying to assess new media literacies skills. Needless to say, people who follow this blog and my Twitter account are apt to have a higher degree of technical and cultural literacy than the general population, but we were looking for a sample base large enough to be able to test and refine our instruments before applying them to other populations, such as the students at the schools where we are doing after-school programs or which are adopting some of our curricular recommendations. Given the intense response we received, and our deep gratitude for everyone who participated in the survey, I wanted to make sure we shared the results with you in a timely fashion. Ioana Literat, a PhD candidate in the Annenberg School of Communications and Journalism at the University of Southern California, did the work as part of a class project in Lynn Miller's class, COMM 550: Research Methods in Communication. She also is part of the Project New Media Literacies research team and we are immediately putting her tool and her insights to work by pre-testing students entering our programs here in Los Angeles.

Her results are interesting in that they do suggest that the skills we have identified through the White Paper I helped to write for the MacArthur Foundation do cohere in real world contexts and that these skills improve through engaging with new media platforms and practices. I should stress here that we believe that the relations between increased skills and increased use of new media tools does not simply mean that the people who consume more media get better at it.

As writers like James Paul Gee have argued, these "affinity spaces" contain powerful forms of informal learning which motivate and support the acquisitions of these skills in a way that would not be true for most people watching films and television outside of the context of a fan community, which might perform similar work for its members. Further, we are not simply describing consumption per se, but rather we are talking about forms of participation which involve applying those skills rather than simply observing. At its heart, then, the argument is that participatory culture communities and practices actively support the learning of their members and reversely, that as we first asserted certain skills have emerged as characteristic of and often necessary for meaningful involvement in participatory culture. Today, we are going to lay out the methods behind this research; next time, the findings.

Measuring New Media Literacies:

Towards the Development of a Comprehensive Assessment Tool

by Ioana Literat

The present study was motivated by our observation that, in spite of the increasing popularity and impact of Henry Jenkins' New Media Literacies framework, there was a lack of an appropriate quantitative measurement tool to assess these new media literacy skills. Certainly, existing tools do not capture the full spectrum of skills and propensities suggested by Jenkins. Furthermore, the reliance on qualitative data - which is typical of most studies in this field - means that such assessment projects are not feasibly replicable with larger groups. Therefore, this study aimed to address methodological lacunae within the NML framework by developing and validating a comprehensive quantitative assessment tool that could be used to measure new media literacies (NMLs) in both adult and juvenile populations.

Below, you will find an overview of the survey instrument and a summary of the results. If you would like to see the complete NML questionnaire that was used for this assessment, as well as the full report on the findings of this study (including all the statistical data), we encourage you to contact Ioana Literat at iliterat@usc.edu.

In assessing the psychometric properties of this new assessment tool, survey data was first factor analyzed in order to assess the reliability of the measure, and determine how these emergent factors compared with Jenkins' original 12 NML skills. If the survey instrument was accurately constructed, we expected to see 12 separate subscales - similar to the 12 NMLs identified by Jenkins - resulting from the factor analysis. In terms of the relationship between media exposure and NMLs, we hypothesized that higher levels of new media literacies would correlate with a higher degree of engagement with media forms - particularly new digital media - and that there would therefore be a significant difference in NMLs between people with low versus high levels of media exposure. An increased degree of digital participation in various Web 2.0 platforms should also relate to high NML levels, with light users scoring lower in media literacy than heavy users of these digital platforms. Finally, we also hypothesized that high NML levels should predict a greater propensity for multimedia creation, and, respectively, civic engagement.

The sample for this study (N=327) was a convenience sample of normal volunteers over the age of 18, who completed the survey online. In terms of gender distribution, the sample contained 131 male respondents and 187 female respondents. The average age was 33.7 years (SD=11.7). In regards to ethnicity, 83.9% of respondents were white, and 77.3% indicated English as their primary language. Income and education levels were normally distributed.

Survey Design

The survey was structured around 4 main sections: demographics, media use habits, new media literacies (NMLs), and civic engagement. All questions were randomized, so that each participant received them in a different order, to maximize the validity of the findings.

The section on media use habits queried respondents about their access to a computer and to the Internet, the extent of their exposure to different media forms, their digital memberships and affiliations, and their creative engagement with multimedia. The NML section of the survey - the most extensive and critical part of this instrument - aimed to assess respondents' new media literacy skills (NMLs) by presenting them with a randomized series of 60 statements about their personality, social and cultural modes of engagement, online and offline peer interaction, learning styles, and media consumption and creation patterns. The statements were conceptually built around the 12 NML skills identified by Jenkins (2006): play, performance, simulation, appropriation, multitasking, distributed cognition, collective intelligence, judgment, transmedia navigation, networking, negotiation, and visualization. To ensure an adequate factor analysis while minimizing the duration of the survey, we decided to include 5 items for each NML, for a total of 60 questions. These items addressed both technology-related and non-technology-related behaviors, in accordance with our view that the NML skills are social and cultural competencies that stretch beyond media expertise or technological capability.

Finally, the last section of the questionnaire contained a set of 5 questions that attempted to measure the respondents' degree of civic engagement, by addressing three principal dimensions of civic engagement: self-efficacy, civic responsibility, and commitment to civic action.

Ioana Literat is a PhD student at USC's Annenberg School of Communication and a research assistant for New Media Literacies. She has taught digital storytelling and social justice curricula to children in the Dominican Republic, Romania, Uruguay and India, and spent the last year working as the field coordinator of The Modern Story program in India. At USC, Ioana is researching the social impact of media and its potential to stimulate positive change, with a special focus on the future of educational media and virtual communities. As a result of her extensive international experience, she is particularly interested in the global scalability of NML projects, and the applicability of such educational initiatives in the developing world.