Measuring New Media Literacies: Towards the Development of a Comprehensive Assessment Tool (Part Two)

Measuring New Media Literacies: Towards the Development of A Comprehensive Assessment Tool (Part Two) by Ioana Literat

Results

Although all of our scale items collectively attempt to measure new media literacy levels, and the overall reliability of the scale was high (Chronbach's α=.903), we were interested in identifying the specific subcomponents that make up this concept. Our initial research question was whether the subscales of this survey instrument map well onto Jenkins' 12 NMLs. Particularly, we were interested in seeing if, as predicted, the scale would break down into components that were similar to those identified by Jenkins.

To address this question, we performed a factor analysis on the 60 items, and then assessed the reliability of each separate subscale that emerged from the factor analysis. With the exception of 2 NMLs (collective intelligence and simulation), the factors identified in this analysis mapped well onto Jenkins' 12 NML skills, indicating the definite existence of subcomponents that tap into dichotomous skill sets. Thus, out of the 12 NML skills that make up Jenkins' framework, 10 were identified in the factor analysis of our scale; furthermore, all 10 of these components had adequate reliability. This is a rather impressive and encouraging finding, especially given the fact that all 60 items of the scale were completely randomized and thus the items that made up each of these 12 subscales never appeared in order. The two NMLs that did not distinctly emerged from the factor analysis were collective intelligence and simulation; rather than clustering together as distinct factor components, the items measuring these two dimensions ended up being spread out over the different subscales.

Once the factor analysis revealed the various new media literacy skills that the scale constituted of, we proceeded to explore the relationship between these NMLs and patterns of media exposure and digital participation, by running multivariate analyses of variance (MANOVAs).

We first looked at respondents' cumulative media exposure, which included time spent with all forms of media: Internet, television, print media, and videogames. According to our second hypothesis, we expected to see a significant difference in NML skills between high and low media users. The multivariate difference in media literacy levels assessed using MANOVA was indeed significant: F(10, 316)= 3.025, p=.001, with avid media consumers scoring higher across all NML skills than less enthusiastic media consumers. The univariate differences between the high and low media exposure groups were particularly pronounced in the areas of negotiation, networking, appropriation, play, multitasking, and transmedia navigation.

Next, we explored the relationship between NMLs and exposure to specific media. In terms of Internet use, there was a significant difference between low and high users: F(10, 316)= 3.171, p=.001, with the most striking contrast occurring in terms of networking skills. Due to the interconnecting and socializing features of the Internet, less enthusiastic internet users scored much lower in networking skills than frequent users. For videogames, the difference between frequent and infrequent users was also significant (F(1, 316)=2.811, p=.002), with avid gamers scoring substantially higher than their peers in the domain of play, or experimental problem-solving.

Our questionnaire addressed users' exposure to four different forms of media: two new ones (internet and videogames) and two old ones (television and print media). Interestingly enough, while the difference in NML skills between light and heavy users of the Internet and videogames - i.e. new media - was substantial, this difference was not significant in the case of traditional media. This is an interesting conclusion, which supports the view that new digital media, due to their interactive and highly socializing nature, are more adept at breeding the social and cultural competencies needed for a full participation in today's digital environment than traditional media, which are inherently more passive.

In terms of digital participation, we hypothesized that higher levels of media literacy should predict a higher degree of engagement with Web 2.0 platforms, as well as an increased propensity for multimedia creation. This hypothesis was fully supported: the difference in NMLs between users with high digital participation levels versus those with lower participation levels was indeed significant (F(10, 316)=3.172, p=.001). Out of the digital platforms we explored in this study, the ones that emerged as particularly significant in this analysis were Facebook (F(10, 316)=5.294, p<.001), Twitter (F(10, 316)=3.181, p=.001), YouTube (F(10, 316)=4.553, p<.001), and blogging (F(10, 316)=4.747, p<.001).

For Facebook, the difference between light and heavy users was especially pronounced in the area of networking, with enthusiastic Facebook users displaying extremely high networking skills. This result is unsurprising, given the function of Facebook as a social networking site, but this connection is important in regards to the applicability of such online-learned skills in the context of one's offline behavior.

In the case of Twitter, the two main NMLs where light and heavy users significantly differed were networking and transmedia navigation. We found that light Twitter users (including non-users) scored much lower in these 2 NMLs than more enthusiastic tweeters. This conclusion makes sense, and can be explained by the hyperlinked and social nature of the Twitter platform.

YouTube also emerged as an extremely significant platform in terms of NML skills.

The NMLs that YouTube users excelled at were appropriation and transmedia navigation, but also, to a less astounding degree, performance and negotiation. These results are most likely explained by the primary functions of the YouTube platform as a crucial depository of popular culture clips (to be used in appropriation processes) and as a source of multimedia information (encouraging transmedia navigation), but also a democratic limelight for stardom and personal opinion (performance) and a transnational hub that facilitates intercultural learning (negotiation).

Finally, blogging emerged as another particularly important platform in terms of NML skills. We found a significant difference in overall NML skills between bloggers and non-bloggers, and individuals who keep a blog scored much higher in appropriation and networking skills. Most likely, this is due to the increasingly interlinked nature of the "blogosphere", with writers linking to other blogs of interest, keeping a blogroll on their personal page, republishing relevant posts, and so on. This process of hyperlinked interconnectedness, while gradually transforming the personalized "blogosphere" into one global community, increasingly requires networking and appropriation skills that allow one to most effectively tap into this informal community.

The results of this study also supported the connection between multimedia creation and NMLs. As hypothesized, higher NML levels predicted a propensity for multimedia creation, and the difference between frequent and infrequent digital creators was extremely significant (F(10, 315)=6.635, p<.001), with the most acute contrast occurring, not surprisingly, in the area of appropriation. This is in line with the literature in the field, which claims that the ability to creatively produce and distribute multimedia texts should correlate strongly with higher levels of media literacy.

Similarly, the results also confirm the connection between new media literacies and civic engagement, which is emerging as a critical application of NML educational initiatives. Our hypothesis regarding the positive relation between media literacy and civic engagement was fully supported, with respondents that scored highly across the NMLs showing much higher degrees of civic engagement than their less media literate peers (F(10, 313)=3.516, p<.001).

In conclusion, as evidenced by the support for our main conceptual hypothesis, the data gathered in this study will be instrumental in perfecting a validated quantitative assessment tool to complement NML initiatives built around this particular framework. So far, educational endeavors aimed at cultivating these skills only benefitted from qualitative evaluation tools, which are inherently unfit for use with large samples, and are much harder to implement due to logistical considerations. We therefore hope that this questionnaire, especially used as a baseline measure of new media literacies, will help provide a more accurate and comprehensive picture of individuals' abilities in this domain.

Furthermore, the study provided critical information about the connections between new media literacies, media exposure, and engagement with different Web 2.0 platforms; this represented a much-needed addition to the literature on media education, which so far did not address these specific correlations. In terms of the validity of the present assessment tool, the fact that our hypotheses regarding the connection between media literacy and media use habits were strongly supported lends additional predictive validity to this survey instrument. This is a highly significant conclusion that adds further import to the current study. While the causal relationships between these variables would need to be examined longitudinally, over time, it is our interpretation that the relationship between media use and media literacy is a circular one, involving a virtuous feedback loop: for instance, while extensive use of the internet raises one's new media literacy levels, individuals with high NML levels are also more likely to access the internet considerably more.

While further research is certainly needed regarding the feasibility and scalability of quantitative methods of assessment in the field of new media literacies, we believe our study is a valuable starting point in this direction, and a much-needed inquiry into the challenges facing such assessments in both national and international contexts. While this particular study represented a pre-test of the validity of the current survey instrument, we are now working on its practical application as a baseline measure of NML levels at the Robert F. Kennedy Community Schools, where Project New Media Literacies will be implementing an after-school program starting in February. Stay posted for updates regarding this initiative, and an upcoming report on the quantitative assessment of new media literacies among the high school students at RFK!

Ioana Literat is a PhD student at USC's Annenberg School of Communication and a research assistant for New Media Literacies. She has taught digital storytelling and social justice curricula to children in the Dominican Republic, Romania, Uruguay and India, and spent the last year working as the field coordinator of The Modern Story program in India. At USC, Ioana is researching the social impact of media and its potential to stimulate positive change, with a special focus on the future of educational media and virtual communities. As a result of her extensive international experience, she is particularly interested in the global scalability of NML projects, and the applicability of such educational initiatives in the developing world.