Experimentation and virtual worlds: why worry?

In the spirit of backlog* (which is the real theme of my blog), I've been meaning to write about a couple of concerns I had after reading Edward Castranova and Matthew Falk's  article in the October issue of Games and Culture entitled Virtual Worlds: Petri Dishes, Rat Mazes, and Supercolliders. Now let me start by saying that Professor Castranova has the unenviable task of representing a relatively controversial research ideology within the context of the emerging field of virtual worlds research. There are a lot of methodologically qualitative/ epistemologically constructivist researchers in this field, and Castranova does generally quantitative work operating from a more or less Post-Positivist perspective. For the non-researchers among you, this means that he uses numbers a lot and that (as near as I can figure) he proceeds from an assumption that truth claims can be proven through measurement provided the tools are good enough.

Motivational Poster borrowed from xdr.com - In case you didn't notice I'm a big fan of these posters. This one has only a tenuous connection to this blog post.

Before I go any further I should probably qualify my own perspective on this matter of truth in research. I definitely approach things from a constructivist epistemology. Without getting into the muck too much, this means that I think different truth claims have various levels of utility in different contexts. In other words, reality is messy, we should be very careful of oversimplifying it, and measurement is not always a viable means for describing the world around us. To be clear, while I am primarily a qualitative researcher, I don't fear numbers and I definitely think they are often highly serviceable tools for describing the world. They just have their limitations like everything else.

Alright, with that out of the way let me get to the heart of this thing. I won't summarize the entire article here (in fact I suggest you go read it before you read the rest of this post), but in order to explain my concerns I do need to offer a quick explanation of the portion of this article I found problematic. Castranova and Falk offer the following possible arguments which other virtual world researchers might present against the idea of using virtual worlds as experimental spaces:

1. Virtual worlds do not produce generalizible results.
2. Virtual worlds (or any society worth studying) are too complex to be controlled.

Castranova and Falk generally do a fine job (at least to my eye) of rebutting these issues which they theorize as being the core concerns of other researchers. However, these are the sorts of concerns, endemic enough in academia, which are  largely tied to what professor Clif Conrad refers to as issues of methodological correctness.These concerns certainly have their place, but they're far more important to other academics than they are at the sort of policy level which Castranova aspires to inform through virtual world studies. In other words, I think there's a much deeper issue here, and that issue isn't about the sort of intellectual one-upmanship that researchers so love to engage in with each other around issues of method and scientific rigor.

The problems for me with the idea of using MMOs as experimental spaces are instead ethical and practical in nature. My first concern can be summed up with the phrase, "Just because you can doesn't mean you should." This is a phrase I often use with my parent's over size standard poodle Bela who is tall enough to put his head on tables and eat food off of them if no one is paying particuarly good attention.

The dog in this image is not intended as a derogatory reference to any of the researchers discussed here. It is simply a visual illustration of the first caveat.

Of course, Bela is a dog and has no actual way of understanding what the heck I'm trying to tell him when I say this, except perhaps the part about getting his head off of the table. However, virtual worlds researchers are another story. While the idea that we can manipulate variables and change the experiences of players in a virtual world is deeply appealing, it also has the potential to be deeply unethical. The short version is this: If these spaces are real enough to offer us effective answers for vexing questions, then we must live up to extremely high ethical standards in the conduct of research in these spaces just as we would in meat space. Players in them devote hours and hours of time and energy making them higher stakes spaces than they might appear to be to the casual observer. Even if these spaces can tell us powerful things about human behavior in conflict situations, we really don't have the right to engineer these spaces to answer many of the difficult questions about human behavior any more than Philip Zimbardo did when he created a simulated prison with college students.

My second concern is born of the policy studies which I've been engaged with peripherally since I began studying school leadership. Policy making is a funny game, and policy makers do not have the best track record at being critical consumers of research. Pragmatically, feeding the call for empirical evidence based research as a basis for policy making by using MMOs can be extremely dangerous. In terms of research as evidence for the policy context, we cannot make the mistake of thinking that experiments through MMOs would be anything less than a powerful tool. We unfortunately have a long record in politics around the misuse of tools, and if an MMO based experiment can produce data powerful enough to be useful, it is also powerful enough to do a lot of damage. What exactly am I saying here? Simply that virtual worlds can help us produce seemingly iron clad cases to a layman who doesn't know how to read statistics, and as far as I've been able to discern politicians and many other policy makers are either no good at reading statistics, or prefer to use them as rhetorical bludgeons rather than evaluate them with any kind of scientific rigor.

These are my concerns. I have no idea if they're actually shared by other games or virtual worlds researchers, but I do know that they are not the concerns highlighted by Castranova and Falk. With all of this said however, I want to acknowledge that Castranova studies economic issues, and that these issues are (imho) less likely to be ethically or practically problematic in the ways I've attempted to highlight. In fact, Castranova's recent work makes a powerful case for how these worlds can very effectively inform economic research in meaningful ways. However, I dwell much more in conflict resolution studies and other "softer" regions of human behavior. As near as I can tell, the sorts of variables one would need to tweak to better understand conflict behaviors through virtual worlds research are exactly the ones which send up a big red flag when I consider ensuring the well being of my research subjects. Furthermore, I'm pretty sure that virtual worlds can tell us a whole lot about these behaviors without researchers running their own experimental games. Personally I'd say all you've really got to do to put some of our most unsavory behaviors under the microscope is join a raiding guild full of adolescents or adults who act like them ;).

* Because this is a backlogged post and I meant to get it out weeks ago, I didn't necessarily edit it as carefully as I would've liked to. Just let me know if you see factual (or grammatical) issues and I will correct them.