Why Basic Research Methodology Is Important To Magical Knowledge

Quick note–a couple of days after I wrote this but before it was scheduled to go live, I was interviewed regarding Otherkin on the Pagan Musings podcast with KaliSara and RevKess, as well as guest “Arthur”. It was a really good discussion; I jumped in about 45 minutes into the show as the special “surprise” guest. Take a listen if you’re interested; we get into what basics of what Otherkin are, but also some of the spiritual/religious and psychological elements of the phenomenon as well.

So. On to the intended post.

Recently on Livejournal I wrote a response to a post someone else wrote about proposed experiments to try to “prove” the objective existence of Otherkin. These experiments ranged from Kirlian photography to try to get pictures of phantom limbs, to using EEG to measure any neurological abnormalities in Otherkin compared to the general population. I feel it applies not only to proving Otherkin as something other than collective imagination, but also proving the objective existence of magic. Here’s what I wrote (with a couple of minor edits and some helpful links added):

With regards to experiments, most of the proposed quantitative experiments over time have been horribly flawed and have not been designed with solid research methodology. Here are a few particular potential flaws:

–Poor research design: A good piece of research starts with good design. What is the experiment meant to measure? How is it measured? Is it using any existing instruments, or is one created specifically for the purpose of that experiment? Is the instrument you’re using reliable–does it measure consistently? Is it valid–does it measure what you actually are trying to measure? Finally, the simpler, the better, especially in new territory such as this. Keep it to one independent variable and one dependent variable, if possible–and know which is which.
Confirmation bias: This is a BIG problem with anecdotal “evidence” of Otherkin, magic, etc. Confimation bias basically means seeing what you want to see, and excluding anything that doesn’t support your desired results. This is often done unconsciously. Example: I keep seeing signs that Tiger is my totem. I want Tiger to be my totem, so I give greater attention and value to things that support Tiger being my totem than not, even though, if the evidence is taken by the numbers, the evidence points toward Tiger not being my totem.
Sampling bias: This was a notable reason for why my surveys for the Field Guide were NOT formal research, and a big potential issue with trying to do any experimentation with Otherkin in general. Your sample is most likely going to be biased toward people who A) are willing to be identified in some manner as Otherkin and are not so paranoid as to assume even anonymous research may be used against them personally, and B) more often than not WANT for Otherkin/magic/etc. to be proven. It’s a small population to begin with, too, so you’re most likely going to have a small sample, which can heavily affect whether the research is even solid.
Confounds and Correlation vs. Causation: related to some of the earlier things I talked about, confounding variables are variables other than the identified dependent and independent variables that come into play and affect the results. Another, very closely related concept is “correlation does not equal causation”. Just because two variables seem to affect each other in one’s results does not mean that they automatically are causal to each other. There may be a confound or third variable that is the actual vehicle of causation, or the correlation may be coincidence. This is why multiple experiments need to be run, and the results thoroughly analyzed, before making any theoretical conclusions.
–Applying more significance to results than the statistics show: Statistics are how you analyse your results in various and sundry ways. They allow for a certain level of variation (such as standard deviations from the mean, or identifying outliers) and the statement thereof, and they also help you to rule out whether your results occurred by chance or not (whether your results are statistically significant or not). Through statistics you can use the hard data to determine whether or not you proved your hypothesis (or disproved the null hypothesis).

Because most “evidence” of Otherkin/magic/etc. is anecdotal, and experiments “proving” it often manipulate or inflate the significance of the results, and the best research so far has not supported the objective existence of magic and other spiritual things, any research done to try to “prove” Otherkin/magic/etc. on an objective level needs to be of the highest quality and avoid the above and other pitfalls.

I added one last postscript to my initial response:

(Or, tl;dr – a small handful of people who say “This happens when we do that” does not constitute proper research methodology and does not hold water when trying to prove anything objectively.)

Observing “Well, every time I do this, this happens” is fine if all you want to do is self-confirm a subjective experience. But if you’re trying to prove that magic really works as an independent, objective force (rather than your results being from your own psychological biases, or other external factors that are not “magic”), then you need more rigorous testing then just a handful of people doing the same spell, ritual, or meditation once or twice and comparing their results over coffee. Just because you claim you can replicate your results doesn’t mean that you can prove that your independent variable and your dependent variable are causative as well as correlated. Are you constructing your experiments with a large enough sample to make a statistical difference? Are you doing your best to rule out confounds and confirmation bias? Would your results hold up to heavy statistical analysis?

Every shoddily constructed experiment and instrument, every poorly interpreted or deliberately manipulated set of results, every anecdote held up as firm “evidence” across the board–all these things do absolutely nothing to further your cause, and in fact do much to harm it. This is one example of what happens when people push bad research into the general consciousness. (And before you say “Well, bad magical research never killed anybody!”, here’s a sizable collection of recorded instances of people being injured or killed by the misapplication of everything from faith healing to dream interpretation (and, apparently, also GPS systems.)

And before anyone tries to start a science vs. magic debate, or argue that there’s no such thing as objective reality, both derailments of which are going to get killed before they get on their feet because I do control the comments here*–my point that I am making is that if you are going to claim that magic can be proven through experimentation, then your methodology needs to not be half-assed. If you are going to claim that you have any authority on anything that involves proving something exists objectively, then you need to be literate in the methods used in proving something exists objectively. Finally, understanding the basics of research methodology is an incredibly valuable part of critical thinking skills, skills that are woefully under-represented in magic and spirituality, and really are a necessary part of being human.**

That last paragraph that I just wrote right up there? THAT’S the intended take-away. You want to prove magic (or any other similar force or concept) exists in an objective, consistently measurable manner? Then have the correct tools, and be willing to be wrong, if that’s where the evidence and statistics end up taking your research.

* I’m not avoiding them because I don’t think they’re good topics of debate, but I want to keep things focused on the actual topic I’m discussing here, rather than getting derailed. Thank you for respecting that.

** Even people who have never, and will never, run a formal experiment still benefit from knowing the basics of research methodology so that they can have a better idea of what the people who do those experiments tell the general public through their published results (and why that’s important to everyday life). Yes, people who are experts in their field and have access to knowledge and training the rest of us don’t do have an advantage and authority. But knowing the basic processes by which they acquire their knowledge, to include research methodology, can help those of us on the general level of “consumer” of information and products to have a better understanding of why, for example, “studies show Brand X is the best!” or parse out whether a news story on “This food/medication/material COULD KILL YOU” is worth paying attention to.

About these ads

11 thoughts on “Why Basic Research Methodology Is Important To Magical Knowledge

  1. excellent post. i’d even go so far as to say that i’m not sure why people would want to push magic into the realm of mainstream objective reality, or what they expect to get out of it. haven’t they seen how people associated with magical/occult/witchy/shamanic associations tend to be treated in the cultures and nations that DO recognize magic as as part of the objective reality? they’re almost inevitably taken as a threat–hell, i’ve seen and experienced minor (by comparison) cases of that here and now. i know plenty of people that would absolutely believe me if i said i was doing the work, and those are the last people i’d come out and admit it to because they would not react well. at all.

    there are far worse things than being treated with skepticism. these folks should count their blessings.

    • *nods* I’ve often thought it’s better to be seen as a fool than a threat, though there is the threat of applying truly irrational thinking to crucial issues (like not vaccinating your kids because you believe that vaccinations cause autism).

  2. I would be extremely interested in such a project, as I know others would be. I’m a newbie so it would be fascinating to collect development data. I have a Masters so that gives me academic cred, too. Thanks for this valuable post.

  3. Recently you keep popping up everywhere I look. I found the podcast in an otherkin group I recently joined. Then I found your books, which afterwords I found out I had been on your website unknowingly until mentioned in the podcast… and while looking over wolf skins to dance in or use during rituals i found your blog, and didn’t realize it was you until I selected the option of therianthropy on the left column. It’s getting very odd >.>

  4. An excellent essay on all sorts of levels.

    Even people who have never, and will never, run a formal experiment still benefit from knowing the basics of research methodology so that they can have a better idea of what the people who do those experiments tell the general public through their published results (and why that’s important to everyday life).

    This is an argument I’ve used before when handing people What Educated Citizens Should Know About Statistics and Probability. Just because you don’t do formal experimentation for a living doesn’t mean you can’t benefit from knowing about how it operates, especially given the prevalence of these things in our society.

    I am just bracing myself for the election season, where I will no doubt hear about how Candidate A is “surging” past Candidate B when the numbers are within a margin of error or people propping up the one poll that happens to agree with them -.-

    • One of the most eye-opening things about my grad school career was taking the statistics and research methods courses required; I wish more people would at least familiarize themselves with the basics thereof.

  5. It is a rare treat to hear this kind of wisdom spoken in the Pagan community. I agree 100%.

    I edit a community blog for naturalistic spirituality, devoted to a marriage of science and myth. I think this would make an excellent article. Would you be at all interested in republishing it at http://www.HumanisticPaganism.com? If so, email me at humanisticpaganism [at] gmail.com. Thanks!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s