Treatment Research: An Education with Katie Massey

There is a lot of research that has been provided to parents and interested parties about wilderness therapy over the last 10 + years. I wanted to talk with you about the scope and depth of wilderness therapy and residential treatment research, as it evolves. Katie, you were the Research Director at Evoke Therapy Programs for the last several years and since 2012, you have also been Co-Chair of the OBH Council Research Committee. Now you are returning to graduate school for a Ph.D. 

Before we jump into some of the current research, which is impressive, I wanted to talk to you, the person who has done and is generating some of the doing research publications and helping to compile and broadcast the findings for the Outdoor Behavioral Healthcare Council, I would like you to spell out some of the categorical differences in research and types of research.

It seems like there is a variety of research conducted in the field – some that is conducted by a third party, such as a university, and some that is fully conducted within the program itself.

That is correct. In general you could say that there are two forms: research that is in conjunction with a university, and research that is “internal” to the program. When the research is conducted with a university or outside researcher, the data is collected by the program, but then the data is “de-identified” so that researchers can analyze the data on an aggregate level without giving any personal or identifying information to anyone outside of the program that a client attended.

The other kind of research we see is a program or ownership group doing its own research and compiling data. I like to think of these more as “internal program evaluations”. There is a lot of value to well-designed self-evaluations.  Hopefully the program is using it to inform their program. It gets a little grey though, as often we’ve seen these types of projects as fodder solely for marketing!

During my PhD, I will be working with all of the above (research with colleagues at universities, and Evoke/Second Nature’s internal evaluations). My graduate work allows me to also return to a project with children in out-of-home care and issues around women’s health. It’s an exciting world of opportunity!

What does a reader who is not masterful with statistics need to look for when assessing the validity of a type of research?

Great question! Typically, research that is connected to a third party (university or researcher who is outside of the program), is a good indicator of objective information. That doesn’t mean that those “internal evaluation” projects are not worth looking at; they are invaluable to our field.

I believe some things to look for are whether the work has been presented at a professional conference or published in a peer-reviewed journal; publishing and presenting are steps that show a level of validity, as they have passed through some level of checks and reviews. What is written on a website is not necessarily a great source!

What does it take for a wilderness therapy or residential program to get research going?  I am imagining — time, money and personnel . 

More than anything, it takes leaders with an understanding that research and evaluation are essential. Then, that inevitably leads to money! In the beginning, it takes time to train staff on how to implement the study and how to collect the data. Once the initial learning curve is over, it is relatively smooth sailing.

One thing we have learned in the OBH Council Research Committee is that programs who have a designated person in charge of the research do the best: when a program tries to split it up seven different ways and there isn’t one consistent person overseeing everything – success will be spotty!

Membership in OBHCouncil requires participation in one industry-wide efficacy study.  Isn’t this difficult for a smaller program?

It certainly can be expensive. That said, the OBH Council and NATSAP have created an incredibly cost-effective way to do research and evaluation with the Practice Research Network, or the NATSAP Study.

All programs should be doing some kind of evaluation of their work, whatever mechanism or structure it is through. We owe that to our families and clients. There are low-cost ways to do this.

How many wilderness programs have a dedicated staff/researcher?

There are many programs who have staff who fulfill the research roles in addition to other roles and responsibilities, and many do a great job of it. Very few programs have one staff dedicated only to research. Evoke Therapy Programs is truly a leader in this way, and their vision to invest in this is really impressive to me. There are certainly multiple programs, particularly within the OBH Council, who are really leading the way and investing in research and evaluation.

[Evoke Wilderness was or perhaps still is your employer, correct? Under full disclosure, I want to put that out there.]

It is exciting that other wilderness therapy programs have people dedicated to research as well — Red Cliff, Open Sky, Elements Wilderness, and several other programs in the OBH Council (who are not advertisers on All Kinds of Therapy).

With the independent research that is out there, what are the most useful ways programs can educate the public, regarding valid and significant findings in wilderness therapy & private pay behavioral health?

Conferences are a big way to get the word out, and being published in a peer-reviewed journal is the ultimate goal for researchers. I think programs use their websites and other marketing material mostly though. Unfortunately, the things that really validate in the scholarly world, are not necessarily very accessible, or of interest, to those outside of that “scholar-world”. 

In what national conferences has Evoke or other OBH programs presented? 

Evoke and OBH programs have presented at the conferences within the field that you may expect, such as AEE, NATSAP, and the Wilderness Therapy Symposium. Over the last couple of years though, we have been at conferences outside of our niche field as well. This is very exciting! For example, at Evoke, we have presented at the American Psychological Association (APA) for the last two years, and will be presenting the Council for Social Work Research this year as well. Usually these presentations are in collaboration with OBH researchers and/or other OBH programs.

This is a really big deal to us and to the field of OBH. Wilderness therapy historically has been seen as on the fringe in the larger world of mental health. Acceptance into these broader conferences shows that research is credible, intriguing and working to convince a larger field of psychologists.this idea is changing. 

This is super exciting and shining a positive light and credibility on Wilderness Therapy, which has struggled in the past to show the good work that is and has been happening.  It is impressive how well the collaboration OBH has created between Wilderness Therapy Programs.

Let’s talk a little about the wilderness therapy research. I have to tell you my jaw dropped when I read that the

“Outdoor behavioral healthcare programs average injury rates are 1.12 per 1,000 participant days compared with:

• Backpacking 2.05
• Downhill Skiing 3.28
• Football Practice 19.74

from a four year study.”  What has been some of your favorite tidbits or surprising trends or validation of the work you knew as a program person was always happening? 

Yeah, those numbers are pretty incredible! The risk management research really is the strongest arm of the OBH research and the results from over 10 years of data are very impressive. They are also reliable numbers, which is the most impressive thing to me! 

As far as surprising trends, I am really fascinated by the consistent finding of adolescent girls reporting greater change than boys during OBH treatment. Boys largely outnumber girls as participants in OBH, but it seems like girls actually respond faster (at least according to their own self-reports). We have theories of why this may be the case, but we don’t really know. I personally just love how this finding is contrary to social norms.

A paper that was recently accepted in the Child and Youth Care Forum, looked at parent reports of (adolescents’) change, and we did not see this gender trend. So it seems like this trend may either be stronger or only present in adolescent perceptions of themselves. Ultimately we just need deeper investigations to figure out what’s going on here.

What do you think wilderness therapy or private pay residential treatment needs to investigate in research going forward?

We need to answer:

1. For whom do this treatment work best,
2. HOW does it work?
3. Do treatment effects last?

We can see that there are big changes in clients, but data is fairly limited on predictors of change and longitudinal outcomes. We also really need a control or comparison group, but this is much harder!

A control or comparison group would be a group of very similar adolescents/young adults who do not receive OBH treatment. Then you would compare outcomes between the two groups (with OBH and without OBH). This is how you can definitively say that participation in an OBH treatment program caused this change/positive outcome.

Is there anything else that you want to say about research that professionals or families need to know about when they are reading/digesting Outcome Studies or efficacy statements?

Keep reading and asking questions! Remember that the field is still fairly young as is our research. So, a new finding in one study isn’t necessarily conclusive, it’s interesting and something to pursue further.

For more research on Wilderness Therapy, go to Outdoor Behavioral Healthcare Council.

Stayed tuned for independent research findings into the specifics of Residential Treatment at All Kinds of Therapy, as well.