5.1 Evaluating Your Program

Introduction

We're building our program, we've identified best practices and keys to success, we've formed our alliances, and things seem to be going well. But how do we quantify "going well"? How can we show that our OER program is successful? Inevitably, our bosses want to see proof that their investment (be it your time or their money) is worth it. This subunit will help us identify and evaluate impact measurement strategies for systematically tracking and reporting OER program data that are relevant to your own efforts.

While being able to self-evaluate is important, please don’t get lost here. What we'll be presenting are useful references to be aware of, but they're not the backbone of your program — relationships and communication are. Instead, use measurement strategies and tools as a way to frame conversations and your argument base. 

Required Readings and Resources

This subunit's readings will take you through the process of identifying impact measurement tools, seeing examples of those tools in action at other universities, and finally figuring out how to market those results, again with examples from other institutions. Not every linked resource needs to be read in full; some are more for skimming/exploring. 

• Read these four articles discussing OER impact:
• SPARC Impact Story, “Reducing Textbook Cost to $0: Tidewater Community College Links to an external site." (CC BY 4.0 Links to an external site.)
• David Wiley’s “Thinking About Impact Links to an external site.” (CC BY 4.0 Links to an external site.)
• OpenOregon's "Is the average cost of a textbook $100? Links to an external site." (CC BY 4.0 Links to an external site.)
• Open Textbook Network's "OTN Data Collection and Reporting Plan Links to an external site."
  
• Zooming out, here are two great resources to review a list of impact measurement tools (skim these, getting ideas for what strategies are out there to measure impact):
• OER Hub Links to an external site.
• Open Education Group Links to an external site.
• Start to think how you might develop a marketing campaign for promoting open education that aligns with your institutional culture and university mission. See the Community College Consortium for Open Educational Resources (CCCOER)'s "12 Proven Strategies for Marketing OERs Links to an external site." (CC BY 4.0 Links to an external site.).
• As you think about implementing and maintaining your open initiative, think about sustainability measures as well. How will you ensure that your initiative maintains momentum? See this robust array of resources on OER sustainability (Links to an external site.) from SUNY.

Calculating Impact

How can you calculate your impact? The Open Education Group's COUP Framework? Links to an external site.Do any of the examples highlighted in the Open Education Group's literature review Links to an external site. intrigue you? This is a great resource to pull data from during presentations to faculty and administration about the effectiveness and value of OER. Lumen Learning's OER Adoption Impact Calculator Links to an external site. allows you to customize the calculator to measure a variety of elements related to OER adoptions. 

Calculating student savings on textbooks is one of the cornerstone metrics of any OER program — and the greater OER community for that matter. In SPARC's "A New Method for Estimating OER Savings," Links to an external site. it explains how SPARC launched a community-wide effort to track $1 billion in worldwide savings through the use of OER. At the root of this calculation is a question: How much do students save per course when OER is adopted in place of traditional textbooks?

According to SPARC, that number is $116.94. OpenStax uses a much more conservative figure of $79.37 Links to an external site.. OTN settles on an even $100 to split the difference and make calculations easier. Some OER programs don't use a set number at all, and will instead calculate savings on a course-by-course basis. For example, CHEM100 students might be saving $200 per course, while ANTH100 student may only be saving $30.

Moreover, many programs report student savings via other means besides just OER. As mentioned previously in this course, OER efforts are often folded into larger affordability initiatives. As such, student savings don't need to be a result of only OER. Faculty use of library-licensed ebooks and other content that the school pays for, but not the students, are often reported in same breath as OER. 

The choice is yours to select whichever method works best for you. 

The resources above discuss calculating student savings, but how might you calculate return on investment (ROI)? What about calculating the impact on teaching and learning? These latter two are trickier, and could certainly use more research — perhaps from you!

Communicating Impact

Lastly, let's talk about communicating your impact. Here are a few examples:

• Textbook Heroes @ UTA Links to an external site. (CC BY NC 4.0 Links to an external site.)
• Library Stories @ NCSU Links to an external site. (Note that not all these projects are OER-related. The idea here is more about communicating impact. A couple of good OER-related examples worth looking at specifically are “Don’t Throw Away That Assignment Links to an external site.” and “When the Textbook You Need Hasn’t Been Written Yet Links to an external site..")

What usually captures your attention during presentations — data or stories? What do you remember after two months have passed? What inspires you to take action? These answers won't be the same for everyone. But in general, try to think of ways you can combine data and stories. 

If you only take away one thing from this unit, let it be this: This is hard stuff. No one has all the answers. A big part of open education work is open values. Be human. Embrace not knowing. Engage in open dialogue with others — explain why you're doing what you're doing now and why you believe that to be the best path forward, but always be open to learning, growing, and changing course. 

Next

Now that you've learned about evaluating open education programs, go on to test your understanding with a self test by selecting the Next button below.