Showing our Work: Evaluating LA Makerspace Library Programs with CS OPEN

Contributed by Rebecca Herr Stephenson

Over the past 3-and-a-half years, LA Makerspace has helped 7,200 kids and teens across Los Angeles County learn science and technology skills through Maker workshops in their local public libraries. In addition to reaching youth, we trained 160 librarians in Maker Education, an approach to facilitating learning that emphasizes interest-driven, project-based learning, equipping them with the tools to offer Maker workshops on their own within their library branches.

Our busy workshop and training schedule, combined with our ongoing efforts to build and improve our offerings, means that we often find ourselves in the position of “building the plane while flying it.” On the one hand, this kind of trial-and-error approach is characteristic of Maker culture. On the other, it means that even though we have pretty good anecdotal evidence that our method works, we didn’t have good solid data to prove our hunches and back up our stories.

Fortunately for us, all of this changed when we were accepted into the second cohort of CS OPEN. With the support we received from the CS OPEN program, we were able to design a comprehensive evaluation, create tools for measuring learning and development, and create a plan for rolling out the evaluation in stages based on our capacity and our data needs.

What do librarians know about Making?

The first piece of evaluation we did was to survey librarians who have previously participated in our trainings to ask about their knowledge of maker culture and confidence in their ability to teach maker workshops in their libraries. You can see their responses in the table below:

Librarian Maker knowledge table

We were very excited to see that almost all (96.3%) of the librarians know what a Makerspace is. We have a unique definition of Makerspaces that is an essential part of our philosophy, so seeing that most participants left the workshops with a solid understanding of this concept is encouraging. A second important part of our philosophy is promoting the “Maker Mindset,” a way of thinking that emphasizes creative problem solving. Most participants (88.9%) agreed or strongly agreed that they can identify ways of thinking that reflect this type of mindset. In written comments on the survey, Librarians again demonstrated their understanding and embrace of the Maker Mindset:

[What I found most valuable about the training was] the learning process of trial and error. The goal is not about perfection or getting it right the first time, it's about tinkering, experimenting and thinking outside the box to help you improve on your ideas.

Making robots myself with easily accessible materials demystified the process for me. I would not have been able to facilitate a robotics workshop at the library without having built (and failed) and again built my models first.

In both of these comments, the librarians emphasized trial and error, tinkering, and resilience--all essential parts of the Maker Education philosophy we teach—another success! 

How can libraries participate in Maker culture?

LA Makerspace is part of a growing group of people and organizations who recognize that the contemporary library is much more than a place to find information and borrow books. Libraries are also a place for connecting, building community, and making stuff. Our evaluation data show that the librarians who participated in our trainings are on the same page about the purpose of their libraries. Nearly all of the librarians who responded to our evaluation survey (96.3%) stated that they understand why libraries are interested in supporting Makerspaces and Maker workshops. Data from later in the survey indicate that these librarians are highly supportive of offering Maker workshops at their library branches. Moreover, a majority (81.4%) of librarians anticipate that patrons at their branch would be excited to participate in Maker workshops, and 100% of librarians note that library Maker workshops give patrons access to unique resources.

Librarian attitudes toward workshops

What Did We Learn?

Our first foray into systematic evaluation was not without fits and starts and stumbling blocks. Through the year-long process of developing and piloting our evaluation, we learned some important lessons:

First, evaluation takes time. Some research is quick-and-dirty, done without a lot of advanced planning or development. Prior to our participation in CS OPEN, this was the only kind of evaluation we used. Through CS OPEN, however, we learned the importance and value of having a well-designed and thoughtful approach to evaluation. We also learned that this takes WAY longer than you’d ever anticipate!

Second, support is essential. The assistance and guidance we received from CS OPEN provided structure and inspiration, even when challenges arose or our motivation lagged. We learned not only from the CS OPEN leaders, but from our peers in other nonprofit organizations within the cohort. Perhaps more importantly, we were encouraged and challenged by the community to do our best work in service of our constituents.

Finally, evaluation is never done. We’ll be launching a new phase of evaluation, including some new tools and methods, this spring. We’re excited to see what information those efforts will yield.

Rebecca Herr Stephenson, Ph.D., is an LA Makerspace Board member and Education Committee Chair. She is a researcher focused on teaching and learning with popular culture and technology. Becky is co-author of two books, Teaching Harry Potter: The Power of Imagination in Multicultural Classrooms (2011, Palgrave MacMillan) and Hanging Out, Messing Around, and Geeking Out: Kids Living and Learning with New Media (2009, MIT Press).

L.A. Makerspace is a 501(c)(3) nonprofit and a fiscally sponsored project of Community Partners. Our mission is to ensure that all Los Angeles children and youth have equal opportunity for hands-on STEM (Science, Technology, Engineering and Math) skills education.