In the last post, the ideas around reflection as a key part of the SoTL cycle were discussed. In this post, we look deeper at how to determine whether the approach used to address the identified problem, challenge, or opportunity had impact. That is, how can we evidence the impact that is (or is not) being had? For the purpose of this post on evidencing impact, the approach taken to address the problem, challenge or opportunity will be referred to as the intervention.
A critical aspect of our SoTL practice is investigating our intervention in order to determine whether it had an impact and what that impact was. This means that we need to collect appropriate evidence (i.e. data) that will speak to the impact. It might be worthwhile revisiting the understandings of evidence and impact in relation to SoTL.
What are evidence and impact?
In SoTL, as in research, evidence is the data that we can collect to support our claims of impact and evaluation of the SoTL intervention. However, a difference is that the evidence typically focuses on students, their learning, and our teaching. Therefore, the evaluations investigate the experiences of and changes in students and their learning in relation to the implemented approach. The SoTL focus on student learning is often examined through asking students about their experiences or through examining their interactions, artefacts (e.g. assessment items), and other objects or observable behaviours (Kelly, 2008).
When thinking about impact, it is useful to return to the original SoTL problem, challenge, or opportunity, and the research question that was developed for this. Impact is broadly the difference your intervention has made in relation to both the research question and the stakeholders involved, e.g. students, academics (IMPEL, 2016). While the impact occurs both during and after the SoTL project, it is important to be able to identify it clearly so that you can talk about it with others and disseminate your work. (Remember: Dissemination is a key part of SoTL and distinguishes SoTL from scholarly teaching.)
What can I do to evidence my impact?
Much like any research project, the evidence that you collect needs to be fit-for-purpose. That is, it needs to be able to talk to the impact that your intervention is having and thus it must align. The research questions, their objectives, indicators of impact, and data collected must be well-aligned in order to evidence the impact. Without this alignment, it will be difficult to demonstrate the impact that you are having and can see in your students and learning and teaching.
To start with, you need to clearly identify the research question, objective(s) of the intervention, indicators of when you will know that the objectives are being met, and what could evidence achievement of that objective(s). That is, what is your intervention aiming to achieve? What will be an indicator of having achieved that? What will be able to evidence that indicator? When you conceptualised your SoTL project, what were the key research questions and how were you going to know if they were achieved? What were the objectives? What indicators did you have in mind? It may be helpful to use a table to visualise this alignment. The table below is an example of how this might appear.
Table 1. Example of alignment between research question, objectives, indicators, and data collection.
Now that you have the alignment clearly visualised, you need to draw out the impact that the intervention is having during the SoTL project implementation and beyond. This will support the demonstration of the value of your intervention, so it is important to clearly articulate the impact, based on the evidence that has been gathered, and to present it in a concise way. One way to do this is by using the Impact Management Planning and Evaluation Ladder (IMPEL) model.
The IMPEL model can be used to identify different levels of impact that can result from SoTL projects (Department of Education & Training, 2016). The model starts by identifying key stakeholders from those closest to the project and then moves outward to students, the university, and then more broadly. The IMPEL table supports the clear identification of the impact at various times as well, starting from the current point in time, project end, six months post-project completion, 12 months post-project completion, and 24 months post-project completion.
Impact vs Outcomes
Now is a good time to check in about the differences between impact and outcomes. These terms are often confused or conflated, so clarity is important, particularly when disseminating your findings and impact to others. As above, impact refers to the difference that is being made and it focuses on doing things whereas outcomes are typically things (nouns) that have been achieved or developed (OALT, 2019). For example, an impact might be increased student engagement but an outcome could be a research paper written about the increased student engagement. It is important to keep these two areas clear and distinct, even though they can (and should!) talk to each other.
Connection to your SoTL project
It's time to bring this back to your SoTL project and how you can connect what you are observing with your intervention and how you can ensure that you have robust evidence to support it. Thinking about your original SoTL problem, challenge, or opportunity, and considering where it is at now, what are your objectives and what are the indicators that will evidence the achievement of these objectives? What data will support the evidencing of these indicators? Take time to draw your own table similar to that above with your research question, objectives, indicators, and data collection ideas. Do they align?
Now thinking about the impact that the intervention is having on you and your students. Start an IMPEL table with three columns: the first has the key stakeholders, the second has the current point in time, and the third has project completion. Can you fill in any of the table cells with ideas of what the current impact is? Remember that impact is actions (doing), not things (outcomes).
Further reading
The following two articles provide different ideas and methods for evaluating SoTL and impact. Which ones align with your SoTL project? Are there any parallels or learnings for you?
Alkema, A. (2011). A Tertiary Practitioner's Guide to Collecting Evidence of Learner Benefit: Ako Aoteroa National Centre for Tertiary Teaching Excellence. https://ako.ac.nz/knowledge-centre/collecting-evidence-of-learner-benefit/a-tertiary-practitioners-guide-to-collecting-evidence-of-learner-benefit/
The Learning Technology Dissemination Initiative (2000). Evaluation cookbook. http://www.icbl.hw.ac.uk/ltdi/cookbook/cookbook.pdf
The following two articles provide insights into SoTL from two different areas: flipped classrooms and academic writing. Skim through the articles and identify what they used as indicators of impact and the data that they used to support these indicators. Do these align? Are there any parallels or learnings for you and your SoTL project?
Lawson, A., Davis, C., & Son, J. (2019). Not All Flipped Classes are the Same. Journal of the Scholarship of Teaching and Learning, 19(5). https://doi.org/10.14434/josotl.v19i5.25856
Sybing, R. (2019). Making Connections: Student-Teacher Rapport in Higher Education Classrooms. Journal of the Scholarship of Teaching and Learning, 19(5). https://doi.org/10.14434/josotl.v19i5.26578
References
Department of Education and Training. (2016). The Impact Management and Planning Evaluation Ladder. https://docs.education.gov.au/documents/impact-management-planning-and-evaluation-ladder-impel
Kelly, D. (2008). Evaluating teaching and learning: enhancing the scholarship of teaching by asking students what they are learning. In Murray, R. (Ed.), Scholarship of Teaching and Learning in Higher Education (pp. 80-90). Online: McGraw-Hill Education.
OALT (2019). Applying for Academic Promotions Workbook: The Learning & Teaching Component. University of Southern Queensland: Toowoomba.