trisha poole 31/1/20 trisha poole 31/1/20

How do I know whether my SoTL-based intervention worked?

In this post, we look deeper at how to determine whether the approach used to address the identified problem, challenge, or opportunity had impact. That is, how can we evidence the impact that is (or is not) being had? A critical aspect of our SoTL practice is investigating our intervention in order to determine whether it had an impact and what that impact was. This means that we need to collect appropriate evidence (i.e. data) that will speak to the impact. It might be worthwhile revisiting the understandings of evidence and impact in relation to SoTL…

In the last post, the ideas around reflection as a key part of the SoTL cycle were discussed. In this post, we look deeper at how to determine whether the approach used to address the identified problem, challenge, or opportunity had impact. That is, how can we evidence the impact that is (or is not) being had? For the purpose of this post on evidencing impact, the approach taken to address the problem, challenge or opportunity will be referred to as the intervention.

A critical aspect of our SoTL practice is investigating our intervention in order to determine whether it had an impact and what that impact was. This means that we need to collect appropriate evidence (i.e. data) that will speak to the impact. It might be worthwhile revisiting the understandings of evidence and impact in relation to SoTL.

What are evidence and impact?

In SoTL, as in research, evidence is the data that we can collect to support our claims of impact and evaluation of the SoTL intervention. However, a difference is that the evidence typically focuses on students, their learning, and our teaching. Therefore, the evaluations investigate the experiences of and changes in students and their learning in relation to the implemented approach. The SoTL focus on student learning is often examined through asking students about their experiences or through examining their interactions, artefacts (e.g. assessment items), and other objects or observable behaviours (Kelly, 2008).

When thinking about impact, it is useful to return to the original SoTL problem, challenge, or opportunity, and the research question that was developed for this. Impact is broadly the difference your intervention has made in relation to both the research question and the stakeholders involved, e.g. students, academics (IMPEL, 2016). While the impact occurs both during and after the SoTL project, it is important to be able to identify it clearly so that you can talk about it with others and disseminate your work. (Remember: Dissemination is a key part of SoTL and distinguishes SoTL from scholarly teaching.)

What can I do to evidence my impact?

Much like any research project, the evidence that you collect needs to be fit-for-purpose. That is, it needs to be able to talk to the impact that your intervention is having and thus it must align. The research questions, their objectives, indicators of impact, and data collected must be well-aligned in order to evidence the impact. Without this alignment, it will be difficult to demonstrate the impact that you are having and can see in your students and learning and teaching.

To start with, you need to clearly identify the research question, objective(s) of the intervention, indicators of when you will know that the objectives are being met, and what could evidence achievement of that objective(s). That is, what is your intervention aiming to achieve? What will be an indicator of having achieved that? What will be able to evidence that indicator? When you conceptualised your SoTL project, what were the key research questions and how were you going to know if they were achieved? What were the objectives? What indicators did you have in mind? It may be helpful to use a table to visualise this alignment. The table below is an example of how this might appear.

Table 1. Example of alignment between research question, objectives, indicators, and data collection.

Now that you have the alignment clearly visualised, you need to draw out the impact that the intervention is having during the SoTL project implementation and beyond. This will support the demonstration of the value of your intervention, so it is important to clearly articulate the impact, based on the evidence that has been gathered, and to present it in a concise way. One way to do this is by using the Impact Management Planning and Evaluation Ladder (IMPEL) model.

The IMPEL model can be used to identify different levels of impact that can result from SoTL projects (Department of Education & Training, 2016). The model starts by identifying key stakeholders from those closest to the project and then moves outward to students, the university, and then more broadly. The IMPEL table supports the clear identification of the impact at various times as well, starting from the current point in time, project end, six months post-project completion, 12 months post-project completion, and 24 months post-project completion.

Impact vs Outcomes

Now is a good time to check in about the differences between impact and outcomes. These terms are often confused or conflated, so clarity is important, particularly when disseminating your findings and impact to others. As above, impact refers to the difference that is being made and it focuses on doing things whereas outcomes are typically things (nouns) that have been achieved or developed (OALT, 2019). For example, an impact might be increased student engagement but an outcome could be a research paper written about the increased student engagement. It is important to keep these two areas clear and distinct, even though they can (and should!) talk to each other.

Connection to your SoTL project

It's time to bring this back to your SoTL project and how you can connect what you are observing with your intervention and how you can ensure that you have robust evidence to support it. Thinking about your original SoTL problem, challenge, or opportunity, and considering where it is at now, what are your objectives and what are the indicators that will evidence the achievement of these objectives? What data will support the evidencing of these indicators? Take time to draw your own table similar to that above with your research question, objectives, indicators, and data collection ideas. Do they align?

Now thinking about the impact that the intervention is having on you and your students. Start an IMPEL table with three columns: the first has the key stakeholders, the second has the current point in time, and the third has project completion. Can you fill in any of the table cells with ideas of what the current impact is? Remember that impact is actions (doing), not things (outcomes).

References

Department of Education and Training. (2016). The Impact Management and Planning Evaluation Ladder. https://docs.education.gov.au/documents/impact-management-planning-and-evaluation-ladder-impel

Kelly, D. (2008). Evaluating teaching and learning: enhancing the scholarship of teaching by asking students what they are learning. In Murray, R. (Ed.), Scholarship of Teaching and Learning in Higher Education (pp. 80-90). Online: McGraw-Hill Education.

OALT (2019). Applying for Academic Promotions Workbook: The Learning & Teaching Component. University of Southern Queensland: Toowoomba.

trisha poole 3/12/19 trisha poole 3/12/19

How do I reflect in and on my SoTL practice?

In the previous post, we looked at how research and SoTL were similar and simultaneously different. As part of that, the SoTL cycle was presented in a way that shows the SoTL process, even though it is not always as linear and neat as the graphic depicted (shown below for reference). Within this cycle, there is one component that is always present: reflection. Reflection is an important part of being a SoTL scholar as SoTL is focused on your practice…

reflection-in-action, reflection-on-action.jpg

In the previous post, we looked at how research and SoTL were similar yet different. As part of that, the SoTL cycle was presented in a way that shows the SoTL process, even though it is not always as linear and neat as the graphic depicted (shown below for reference). However, within this cycle, there is one component that is always present: reflection. Reflection is an important part of being a SoTL scholar as SoTL is focused on our own practice and ways to improve it.

Figure 1. SoTL cycle depiction (but note that it is not always as linear and organised as appears!).

What is reflection in SoTL?

Firstly, let's begin by defining reflection in general, and then we'll look at what it means for our SoTL practice. Reflection can be simply defined as the process of considering something deeply that we might not have thought much about otherwise. In education, we often refer to critical reflection and critically reflective practice and practitioners. Dewey (1910, p. 6) wrote that reflective thought involves the "active, persistent, and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it". This indicates that reflection - reflective thought and subsequently reflective practice - requires a questioning mindset by considering why things are the way they are and how they might be in the future. Taking this further, Brookfield (2017, p. 3) defined critical reflection as "the sustained and intentional process of identifying and checking the accuracy and validity of our teaching assumptions". Bringing this back to our SoTL process, reflection is the continual process of interrogating our assumptions about our SoTL problem, the evidence that we are seeing in our practice that supports or contradicts our assumptions, and considering the possibilities for future practice.

What models of reflection can be used to support our critically reflective practice?

Perhaps one of the most effective ways to understand reflection and critically reflective practice is to look at some models that can be used to support this and examples of these models in practice. Each of the models presented below is an example of a method that can be used to support critical reflection, and there is no single "correct" way to reflect. You might find that it is a combination of these models that supports your reflection best. As we go through the models, they'll become progressively more complex.

Model 1: What? So what? Now what?

This is perhaps the simplest model and it gets to the heart of critical reflection quickly. It was developed by Rolfe et al. (2001) to support nurses to reflect in their practice. In essence, it is the following:

What? What happened? Describe the event in objective detail. Note that the event can be positive or negative - it just needs to be one that is in your mind for further consideration.
So what? What does this mean to you? Why was this event important to reflect on? Describe the importance of the event and why this event has stayed with you.
Now what? What will happen now or in the future? What is the connection to your future practice? This can include connections to the literature or ideas about looking into the literature on related topics or events.

For example, if you have had a negative online teaching experience, you may critically reflect on that. Here is a real-life example that I have experienced and subsequently reflected on.

What? A student in my postgraduate course had not engaged with the content required in order to fully develop their second assignment and only engaged with the feedback from their first assignment three days prior the second assignment being due. This resulted in a panicked message from them late at night, which I responded to the next day. The next engagement from the student was the submission of their second assignment, but the submission was incomplete and there were significant elements of self-plagiarism (even though this had been part of the extensive feedback on the first assignment).
So what? This is an indicator that my feedback is not being read in a timely manner. Perhaps it is the student's fault for not engaging, but this time it has meant that they (significantly) failed the second assignment, and subsequently the course. This is a problem for the student and also for me because I have an unhappy student who believes that they should have passed. Perhaps I need to review the scaffolding for the second assignment to see if it needs to be stronger, and I could place more emphasis on engaging with the feedback on the first assignment. How can I better support students like this? But if it was only one student in the course, how much is my responsibility and how much is theirs?
Now what? I will review my processes for feedback and how I can track which students have engaged with the feedback. Perhaps I should be sending nudges to those who have not engaged with the feedback at one week post-assignment return. An explicit discussion on academic integrity (including self-plagiarism) might also need to be incorporated into the course so that this type of incident does not arise again. I will also review the second assignment scaffolding and ask a colleague to review it. I will also look more into the literature on assessment design and scaffolding the skills needed to successfully complete the assessment item.

Watch the following video for a very brief (2:44 min) overview of this model.

Model 2: Brookfield's four lenses for reflection

Brookfield is a great proponent of critically reflective practice and critical thinking. “The critically reflective process happens when teachers discover and examine their assumptions by viewing their practice through four distinct, though interconnecting lenses’ (Brookfield, 2005, p xiii). He developed a model of reflective practice that focused on viewing events through four lenses:

our own autobiography as teachers and learners,
our students' eyes,
our colleagues’ experiences, and
the critical literature.

Each of these lenses provides a different perspective on the event, and this way we can better understand what happened and how we can move forward with it. The lenses are briefly described here, with links to individual videos for each lens from a higher education perspective.

Autobiographical lens (1:58): This is often referred to the as the "Self lens" and it focuses on our experiences as a teacher in order to reveal aspects of our pedagogy that may need adjustment or strengthening. Teaching philosophies and portfolios can be examples of this lens.
Learner's lens (2:13): This is our students' eyes - their perspectives. Engaging with their view of the learning environment can lead to more responsive teaching. Evaluations (formal and informal), assessments, journals, focus groups, and/or interviews can each provide cues to improve teaching and learning.
Peer lens (1:40): This is our colleagues' experiences and perceptions - both our informal and formal conversations that we engage with and seek out when trying to understand an event. Our colleagues can help highlight hidden habits in our teaching practice and also provide different approaches to teaching problems. Also, colleagues can be inspirational and provide support and solidarity. Peer observation is an example of this lens.
Theoretical lens (1:34): This refers to what the literature says about our problem or opportunity. That is, we seek perspectives and experiences from the literature to inform our understandings. Teaching theory provides the vocabulary for teaching practice and offers different ways to view and understand our teaching.

These lenses can be helpful in better understanding our context and problem/opportunity, and this can be a multi-faceted approach to more critically reflecting on our teaching and learning.

Model 3: 4Rs of reflection

The 4Rs model of reflection presented by Bain et al. (2002) focuses on four actions that we can take as part of our critically reflective practice in teaching and learning. These actions are report/respond, relate, reason, and reconstruct. Note that some refer to this as the 5Rs model because they separate the first R into two actions (report and then respond).

Report/Respond: A descriptive account of the situation, incident, or issue, and an emotional or personal response to the situation, incident, or issue
Relate: Drawing a relationship between current personal or theoretical understandings and the situation, incident or issue
Reason: An exploration, interrogation or explanation of the situation, incident or issue
Reconstruct: Drawing a conclusion and developing a future action plan based upon a reasoned understanding of the situation, incident or issue

A brief overview of these 4Rs is in the infographic below.

If you'd like to view a video to explain this further, then watch this short explanation (3:35) by Associate Professor Rachael Hains-Wesson. An example of this model in practice can be found here and also here.

Other commonly used models

Of course, there are other models of reflection, and they are most effective when you find the one that suits you and your way of thinking. While you can search for models online, here are some that are commonly used in higher education:

Questions for you

Have you seen or heard of these models before? If so, have you used any of them?
Which model speaks to you most?
Which model do you think you will use in your project? Will all of your team members use the same model?

Connection to your L&T project

Let's connect this back to your L&T project. As you have seen through this post and the SoTL content on what it is and how it differs to research, reflection is an important part of the SoTL cycle. Even to be in the position that you are in now in your project indicates that you have reflected on a challenge or opportunity that you face in your teaching. Here are some questions that you can consider among your team.

How will your team reflect on the L&T project?
Will you focus on individual reflections or team reflections?
As reflections are part of the reporting process, do you have a plan about how to manage this in terms what the focus of the reflection will be and how you will present a cohesive reflection?

References

Bain, J.D., Ballantyne, R., Mills, C., & Lester, N.C. (2002). Reflecting on practice: Student Teachers' perspectives. Flaxton, Qld, Australia: Post Pressed.
Brookfield, S.D. (2017). Becoming a critically reflective teacher (2nd ed.). Jossey-Bass: San Francisco, CA.
Dewey, J. (1910) How we think. D.C. Health & Co.: Boston, MA.
Rolfe, G., Freshwater, D., Jasper, M. (2001) Critical reflection in nursing and the helping professions: a user’s guide. Basingstoke: Palgrave Macmillan.

trisha poole 29/10/19 trisha poole 29/10/19

How do I know if I'm doing research or SoTL?

Building on the previous post introducing the scholarship of teaching and learning (SoTL), this post looks into the similarities and differences between SoTL and research. To recap, SoTL focuses on how we, as academics and teachers, use research to support our learning and teaching endeavours and examine our practice with the view to furthering the domain knowledge. It sounds simple, doesn't it? Just like research! But it is different, so let's explore…

What are the key similarities and differences between SoTL and research?

If we understand research to be a structured approach to informing a response to a question, then SoTL refines this further and narrows the bounds. SoTL is a structured approach to informing response/s to question/s about students and their education (including achievement, retention, transitions, and success); using these responses to make improvements; and sharing process and outcome knowledge with other educators (OALT, 2019). Through these two definitions, it is easy to see that SoTL is a more focused approach to research and includes specific elements such as reflection and feedback. The following table (OALT, 2019, p. 31) provides a brief comparison of the elements of research and SoTL.

OALT 2019 p 31 - SoTL vs Research table.png

One element listed in this table that often comes up for discussion is the reduction or buy-out of teaching because academics often want to reduce their teaching load so that they can focus on their research. Due to the nature of SoTL, i.e. that it focuses on one's teaching practice, it is not logical to reduce or buy-out teaching. Therefore, a key difference is the ability to use research funding to buy-out/reduce teaching and to "shift" some of the research work onto a research assistant. However, this is not possible in SoTL: in SoTL, it is expected that the academic/teacher will be conducting this research on their teaching in their context using their experiences and knowledge. This is not practical if the academic/teacher is not teaching.

So, how do we "do" SoTL?

Conducting SoTL has a broad yet consistent approach, regardless of the context. The process is similar to that of research, but also slightly different. Typically, an academic/teacher identifies a student problem, opportunity, or challenge (often simply referred to as the "problem"), and then uses the literature to support an approach to addressing this. That is, the proposed approach to the problem is evidence-based and grounded in research. Part of determining the approach to the problem is also deciding on the appropriate types of evidence that will be collected in order to demonstrate effectiveness and impact. Because a critical aspect of SoTL is dissemination, ethics approval is typically required before collecting the evidence, so this goes into the cycle as well. While implementing the approach, the evidence is collected and analysed. There can be iterative design approaches used where the approach is iterated and refined through a number of teaching cycles. After the approach is implemented, it is evaluated, and the results (positive or negative) are disseminated. Then the process can start again. Throughout the process, there is reflection and refinements. These parts all come together to build a picture and tell the story of how we are improving the student learning journey through our active, engaged learning and teaching practice. The following image is an attempt at describing the SoTL process, but it is typically not as linear and smooth as the image makes it appear.

Remember that the SoTL cycle depicted here is the smooth, flowing version, not necessarily the reality. SoTL can be an exciting process in which we can become caught in a loop of iteration and refinement because we always want to improve what we are doing. At times, this iteration and refinement loop may seem never-ending, but we need to disseminate so that we can receive feedback from peers and then continue to refine and iterate our approach. Keep in mind that our teaching contexts rarely stay constant, so our approach also needs some flexibility and adaptability.

How does this connect to your GELTS?

Through your GELTS project, you should be working through this SoTL cycle. In the application phase, you have already identified your student problem, the related literature, and your proposed approach. Now, you are working on implementing your proposed approach, which should also include ethics approval to collect the data that you are using to evidence your approach.

With that in mind, consider the following questions and let's start a discussion about your GELTS project in response to this thread.

In brief, what is your SoTL project?
How is it similar to or different from your typical discipline research focus?
How have you engaged with the SoTL cycle so far?
Have you considered reflection activities as part of your SoTL project?

Note that if you want to share a SoTL focus that you have outside of your GELTS, then feel free to do that, too.

Delving further

The field of SoTL is ever-expanding, so a quick Google Scholar or Library search will result in different articles in different disciplines for further reading and consideration. I have included two articles below if you would like to start with these. The first article provides an overview of SoTL from Hutchings' perspective (which is the introduction to his edited book on SoTL, Opening Lines), and the second article is a write up of a SoTL project in the science discipline. (I will endeavour to include a SoTL article from different disciplines as we go through the GELTS Coffee Course.)

Hutchings, P. (2000). Approaching the scholarship of teaching and learning. In: P. Hutchings (Ed.) Opening Lines: Approaches to the Scholarship of Teaching and Learning, (pp. 1-10) Princeton, NJ: Carnegie Foundation for the Advancement of Teaching.

Rowland, S. L. and Myatt, P. M. (2014). Getting started in the scholarship of teaching and learning: A “how to” guide for science academics. Biochem. Mol. Biol. Educ., 42,6-14. doi: 10.1002/bmb.20748

References

Office for the Advancement of Learning & Teaching [OALT] (2019). Applying for Academic Promotions Workbook: The Learning & Teaching Component. University of Southern Queensland: Toowoomba.

trisha poole 25/9/19 trisha poole 25/9/19

What is the scholarship of teaching and learning (SoTL)?

The scholarship of teaching and learning (SoTL) is becoming a more widely known and used term in higher education. It focuses on how we, as academics and teachers, use research to support our learning and teaching endeavours and research our own practice with the view to furthering the domain knowledge. When we use research and evidence to support our practice, we are undertaking scholarly teaching, and when we share that with our colleagues, we are embarking on the scholarship of teaching and learning. To understand the difference between scholarly teaching and SoTL…

How does this discussion impact my understanding of SoTL?
What are my opinions on this definitional debate?
What connections am I making to my own practice?

So what does it actually mean?

So far, SoTL might seem a bit abstract and you might already be doing it. So, for further clarification, Huber & Hutchings (as cited in Hutchings et al., 2011, Chapter 1) described today’s scholars of teaching and learning as ones who:

treat their classrooms and programs as a source of interesting questions about learning;
find ways to explore and shed light on these questions;
use this evidence in designing and refining new activities and assessments; and
share what they’ve found with colleagues who can comment, critique and build on new insights.

Thinking about this more broadly, SoTL provides us with evidence to underpin our learning and teaching practice. As with most other disciplines, tertiary teaching has a growing research literature that allows us to teach in new and improved ways. We don’t need to teach as we were taught.

To engage in SoTL, we need to:

be informed;
‘experiment’ with our teaching; and
make public the outcomes of that experimentation.

These three elements of SoTL are on continuums from informal practice through to formal practice. The following image shows the continuum of what would be considered scholarly teaching and SoTL.

That is, "being informed" ranges from informal activities like discussions with colleagues through to conducting a literature review with a clear SoTL focus. "Enquiry", or experimenting with our teaching, range from the typical changes that we make to our courses at the beginning of the semester based on feedback and experience through to conducting a research project. The dissemination continuum moves from simply talking with colleagues and sharing your work through to peer-reviewed articles, conferences, and more.

How does this relate to me in my GELTS?

Simply put, your GELTS is SoTL. In your GELTS application, you were asked to identify your student problem, provide some background to the problem, and propose a way forward. These form part of your SoTL focus and are the first steps in the SoTL cycle (which will be discussed in a later post). For now, consider the following questions:

What is your SoTL focus?
What does the literature say about your SoTL focus?
What are your plans to address your SoTL focus?

Discuss your responses to these questions below in the comments. If you want to share a SoTL focus that you have outside of your GELTS, then feel free to do that, too.

Further information

The following resources can provide you with more details about SoTL and how it can look for an academic. These resources are provided as a starting point in your SoTL journey with your GELTS.

Bass, R. (1999) The Scholarship of Teaching: What's the Problem? Creative thinking about learning and teaching, 1(1).
Kerber, C. (2007) What‘s It Really All About? The Scholarship of Teaching and Learning as an Authentic Practice. International Journal for the Scholarship of Teaching and Learning, 1(1), Article 3. doi: 10.20429/ijsotl.2007.010103

References

Hutchings, P., Huber, M. T., & Ciccone, A. (2011). The scholarship of teaching and learning reconsidered: Institutional integration and impact, Jossey-Bass: San Francisco, CA.