Week 5 - reading reactions - quality assurance in blended learning

Table of Contents 

• Questions to Ponder (before reading)
  After reading 
• Key take aways?  Blended learning quality - says who?
• What about online learning standards? Do they apply?
• So what to do? 
• journal one's work 
• use formative and summative feedback on your efforts 
• invite peer review 
• and traditional ways to measure effectiveness? 
• Engage in Scholarship of Teaching and Learning
• References  

Questions to Ponder (before reading)

• How will you know whether your blended learning course is sound prior to teaching it? How will you know whether your teaching of the course was effective once it has concluded?
• I would prefer to know more about the general nature / background of my students first - especially their interests, competencies. With these, I have a clearer idea as to how I might be able to fully motivate them. I am also fully aware how even with this, I'm forced to make some assumptions that will need to be checked once the Course has started. That translates into frequent surveys and/ or focus groups. Everything else that is part of planning a course is informed by this. This typically starts with a look at desired learning outcomes (typically driven by course stakeholders) and then the available learning resources (limitations on classroom access,  timings, frequency). With these clarified, work can begin on designing instructional methods. 
• With which of your trusted colleagues might you discuss effective teaching of blended learning courses? Is there someone you might ask to review your course materials prior to teaching your blended course? How will you make it easy for this colleague to provide helpful feedback?
• Who would I trust to give me an effective review of my blended learning courses? Ideally this would happen with others who also have an understanding of what is involved in designing blended learning courses. I can realize much of this by making transparent much of the design work listed above - perhaps through Google docs of some sort of online portfolio. 
• How are “quality” and “success” in blended learning operationally defined by those whose opinions matter to you? Has your institution adopted standards to guide formal/informal evaluation?
• Good question. How do we typically measure quality and success in any learning venture? Ultimately that can be seen in the scope and richness of the learning artifacts that learners produce as a result of the course methods .. and then a look at how well these align with the course's learning outcomes. The operative word here is "effectiveness" and then the problematic nature of measuring that effectiveness. It should actually be easier to do with blended learning now that more and more of the processes used can be made transparent to others. 
• Which articulations of quality from existing course standards and course review forms might prove helpful to you and your colleagues as you prepare to teach blended learning courses?
• Not to elude the question but Quality is typically being measured in terms of "effectiveness" and "efficacy" and the discovering of both involves examining all of the variables that went into the design of the Course plus the learning artefacts .. but most important of all  

"Did it change anything for the learner?" 

• How is that last point measured? Feedback from the learners. How to get quality feedback from learners? Surveys, an examination of the learning artifacts, focus groups. Also important will be alignment - checking to see that what we "teach" is driven by the desired "learning outcomes" .. and that this has been learned by the student as demonstrated by the "learning artifacts". 

After reading

Key take aways? Blended Learning Quality - says who?

Still early innings on blended learning. By that I mean, there seems to be so little research on blended learning in general and especially on how to measure quality and effectiveness. Very problematic to try and identify what constitutes the best combination of online and face to face interactions. The same could be said for any instructional design - traditional face to face included. The article suggests that this may be partly due to the almost complete absence of any standards. (No surprise to learn) Attempts have been made to suggest what contributes to "quality" blended learning. Noted here is 

• rigorous learning assessment Picciano and Dzuiban, 2007)
• responsiveness of learner characteristics (Skibba et al, 2014)
• student engagement (Vaughan et al, 2014) 
• high quality faculty development (Dzuiban, Hartman and Mehaffy, 2014)  

Important to note that the major issue with any set of standards is the risk of them becoming prescriptive. I think Thompson makes a real valid follow up point here when he notes that there may be a tendency for any standards that do exist, to be taken up and applied indiscriminantly by educators. Perhaps in their zeal to apply something to elevate the quality of their blended learning work, they are just grateful to find anything and thus may not take the time to critically evaluate such standards - especially in the current status (history) of BL.  Yet it is clear that we need to be critical of any such standards in the current "exploratory" phase of blended learning. Key questions? How were any such standards arrived at? By whom? In what context? 

What about online learning standards? Do they apply?

Because online learning has been subjected to much more scrutiny in comparison with blended learning, what research has been done here may hold some answers.  Big surprise here was to read how such standards have emerged not from traditional educators but from for-profit companies. Perhaps this is motivated by the power of standards to attract the attention of educators and thus help market / promote their product (e.g. Blackboard's Exemplary Course Program). These are usually presented in the form of a checklist (e.g. CSU Chico's Rubric for Online Instruction, and Illinois Quality Online Course Initiative)

So what to do? 

 

Journal ones own work. 

Observe your work and attempt to identify your own standards. Check and evaluate. Experience is the best judge of this..

 

Use formative and summative feedback on your efforts 

This involves soliciting students for feedback on any / all of your designed interventions as to student perceptions of their purpose and effectiveness - during the course or after a course ends. 

 

Invite a peer to review your work  

Invite one's Blended Learning peers (wherever to find these?) to engage with my own review of my work and to critically challenge it. Invite peers to review resources, activities, assessment strategies, course structure and communication strategies. Essentially what I am hearing here is the need to mobilize a community of practice that is focused on "blended learning" where such communities become the incubator to sharing ideas, and over time, help identify best practices. Perhaps they also ascend to the lofty heights of creating a set of "minimum guidelines" or even standards that most blended learning educators could support.



And traditional ways to measure teaching effectiveness? 

A teacher's effectiveness continues to be measured in how they make a difference to students and their quest to succeed. Factors that have been identified here include 

• organization of the course
• knowledge of course material
• ability to communicate clearly with students 
• frequency and quality of feedback to students 

 

Engage in the Scholarship of Teaching and Learning  

Good quote.

Those of us involved with blended design will need to adopt the attitude of learners, examining our practices and seeking continually to improve based upon the most current information available. Perhaps this is done in dialogue with trusted colleagues. - Thompson

References 

Blackboard. (n.d.). Exemplary Course Program | Blackboard. Retrieved March 28, 2016, from http://www.blackboard.com/consulting-training/training-technical-services/exemplary-course-program.aspx

Blended Learning in K-12 - Wikibooks, open books for an open world. (n.d.). [wiki]. Retrieved March 28, 2016, from https://en.wikibooks.org/wiki/Blended_Learning_in_K-12

CSU, Chico. (n.d.). Welcome to the Exemplary Online Instruction (EOI) website - Exemplary Online Instruction. Retrieved March 28, 2016, from http://www.csuchico.edu/eoi/

Eades, J. (2014, September 30). 3 Ways To Measure Training Effectiveness. Retrieved March 28, 2016, from http://elearningindustry.com/3-ways-measure-training-effectiveness

Hirtz, S., & Harper, D. G. (2008). Education for a Digital World: Advice, Guidelines and Effective Practice from Around Globe. Commonwealth of Learning, Vancouver and BC Campus, Canada. Retrieved from http://oasis.col.org/handle/11599/52

ION Initiatives. (n.d.). Quality Online Course Initiative. Retrieved March 28, 2016, from http://www.ion.uillinois.edu/initiatives/qoci/index.asp

Morrison, D. (2015, May 26). How “Good” is Your Online Course? Five Steps to Assess Course Quality. Retrieved from https://onlinelearninginsights.wordpress.com/2015/05/26/how-good-is-your-online-course-five-steps-to-assess-course-quality/

Picciano, A. G., Dziuban, C. D., & Graham, C. R. (2013). Implementation of Blended Learning for the Improvement of Student Learning. In Blended Learning: Research Perspectives. Routledge.

Singh, H., & Reed, C. (2001). Achieving Success with Blended Learning. Centra Software. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.114.821&rep=rep1&type=pdf

Thompson, K. (2005a). Constructing educational criticism of online courses: A model for implementation by practitioners. Retrieved from http://purl.fcla.edu/fcla/etd/CFE0000657

Thompson, K. (2005b). What is Online Course Quality? Retrieved from http://ofcoursesonline.com/wp-content/uploads/2008/06/thompson_onlinecoursequality.pdf

Thompson, K. (2011, June 27). BlendKit Course: BlendKit Reader: Chapter 5. Retrieved from https://blended.online.ucf.edu/blendkit-course-blendkit-reader-chapter-5/

Week 3 - reading reactions (before / after reading)

This week was a bit of a disappointment but perhaps it should be viewed as reassuring too. Much of what was said and written was largely already known to me prior to taking the Course. A few new approaches and perhaps a refresher on ideas that I hadn't come into contact in quite some time. 

Table of contents

• Focus Question 
• Objectives 
• Questions to Ponder 
• Notes from reading 
  • Not a whole lot new learned here 
  • There were a few bright spots however
    • non graded multiple choice quizzes
    • presenting video examples or explanations 
    • presenting non credit, online practice exams 
    • key online assessment tool features 
      • T/F alternative
    • informal assessment design checklist 
    • one sentence summary 

Focus Question

How will I determine students are learning?

Direct creation of learning artifacts by students that demonstrate their understanding of concepts through their application of concepts to realize something. Ideally whatever is produced is authentic in the sense that it invites application to a situation / simulation similar to one where these same skills and concepts would need to be applied. 

 Authentic assessment—assessing student abilities to apply knowledge, skills, and attitudes to real world problems—is not only possible in an online environment; it is getting more popular. (Thompson, n.d.)

Objectives

After completing all of this week’s activities, you should be able to:

• Determine  assessments appropriate for the learning objectives you’ve established in your blended course
  • assessments are aligned with learning objectives; methods lead practicing skills and competencies that are to be tested in assessments (everything connects) 
• Revise existing course documents as appropriate based upon assessment decisions
• Configure online quizzes for your blended course
  • self directed, formative assessment quizzes 
• Create written student instructions for each graded assignment in your blended course 
  • checklist

Questions to Ponder (before reading)

• How much of the final course grade do you typically allot to testing? How many tests/exams do you usually require? How can you avoid creating a “high stakes” environment that may inadvertently set students up for failure/cheating?
  • How much of the final course grade do I typically allot to testing? For the most part, it has not been determined by me but by those responsible for the course syllabus. However, as a rule, I would want to opt to have assignments weighted equally throughout the course and also have assignments be sequenced in such a way that they build upon earlier assessments and build towards subsequent assessments. 
  • I've learned that putting more than 20% weight on any one assessment typically means making it "high stakes".  The draw back? Performance anxiety - a one off "disaster" can effect a student's final grade - thus explaining the "high stakes" nature of it. We want students to demonstrate their learning and unless managing "stress" is considered an important part of the assessment, a learning objective, then it should be minimized as much as possible. How to avoid it? Simple. Don't make one assignment so high stakes. Another option is to design an assessment so that it is ALSO a learning experience. Even invite students to make notes or drawn on notes that they think will help support them in the completion of an assignment - perhaps even make this a group / collaborative activity so they once again, engage with course contents, negotiate understanding and meaning prior to completing the assignment. Invite students to even use these notes to help them complete an assessment.

• What expectations do you have for online assessments? How do these expectations compare to those you have for face-to-face assessments? Are you harboring any biases?
  • I think that I should value online assessments as much as any other assignment that is given in a face to face context. However, I can imagine a healthy dose of skepticism from peers and supervisors (even students) who believe that when done sight unseen, students are inclined to "take advantage" of the situation to round up help and thus undermine the validity of the testing. Yet the few articles that I've read, surprisingly indicate that the incidence of cheating online is not even marginally significant when compared with that in a face to face context. 

• What trade-offs do you see between the affordances of auto-scored online quizzes and project-based assessments? How will you strike the right balance in your blended learning course?
  • Trade offs with auto scored online quizzes? Immediately what springs to mind is how objective testing is essentially inviting practice in lower order thinking skills (memorization, identification, classification) but far less likely to be taxing higher order thinking skills. However, any course of study does require a basic competency in the discourse of its subject and these sorts of online quizzes become useful for both ensuring students make them a focus of study and for monitoring student success in understanding them. Project based assessment would be far more welcomed for its ability to promote application, judgement and synthesis of concepts and thus promote higher order (critical thinking) skills. But the caveat with them is in how much more time they require to be reviewed and assessed. In a MOOC with participants that number in the 100s or even 1000s, this is just not feasible. What then needs to be entertained is the use of peer review with a clear and comprehensive rubric - perhaps supported by an instructor's own reflections and guidance (thus the peer assessment also becomes yet another "learning / application / judgement experience"). The balance? Between what is humanely possible for the instructor and what is most beneficial to the students - essentially choosing the "biggest bang for the buck".

• How will you implement formal and informal assessments of learning into your blended learning course? Will these all take place face-to-face, online, or in a combination?
  • My (yet to be identified) blended learning course would definitely see a combination of the two and ideally with one supporting the next. By this I mean that the concepts studied and learned for one assessment are continued / built upon / reapplied / reinforced in subsequent assessments so there is a cumulative reinforcement of them. 

Week 1 - my notes from webinar

The Week 1 Webinar

with Dr. Glenda Gunter (edtech integration) & Dr. Beth Young (BL English instruction)

How has BL changed just in the past 10 years?

Dramatically .. much more social than it once was. New technologies such as Google Hangout are  making it easier to do so. As a result, online assignments can be made social.

Biggest challenges in BL?

Academic Integrity  (in other words - how to minimize cheating?)

• Suggestion made by both speakers that perhaps academic integrity is no greater an issue with online assignments then it is with a face to face assignment. Solutions to managing it included creating question pools to help realize randomized Quizzes.

Deciding what to include (temptation to include too much)

• Suggestion made to be critical about what is most important to include to cover course objectives. 

Making courses too intense (to overlook demands on student time to complete assignments)

• Similar to "deciding what to include" there is a need to critically think about just how much time it will take for students to complete a given assignment. Choose assignments that make the most strategic sense for the student to learn with and/or demonstrate learning yet are not excessive in their demand for time.

From my own studies, we should encourage the design of assessment items so that they invite a student to incorporate material that has been covered in the course into the creation of something that helps them demonstrate their understanding / application of it .. Better yet design assessment so that it can be personalized so that it is meaningful to the student - make the assessment not just a vehicle for measuring how much they have learned but a valued learning opportunity for the student too

Common BL problems

• not reviewing the curriculum ahead of time 
• solution? need to be thinking in advance about which assignments best suit being flipped / done online and those that best suit being done face to face 
• lack of organization about what is an online or face to face assignment and when it is due
• solution? generate a course calendar and code assignments as either O (online) or F (face to face) so that it is clear to a student when an assignment is due and in what form it will take
• confusion over feedback ... when is the feedback for an O (online) or F (face to face) assignment 
• solution? make sure feedback is given in the same manner / mode as the assignment was originally presented  (e.g. O assignment = O feedback) 

Top tips

• identify assignments that need significant explanation ... and be sure to make those face to face so it is easier for students to understand and ask clarification questions  
• do things that cannot be done online or are best done in a face to face setting
• do things that cannot be done in a face to face setting or are best done online 

 

Should the instructor give the students a time frame for when he/she will be online?

Difficult to define... but it is important to be aware of where students are at in the program and where one can anticipate students having problems. Suggestion made to set up some online office hours but from the experience of at least one presenter, these are not often used. Instead, one strategy that has worked is the set up of a virtual coffee shop .... to invite students to ask questions between one another. These should be monitored by the instructor and if student understanding appears to be "way off ".. then the instructor can catch it.. and then indirectly address the itme via a post in a regular discussion board (not in the virtual coffee shop itself).


Key resource 

BlendKit UCF. (2016). BlendKit2016 Session 01: Understanding Blended Learning. Retrieved from https://www.youtube.com/watch?v=4DaOPX_8Ud8