Evaluation vs. Research

1. While evaluation and research do overlap on many aspects, and can often be overgeneralized when being defined, it is important to understand that these two processes are very different. By intent evaluation concerns itself with informing particular stakeholders about programs or goals while research is looking to inform those “well beyond the stakeholders” in order to move forward with new learning and knowledge (Boulmetis & Dutmin, 2011, p. 171). By design and by purpose priorities it is often fairly obvious whether or not evaluation or research is being conducted. Research is often more concerned with cause-and-effect, whereas evaluation is designed so that “programs can change” (Boulmetis & Dutmin, 2011, p. 173). In research, the sample is important as it needs to be nonbiased and appropriately collected. However with evaluation, as it is concerned with a specific group or organization, by design will be a limited generalized sample.
Again a major difference between both processes is the audience. Evaluations are very much designed and implemented and shared with particular stakeholders for specific purposes, in fact, often the results are never shared beyond this group. However, research is shared via peer review and often published for others to learn from the research.
2. The Wikipedia article discusses the reasons behind conducting the evaluation as efficiency and effectiveness, it leaves out the possible purpose of impact. The entry targets the importance of stakeholders as the reason that many evaluations are started and also discusses the importance of engaging with stakeholders. I thought it was interesting that the entry also laid out the pros and cons of external versus internal evaluators which we discussed in our coursework as well. I thought something lacking from the entry was an outline of the major evaluation designs (e.g. goal-free model). Both our course work and the Wikipedia entry highlight the evaluation framework as stages requiring 1) needs assessment and 2) program planning and then the stages differ. I didn’t notice a specific Evaluator’s Program Description framework within the online entry article which was a major piece of our work so far this semester.
3. First off, I found it interesting that there is an Amercian Evaluation Association, I definitely have not run into very specific organization before. I appreciate that the organization offers a lot by way of connections to its members through groups and social media connections. I took the opportunity to search for evaluators for the District of Columbia and was amazed by a number of names and organizations that filtered for possible connections if I was looking for an evaluator. I took some time to review their mission and vision and was pleased to see they have quite the set of governing policies complete with a guidebook, end goals, and strategic plans. This reminds me of the course I took on Instructional Design and how fascinating it was that this was a whole new world of careers and job focus.

Reference

Boulmetis, J., & Dutwin, P. (2011). The ABCs of evaluation: Timeless techniques for program and project managers. San Francisco: Jossey-Bass

Advertisements

Screencasting – Short Form Educational Video

For Module 4, I explored the topic of screencasts as a short educational video. I had never used Screencast-O-Matic but quickly learned how to manipulate the free software to showcase the tutorial I was looking to teach. There are many screencast programs available some paid and others have free and pro models with differing capabilities.

Screencasts are recordings of both the audio and visual of what occurred on a computer screen throughout the recording. This can include the instructor switching between tabs as well as the possibility of recording a webcam image or possible cursor highlights or indicators

Screencasting has many powerful implications for the educational world. The major concept of the “Flipped Classroom” thrives on the ability for teachers to create and record videos teaching a particular concept or topic. Teachers can record themselves teaching with whiteboard apps or powerpoint slides as a possible way to create these videos. Flipped classroom videos allow students to “move at their own pace” (Ruffini, 2012). With videos, including screencasts, students can pause, rewind, fast forward all with the purpose of enabling students to control their understanding and increase their learning and comprehension. Michael Ruffini (2012), with EDUCAUSE Review Online, suggests also embedding Google Forms below screencast videos on classroom blogs for students to turn in answers to questions or write reflections based on the video. This reminded me of our Media Literacy lessons where we embedded videos into our Google Form itself – as this has been an upgrade since 2012.

Screencasts can also be a way for teachers to record themselves leaving feedback as they reflect on digital student work. Imagine a teacher recording a video as they score or revise a student’s paper. That student can watch the video of their own teacher editing their work for better insights into the feedback being delivered. This method will allow for much more feedback through audio rather than a teacher trying to write or type everything.

A third way that screencasts can be meaningful for students is if they themselves are creating videos. Apps like Educreations and Showme easily allow students to do this. I also encourage my teachers to use their SMART boards as a center and teach kids to create a screencast (using SMART record) to teach a concept to the class. Sometimes these videos are shared with parents, played for the whole class or shared with a small group. This is very motivation for students and allows them to practice many language skills.

These are just three of the many ways screencasts can be used in the K12 classroom. Have you tried integrating screencasts into your work? Have you had any successes you would like to share?

Student Screen Cast Sample

Resource:

Ruffini, M. F. (2012). Screencasting to engage learning. EDUCAUSE Review Online. Retrieved fromhttp://er.educause.edu/articles/2012/11/screencasting-to-engage-learning

ST Math – Blended Learning in California

The MIND Research Institute contracted the Evaluation Research Program at WestEd to assess their blended learning program, Spatial-Temporal Math (ST Math), in an elementary school setting in California. WestEd used data from the CST (California Standards Test) from grades 2-5 to compare CST scores of students using ST Math to those similar groups of students not using ST Math. WestEd compared first-year full implementation usage at schools to those not using ST Math. Full implementation usage was defined as 85% of students completing at least 50% of the blended learning curriculum during the course of the school year. Results showed that grades using ST Math showed higher scores than those students not provided with ST Math and that the most significant gains occurred in 2nd, 3rd, and 5th grades.

Interestingly enough the research group made sure to clearly define that grades were made up of all the classes within participating schools. For example, if a school had six grade 2 classes, all were included regardless of individual classroom implementation percentages. And that 212 schools were included in the full implementation groupings. It was also interesting how detailed the evaluation went by breaking down the CST data not only overall but also by growth for advanced and proficient students. I would have been interested to see results for far below basic and basic performance level as well.

It was also intriguing how clearly WestEd wrote up the limitations to the evaluation. They suggested that data may have been affected as ST Math was implemented by choice within schools and motivation, surrounding math in general, could have been high with the self-elected new program. The evaluation groups also suggested that some limitation existed as it was impossible to control that students hadn’t used the program in previous years, even though the study indicated it was a first-year usage evaluation.

Wendt, S., Rice, J., & Nakamoto J. (2014). Evaluation of the MIND Research Institute’s spatial-temporal math (ST Math) program in California. WestEd. Retrieved from https://www.wested.org/wp-content/files_mf/1415393677Evaluation_STMath_Program_20141107.pdf

ID: Learning Outcomes

When reading this statement, reflect on the Instructivist and Constructivist.   Do you see a connection between the two learning outcomes?  Illustrate your position by using an instructional experience and an understanding of Bloom’s.

Instructivist theory is the idea that students will learn through direct instruction approaches and constructivist hinges on students learning through discovery. It is important that teachers do not decide to pick one or the other exclusively, but rather approach teaching from the lens of a responsive pendulum. The video in the presentation really challenged teachers to go beyond one vs. the other but to utilize both theories (along with connectivism) to support individual learners where they are and move them forward.

The best teachers are those that model or scaffold through discovery moments when deemed appropriate and only for the students who need that support. This is discovered by the teacher through continually gathering formative feedback. It is important that students participate in constructing their own learning so that they can make connections and create meaning, but the teacher doesn’t have to leave the room for this to happen. The teacher can still play a pivotal role for students who need guidance in this process.

I recently was observing a teacher deliver a whole group mini lesson. She quickly determined through a formative poll that many students were ready to explore or practice on their own. She quickly changed her plan and separated students so that some were working on creating Educreation videos to teach others the concept. Another group took a sentence stem handout to work through the problem, and she invited another 7-10 students to remain on the carpet to continue working through the mini-lesson. Even then students were “released” from the mini lesson at different times as they gained the confidence needed to construct meaning around making tens. This teacher had students working within differing Blooms levels, after realizing every student did not need direct instruction. Did I mention this was a first-grade classroom with 32 students?

Larson and Lockee state “‘Changing workforce needs in the 21st-century demand that instruction and assessments build learner abilities such as expert thinking and complex communications, defined by Willis as ‘recognizing and organizing patterns and relationships and identifying and solving new problems as they arise.'” Explain why you agree or disagree with this statement. How should this impact the assessments we create?  

I agree with Willis’ (2006) statement as Larson and Lockee point out the changing landscape for learners today. There is so much access to technologies and information learners need to understand how to access in order to solve problems and redefine our understandings. It isn’t enough to simply memorize and regurgitate, as access to information is immediate.  I appreciated the comic at the beginning of chapter 7, that suggests people wouldn’t be happy allowing someone to give them a shot if they have never practiced actually doing this before. Getting an A on the test wouldn’t cut it as a learning assessment for this learning outcome. It also was pointed out that a multiple choice exam can highlight if a student can recall or can recognize – and recognition is sometimes the key necessity to follow-up application.

It is important that educators take care to also develop alternative assessments to foster assessment for and assessment as learning. For learners to grow to be capable of supporting and enhancing the 21st-Century workforce they need experiences authentically showcasing their understanding of learning outcomes throughout their educational career. The ability to respond to open-ended questions, produce a concept map or successfully navigate a performance assessment asks a student to exist on a higher level of skills.

Explain a lesson from your class and the levels of interactions within that lesson. Are there some interactions that are more essential or more important than other interactions? Explain what makes the difference. How did these interactions change the instructional strategies you chose? Be sure to discuss learner-to-content/instructor/context/learner/self.  

Recently I was working with a 1st-grade class that had to write letters based on books they had read. The letters were written for a kindergarten class to help them learn the lessons from the books that the 1st-grades had learned. The Learner-to-Context interaction had the 1st-grades interpreting the morals of stories and describing them in ways younger learners could then understand them.  The Learner-to-Instructor interaction came in the form of the teacher asking probing questions to help the 1st-grade writer determine if they had included enough detail for their future kindergarten audience. The Learner-to-Context interaction was well thought out in this classroom. Students work at small table groups but are encouraged to keep voices down during conferencing to minimize distractions. When peers are conferencing they sometimes go to the reading corner or other areas to allow students who are still writing to continue unphased. The Learner-to-Learner interactions come as discussion and peer conferencing. The Learner-to-Self interaction was highlighted in the form of a self-reflective rubric. Students were asked to make sure that their letter contained certain elements and the rubric was used by students to self-evaluate.

In this example all of the learner interactions were important. That by itself caused the teacher to create procedures and methods for students to write, conference, peer conference and self-evaluate. I have seen this teacher incorporate student friendly rubrics this year in an effort to support learner-to-self interactions as it was determined that some students needed assistance and scaffolding to become more self-aware and reflective.

Different pedagogical approaches use different planning processes to address content and learning experiences. Which of the five strategy frameworks have you used to develop the type of learning outcomes you have identified in one of your lessons or assignments?

Prior to this course, the only strategy framework that was somewhat familiar to me and my work was Gagne’s Nine Events, and even now I have a much better handle on it. However, Keller’s ARCS Motivation Model has really stood out to me as something I want to keep in the forefront when I plan. As a technology instructional coach, I am asked to take data throughout my cycle with a teacher to show student growth over time with relationship to our coaching goal. Many times the goal hinges on student engagement and time on task. The ARCS model alone or even in conjunction with other models fits this goal so well. What better way to increase engagement and time on task than with planned increases in confidence, motivation, relevance and attention-getting strategies.

Case study 06 was about K-12. Case study 08 discussed higher education. Case study 23 focused on the private sector. Which one did you most identify with and why? 

I definitely identified most with Case Study 6, due to the K-12 content. Not to say that Case Study 8 wasn’t interesting, I am always facisnated by people who do not work in a school setting all day! Learning about the main character who got a masters in instructional design reminded me that I wasn’t even aware of this position a few short weeks ago,  however, I digress.

I found Case 6 interesting as our school is newly rolling out some new technologies due to a Summer renovation. We are adding an additional wing this upcoming fall and we again will be funded for technologies within there buildings thanks to student/teacher success, and my coaching trackers. It it is happy to see that technology and the role of the technology instructional coach is being valued and having a positive affect at my site. If only this could happen everywhere.

ID: Analyzing the Context of Instruction

1. Describe an instructional design scenario in which Gagne’s “Conditions of Learning” are applicable.
As an instructional design theorist, Robert Gagné, theorized that there were five major groups or categories of learning in his book “Conditions of Learning”. These categories are intellectual skills, cognitive skills, verbal information skills, motor skills and attitude. While Gagné developed instructional design theory for learning and planning he emphasized the need for design to be created and altered to fit particular learners needs. While there was another set of “conditions of learning” in the group’s Youtube clip, I am going to focus on Gagné’s five categories of learning for this response. I would bring up project-based learning as an example where these categories of learning may be applicable. Perhaps students are tasked with designing a toy after researching and studying what makes a good product and invention. Students would have to use motor skills and have an excellent attitude both to collaborate and sell their product.
 
2. Within cultural context, the text describes the impact of danger of “ethnocentric design”.  Consider a scenario where ethnocentric design has been illustrated in the design of instruction, and discuss opportunities to learn, inform, and remedy the design process.
At my school site, we have what are called Academic Parent Teacher Team meetings. They happen about four times each school year with the classroom teacher and parents coming together for about 75 minutes to focus on skills, goals and action steps as a way to support classroom learning at home. Some teachers have varied success as far as attendance and overall parent satisfaction with the approach. With feedback and thoughtful reflectiveness, we as a staff, are continuing to work hard to eliminate ethnocentric design for our APTT meetings. Teachers have had to work hard to develop their sessions in a way to support all families. We now offer all the sessions and all materials in Spanish and English. Teachers have been placing QR codes on materials so that students and parents can access the text after the session, no matter their literacy level. Icebreakers are developed to be safe and fun and not to put any one parent on the spot. Coteachers circulate during the evening to assist parents and teachers no longer expect parents to ask questions in front of the whole group. Teachers have worked hard to describe student assessments without teacher jargon so that parents can understand how to support the learning goals of the grade level. Attendance has continued to grow as teachers take into account the needs of parents in our community.
 
3. In what ways do the elements of KASI- Knowledge, Attitudes, Skills, and Interpersonal Skills, help you shape the way you design instruction?
In the Larson and Lockee text, I found it helpful to think of KASI as an example of a “know/do/be framework”. This can be used to help develop design to focus and streamline required skills. At times this even allows the designer to group together common elements that can be taught and linked together for students to make connections. By focusing on knowledge, attitudes, skills and interpersonal skills a designer can hone in on “need to know” vs “nice to know”. As I recently developed a training for staff on using the SMARTboard as a center, it was important to target teachers’ existing knowledge of procedures and rules and attitudes towards the helpfulness of these. This helped to get teachers in the right mindset to create their SMARTboard center procedures. And learn how to grow this into their current practice. Of course I could have taught them all things SMART software related, but this wasn’t the focus of the session. As far as interpersonal skills I try to build in time for teachers to discuss/share and reflect on current practice and new skills as they begin to use them in their own classrooms.
 
4. Describe a situation in your experience where cultural understanding played a positive role in designing instruction or describe a situation where cultural understanding could have avoided a failure in instructional design.
As I have shared,  I work at a bilingual school (Spanish/English). To foster and keep this dual language model alive we employee and maintain many bilingual teachers. We work closely with some programs around the world to help bring teachers from Spanish-speaking countries into our school as needed. We have teachers from Latin America, Spain and the US. This plays a major role in planning professional development for our staff. It is important to understand the role of lesson planning, assessment and feedback in other countries as we work to grow our teachers. Often times our international teachers will have a hard time with the teacher evaluation process at our site, it is so different for them. We have worked to help them understand these differences through videos of best practices, exemplars, coteaching models and scheduling observations to see other teachers. As a coach and designer, I work hard to design instruction for this group of staff that will help highlight differences in the American educational system and ways to support them as they figure it all out.

Resource:

Larson, M., & Lockee, B. (2013). Streamlined ID: A practical guide to instructional design (pp.269-270). New York: Routledge Taylor & Francis Group.

PBL: Writing a Driving Question

It all stems from a driving question

Google isn’t always the answer. Not when you want students to think critically and grapple with a challenging or thought provoking question or concept. The essential or driving question that fuels a problem-based learning experience must take students on a journey that cannot end before it begins – which is where most directed or single-answer questions go to die. If students can answer yes or no, or Google can give them all they need in one perfectly crafted search query then, the question being posed is not a driving question.

A driving question allows the learner to be challenged with an experience that will take multiple activities or experiences to discover the open-ended answers. It is important to keep students interested so it must be an engaging question. It should obvious align itself to learning goals and standards as well.

As I work to develop my PBL “Discovering Flight” for 1st-grade students, I have crafted a driving question which attempts to include the necessary components of a successful driving question and sub-questions.

Driving Question: How and why do we use flight in our community to make our lives better?

Sub-Questions:

What is flight?
What items or things can fly? Living or non-living?
Why do things fly?
Where does the power of flight come from?
What causes things to fly?
How did humans get involved with flight?
What is the history of flight?
How would the world be different if there was no flight?
How has flying changed and improved over time?
What possibilities does flight have for the future?

My current driving question is open-ended and definitely cannot be answered with a simple Google search. It gets students to tackle many sub-questions in order to attempt to answer the driving questions. The driving question will interest students as it gets them to look into their own community as well as meet with a specialist (pilot) and take a field trip. Students will be interested in the unit texts and science experiences with flight.  This driving question is aligned with learning goals as it asks that student collaborate, read, write, listen, speak, experiment and explore. The sub-questions are laid out purposefully to guide students to understand all of the individual components of the driving question and all the components of flight, flight history and flight implications.  I look forward to further development of this project and look forward to feedback!


 

Resources

BIE Webinar – “Driving Questions: What is it?”

 

ID: Prepare for Success

1. Analyze a learning context where initial needs assessment might be tricky. Is the performance gap clear? Is the reason for the gap related to instruction, knowledge and/or skills. Are there other factors that might need to be addressed before determining whether or not there is an instructional need?

At my site, we have a math blended learning program that all teachers are expected to use with their students twice weekly for a total of 90 minutes. The time is scheduled into the master schedule and is protected in terms of lab/mobile lab check out. Students are expected to master 3% of the content each week to be “ready” for standards-based assessments come this Spring. However, each year consistently Kindergarten and 5th meet this goal, while all other grades fall short. As a leadership team, we have strategized a lot about this issue. We have laid out expectations for teachers to facilitate and question students during this time. We have shared research and data findings with staff as to the track-record with this program. We have required that staff complete the online training to better facilitate this program. Coaching has been offered to teachers who struggle using the program with their students. I don’t think that there is an instructional need. I know that teachers have trained in the program, they feel that with so many other things on their plate, the time students are using the blended learning program is a time teachers can do things they normally have no time for. Often times I see teachers testing students (one-on-one testing for reading levels), answering emails, or grading papers. None of these are a bad use of time as they all need to happen. I don’t think there is an instructional need for teachers to learn the program, they have done this. I think teachers know how to use and facilitate the program, however, time and time management gets in the way. I think it would be powerful for admin to speak with teachers about their belief in the program, but also open up a dialogue for less scheduled meeting time so teachers can focus on their class and help them grow even during blended learning moments.

2. Compare a situation where instructional materials seem to meet the needs of the learners with one where they do not.  Are there clues that needs analysis is/is not part of the design? Do the materials provide formative assessment?

I will use an example from the ID Casebook as an example of instructional materials not meeting learners needs. In case 28, Natalie is tasked with  with presenting solutions to a broken manufacturing training program. The current system is so entirely lacking in structure that trainers aren’t holding trainees to the same standards. While formative assessments were thought to be in place, it is evident, as Natalie interviews managers and trainers that not everyone is upholding standards or even understands their importance. It is evident that a needs analysis had not been completed as the current “program” is flawed in many ways. The most glaring lack of analysis was that non-native English speaking trainees can only advance in certain areas if they speak Vietnamese or Spanish, as these are the languages spoken by the trainers. I am confident that this, along with all of the other obvious problems, will be resolved as Natalie was brought in as a consultant to develop an instructional design.

An example of instructional goals and materials meeting the needs of the learners can be taken from a recent PD at my site. We brought in a trainer for eduCanon (PlayPosit), as our district recently purchased licenses. I was able to pre-conference with the trainer to give her background about our teachers, our students and best fits or needs for the integrative video tool. I was able to meet with teachers to understand their needs and hopes for the program if they were to attend the PD. Overwhelming and not surprisingly, ahem Adult Learning Theory, teachers wanted to walk away with usable products they could implement later that same week. I also prepped the trainer to come with supports that program had for audio and visuals for our kinder teachers so they weren’t turned off by the program. Needless to say, the pre-work paid off. Teachers came with video links they were ready to manipulate and they were trained to work with their own content in a very real and usable way.  Formative assessments were used throughout the process as checks for teacher understanding were put into place. I am happy to report, from the admin dashboard, all teachers in attendance have gone on to create and implement – with students – additional video resources.

3. Describe  an instructional situation in which there are elements of behaviorism, cognitivism or constructivism in the instructional design or delivery.

As I work in a bilingual (Spanish/English model) school, while reflecting, I realized I am able to bring all three learning theories into focus. A major goal of our dual language program is bilingualism but also biliteracy, with this lens a lot of work needs to be done for students to reach this level of proficiency.  Behaviorism practices  can be seen in flashcard work or centers activities as student work with sight words, letters, and syllables. Teachers will prepare interactive sorts for the whiteboard that give positive feedback for correct input. An example of cognitivism is witnessed as students are charting and discussing cognates and false cognates as a connection for themselves between the languages. And constructivism can be experienced as students present their final PBL projects in both English and Spanish at unit’s end.

4. Explain how the emphasis of one or more of these theoretical approaches may impact learning.

It is important that designers and teachers understand these different theoretical approaches so that they are intentional in their lesson design and implementation. There is a time and place for behaviorism, as I think to our young student who need to practice and understand letters, letters sounds, and print concepts. These reading behaviors need to be practiced and mastered so that meaning can be applied as they attempt to participate in higher level thinking activities. These foundational skills are so important, but if learning continued in this way for everything, meaning and connections for that learner would never develop. The real “meat” to knowledge and learning comes with the connection building and meaning creation through higher level processes. The teacher or designer must embed these intentionally for a student to find joy and power in their learning. Again gravitating toward younger students, while a child may still be practicing their letters and sight words, this doesn’t mean they cannot participate in a book discussion or sequencing activity. It is important designers utilize all theories to best support their learners based on the goal or objective.