505 And so it ends…

This is the last entry into the ed tech learning log for 505-evaluation and I have to say it has been challenging, enlightening, and thought provoking. The assignments and instructional material have ranged from collaborative definitions, to the creation of metaphorical images representing different elements of evaluation to analysis of mock evaluations and finally to the final evaluation.

Through group collaboration students examined the details of the projects to determine need and direction of an evaluation. These assignments enabled students to conduct an in-depth analysis of the evaluation process in a simulated “real-life” environment. The collaborative assignments I found to be especially valuable because I often require this sort of assignment in my own classes and being on the student end of things made me realize just how difficult it can be. Different personalities, learning styles and levels of organization within groups can make for an interesting experience. I would compare it to navigating a ship through icy waters…you are constantly making course corrections.

Specific areas of learning included addressing the differences between evaluation and research, a comprehensive look at types of data, its collection and analysis, sampling, and bias. All of the above areas were made clear through the practical application of knowledge in the development and implementation of the final evaluation project.

My final evaluation was an excellent exercise in which I learned a great deal about the process. The results were somewhat predictable, however I did discover some inefficiencies in the program that will be addressed by the program director in the near future.

This was, hands down, the most challenging of the ed tech courses I have taken so far but it was a great way to refocused my professional direction with regards to creating, delivering, and assessing educational units, programs, and products. The instructional material and assignments have re-emphasized the importance of quality evaluation in the development of quality instruction.

505 – Formative/Summative Assessment etc.

My evaluation final product is a formative evaluation in that it is assess the degree to which the Mount Douglas Challenge Program is successful in achieving their stated objectives. The program is continuous and ongoing, therefore the assessment is not summative, rather formative in nature. The evaluation is an objective based design, thereby decreasing the subjectivity present in other forms of evaluation. (Dick, Carey) My evaluation is a variation on the discrepancy model of evaluation which “…establishes priority educational goals and objectives (of the program), selecting variables and measures of attainment, establishing standards of acceptable performance, actually measuring performance, and finally, identifying any discrepancies between the established standards and the actual performance levels obtained.” (Dick, Carey) My evaluation is designed to evaluate the effectiveness of instruction within the targeted learning group of students enrolled in the challenge program. According to the text, in order for this evaluation to be a truly formative it must include subsequent trials in order to assess the success or failure of modifications made based on initial recommendations. As I am part of this program at Mount Douglas Secondary, I will be able to oversee the extension of my initial evaluation into a second round of assessments adding to the validity and relevance of the entire process. The text suggests that for evaluating instruction typically there are three sequential stages: one-to-one trials, small group trials, and field trials. After each stage of evaluation revisions are made to the instruction based on its findings and recommendations. This process is highly effective in an instructional design setting; however, it does not apply to my evaluation as the program in question is an existing program.

For my evaluation I have collected all the necessary data and am currently involved in the analysis process which can be, as described by the text “…similar to Sherlock Holmes or Miss Marple in a mystery.” I will be using the initial statement of goals and objectives laid out by the program as a framework for assessing its success. All the data collection tools (survey, interviews, observations) were designed to inquire directly about the respondents perceived level of success in relation to the objectives.

Reference

Dick, W. & Carey, L. Formative Evaluation (Chapter 10) Florida State University and University of Southern Florida.

506 – Final Project Draft

This is the last of a series of learning log journal entries for this course. It has been a very interesting journey of discovery. I have particularly enjoyed finding out the reasons why images can be affective and looking at the psychology behind human visual perception. I have spent a great deal of time this week organizing my website and unit of instruction to be submitted in draft form. The unit I have designed is one that I am currently delivering in an Language Arts class and have found the use of graphics to aid instruction highly effective. I’m pretty proud of this work and look forward to hearing some feedback from my peers and instructor.

505 – Post #7-Rubrics

Scoring rubrics are descriptive scoring schemes that are developed by teachers or other evaluators to guide the analysis of the products and/or processes of students’ efforts (Brookhart, 1999; Moskal, 2000). Some controversy sourounds the use of rubrics in the classroom, however the majority of educators seem to believe that they are a valuable tool for assessment when designed correctly. Rubrics can be deemed successful when they are both valid and reliable. Validity is assessed by looking closely at content related evidence such as whether or not an assessment is accurately determining a students knowledge of a specific question, or does the question itself pose difficulties which would invalidate the degree to which it assess the students knowledge in that area. Construct related evidence gives an indication in assessment about the reasoning process of an individual when responding to evaluation. In order to be useful, a valid rubric must strive to identify the internal process a learner goes through when responding. “When the purpose of an assessment is to evaluate reasoning, both the product (i.e., the answer) and the process (i.e., the explanation) should be requested and examined” (Brookhart, 1999; Moskal, 2000). Criterion related evidence in rubrics works to assess learning in relation to external factors such as the application of knowledge in “real-world” settings. Like any well designed lesson, a rubric wiht a high validity level should start with clearly define objectives and each element of the assessment should work to define the level of learning within these objectives.
Another criteria for a useful rubric is its reliability. The scoring of a learner should potentially be consistent when applied at any time or by any evaluator. One suggested strategy for increasing reliability id the use of anchor papers which is a reference sheet for raters to use given a set of test responses prior to administering the assessment rubric. If discrepancies exist between responses and raters than the rubric should be revised. This process would be time consuming and perhaps impractical in a busy public school setting, but nonetheless, it would increase reliability.

Like any type of assessment rubrics have their drawbacks. Teachers are human beings and many times it is very difficult to be completely objective during evaluation. In Understanding Scoring Rubrics: A Guide for Teachers, Carol Boston outlines some factors that at play when scoring such as positive-negative leniency error where the evaluator tends to be too hard or too easy on everyone. Personal bias and teacher student relationships should not be a factor in assessment, but human nature is not so easily beaten. Other considerations outlined by Boston are being swayed by the appearance of a document at the expense of assess the content. Length, fatigue, and order effect can also be a factor in altering the validity of an assessment. A well designed rubric should work to eliminate many of these factors, however some detractors suggest that the rubric is too prescriptive and can reduce a process such as writing to something so mechanical and prescribed that it takes away from the process itself. Personally I have, and will continue to use rubrics, especially as a way to establish criteria for learners before they embark on the learning. One successful strategy I have used in the classroom has been to have the students develop the categories and criteria for an assessment part way through a unit once they have had an opportunity to understand some of the content in the area of study.

Here are some samples of the rubrics used in my classroom as created in the website Rubistar:

MyHero Presentation Rubric
MY HERO Project – Website Rubric
Video Production Rubric
Persuasive Writing Rubric
Blog/Discussion Rubric

 

 

References:

Moskal, Barbara M. & Jon A. Leydens (2000). Scoring rubric development: validity and reliability. Practical Assessment, Research & Evaluation, 7(10). Retrieved November 1, 2011 from http://PAREonline.net/getvn.asp?v=7&n=10

Boston, Carol (2002). Understanding Scoring Rubrics: A Guide for Teachers. Eric Clearinghouse of Assessment and Evaluation, University of Maryland. College Park, MD.

505 – Entry #6

Over the past two weeks we have been engaged in another collaborative assignment which has proven somewhat more difficult than the last. This assignment includes the development of another evaluation based on a senerio that involves making a recommendation to a superintendent about the development of local delivery of online high school credit courses as mandated by the state. The evaluation is to determine potential problems associated with the development of local delivery of these online courses and make recommendations to the superintendent. In finding a direction for this evaluation our group has had some difficulty finding a focus. We have spent considerable time trying to establish the parameters of the evaluation, specifically whether or not we should include a comparison of the potential benefits and drawbacks of online courses at all and whether or not to include a comparison of positive and negative aspects of local versus distributed delivery of the program.

One question we developed: What are the academically established benefits and drawbacks for students enrolled in online courses?

If the state has mandated the requirement should we address whether or not online courses are educationally valuable? My take on this is that if a committee is tasked to find problems with the development of local online courses, one of the issues for teachers (those that will be required to deliver this instruction) will be how is this method better/more efficient for students? Teachers will need convincing that this is a good thing, and we need to be able to tell them why.

Another area of contention is the projected outcome for staff when a mandated online component is required. It is my contention that even though the number of students will remain the same and theoretically there shouldn’t be a reduction in teacher jobs, an online component will require a restructuring of teacher assignments. Some teachers will be unable, or unwilling to adapt. This is a major problem area for me as a member of the evaluation committee.

Our group has arranged to meet tonight on an established Google doc to try and clarify and focus our evaluation…we will see!

In terms of my final evaluation, I have completed my EPD and timeline. After some advise from Dr Perkins I have scaled back the number of over-riding questions in an attempt to reduce the scope of the evaluation. I have interviewed the coordinator of the Challenge program at our school and have developed the surveys for the students, teachers and parents. I will be administering the surveys this week as well as organizing interviews and observations.

505 – Entry #5

Over the last two weeks we have been working in a couple different areas in the course. A  major part of the time was spent on the development of an evaluation quiz in the form of a Google presentation. This was a great way for students to internalize some of the concepts and vocabulary in the world of evaluation, requiring students to find or create images that metaphorically represented the content of each slide. We have also been working on a mock proposal for an evaluation contract individually, and then collaboratively in groups of three. The development of this proposal requires students to consider all elements of a proposal from the introduction to the methods of evaluation to the inclusion of a time line and budget. Through this process learners apply some of the knowledge acquired over the first few weeks of the course in a mock realistic setting. The process of collaborating with peers on a final draft of the proposal has also been valuable as we take the best concepts of all three and find ways to merge them into one. We chose to set up a wiki to house all three proposals and to provide an online space for feedback and comments before drafting a final proposal on a new page on this wiki.

Finally, a major portion of these past two weeks has been the reading assignments on the topic of Problem Analysis. Identifying and analyzing problems is clearly a key element of the job of an evaluator. It is imperative that the evaluator gathers as much information as possible regarding the problem and also be aware of potential problems that may arise during the delivery of a program including the evaluation. It seems to me that a good evaluator needs to be a part time psychologist or at least have the ability to understand the motives and actions of people.

Image Source:http://tinyurl.com/699deh3

505 – Post #4

Driving home everyday I have about forty minutes to get caught up on the news of the world. At home I am bombarded by Sesame Street from the three year old and *cough* Jersey Shore from the seventeen year old.  I was thinking about the collection of data and the statistical analysis of that data and how both the validity of the data and the analysis can be wildly different based on bias and perspective. NPR had a story on President Obama’s announcement of the the introduction of wavers to the No Child Left Behind Law. It seems too many American schools are failing according to the standards set by the NCLB law and school districts can now apply for wavers in order to maintain funding. As a Canadian educator I do not know the details of the criteria established  in the law to achieve a passing grade, but what struck me is the question of how do you accurately measure the success or failure of a school based on statistics derived from exams; moreover, how can you assess the effectiveness of a teacher based on the achievement of her students on an exam?  Basing teacher salaries on the statistical achievement of her students can only result in “teaching for the test”.

In Canada a national think tank called The Fraser Institute releases a yearly ranking of all school in a given province. This ranking is said by the institute to promote healthy competition between schools and encourages teachers and administrators to do better for their students. The institute also claims that parents love the ranking system because they can make informed decisions about where to send there children for school. The collection of data for this ranking is done strictly through gathering government exam results, foundation skills assessments (FSA’s), and graduation rates. There are no “boots on the grounds” of the schools themselves, there are no interviews with children, staff, administration or parents; it is strictly a numerical evaluation of a school’s achievement which lacks any insight into socio-economic factors, ESL, or special needs populations. There is no evaluation of school culture or school as the only place a child can feel safe and get fed in a neighborhood of poverty and danger. The Fraser Institute ranking of schools only serves to demoralize the staff and students of schools that are working hard everyday to educate children, foster curiosity and creativity, and keep children safe. The statistics in this evaluation are skewed and biased. The report compares inner city schools to private schools in the richest neighborhoods in the country. The results favor private schools because of their elitist enrollment policies and offer a wildly narrow view of school achievement.

Here is a link to the ranking of my school which comes in at  112 out of 274, down from 81. (We have had a massive influx of international and ESL students in recent years, so naturally our school has worsened!) There are three High Schools ranked higher in our city; all three are private schools with exclusive enrollment.

No Child Left behind. Obama introduces wavers.

obama-announces-no-child-left-behind-state-waivers

Have a look at this propaganda video released by the Fraser Institute:

The connection I am trying to make here today is directly related to the arguments made about the lack of validity in media comparison studies. The Fraser Institute has an agenda and the strategies and tools used to measure school success and compare school’s do not factor in the variables necessary in conducting a thorough analysis and evaluation.

 

505-Entry #3

This week we looked at research and evaluation and found similarities and differences between the two. What it boils down to is that research is done for the benefit of individuals outside of the participation group, while an evaluation is for the benefit of the participants.

The discussion and information on sampling and bias we had this week I found to be particularly interesting. In my position as staff committee chair at our school I am often charged with gathering the opinions of staff on issues of concern. I have deployed the use of Google Forms, Survey Monkey, and other similar online polling applications many times. Inevitably the same staff members respond to the survey and the same group choose to ignore the document. The text suggests following up with personal contact with those individuals who do not respond to find out their reasons for not responding. I had never really thought about following up on individuals who choose not to respond. It is clear that any information gathered by the same sample of teachers will not be an accurate representation of the thought and feelings of staff. These results may not be an accurate sample based on the lack of adequate number of responses and those who choose to respond may have a particular bias toward the topic of the questionnaire. Another factor may be the wide range of technical abilities on our staff. We have an aging population of teachers who may have a reluctance to using technology, not only in their classrooms, but for themselves as well. If they do not regularly open emails they will be denied access to the survey/questionnaire document all together.
Based on this weeks materials and topics I will be rethinking the process of gathering and analyzing data.

Edtech 505 Post #2

Over the course of this past week in Edtech 505 much of the content has centered around data. We have looked at methods for collecting and analyzing data and how this all fits in to the planning of an evaluation. I have never had much success with numbers, however the explanation in the text has helped to demystify the process of analyzing data. It is interesting to consider how the same results can yield different interpretations based on the technique used to analyze the data. An evaluator can choose a method to look at the numbers based on the intention of the question. It has become obvious to me over the past week that one must consider the techniques used to gather information very carefully based on the particular evaluation model one chooses to use. It has also become clear that the various evaluation models are, in many ways, techniques used by teachers all the time. It is helpful, however to identify the characteristics of each mode in order to accurately use the benefits of each in the development of an evaluation. This week also helped to solidify for me the differences between research and evaluation.

505 – Evaluation – Post #1

In the first week and a half of studying evaluation I have learned about the importance of evaluating the long term impact of a program, or in my case as a secondary educator, long term retention of knowledge, skills, and attitudes. It has been enlightening to consider how summative evaluation at the end of a unit will measure the immediate success of the achievement of learning outcomes for the moment, but to evaluate the success of a program a measurement of true understanding over time would be essential. Unfortunately, in our current educational system we evaluate and move on to the next topic. I recently was at a conference on educational technology that featured Salaman Khan, the founder of the Khan Academy, as a keynote speaker (via Skype). His online videos and accompanying web based exercises are becoming more and more popular in mainstream education as a way to “flip” the classroom; the students watch and learn the material at home and do the projects that deepen understanding in classroom with the teacher as facilitator. The exercises on his web site require mastery before moving to the next level. He used the example in a TED talk (see below) and at this conference of learning to ride a bicycle. If after instruction, a child masters 80% of the skill necessary to ride a bike it seems ridiculous for that student to continue on with Unicycles 101; this is what we do in the school system. Of course, it may not be realistic to expect that all students will achieve 100% mastery at each level of learning, but the analogy endures.
This Khan Academy example also relates to the topic of evaluating impact in that some criticism for this approach is that it relies on rote learning that may achieve mastery in the moment, but will students retain the knowledge, skills, or attitude over time? In other words, has the learning been truly internalized?

Another area of interest for me coming out of the first days of this course has been the notion of credibility of the evaluation and the evaluator. The evaluation process must be part of the development of the overall plan of the program. In fact, the evaluation process should be developed in conjunction with the development of learning outcomes. In building Project Based Learning units it is commonly suggested that one should start with the end in mind. Develop a list of learning outcomes based on content standards and develop procedures for measuring successful achievement of these outcomes before developing the specific strategies of the unit.The evaluation must be an integral part of the overall plan in order for it to have any credibility.

Looking forward into the course I am anxious to discover more specific information on gathering hard and soft data in the preparation of a comprehensive evaluation.