PME 802

Educational Program Evaluation

In my current role as English as an Additional Language Coordinator in a large international school in Beijing, China, I recognize a fundamental disconnect between content and language learning. Over 95% of the students are English Language Learners (ELLs), but less than 25% of these learners receive direct EAL support. Currently, this is determined by overall score below 5.0 of the WIDA Model.

Students are phased into appropriate English Language Acquisition classes according to WIDA Model screener tests, and ongoing performance in the class. However, the school has noticed a disparity between year end academic results of ELL who have scored 6.0 or below and non-ELLs.

To combat this, a new program is being introduced to the staff in September 2018 based on the SIOP® (Sheltered Instruction Observation Protocol,) Model. The SIOP® Model focuses on making content comprehensible for ELLs and has been developing over the past 20 years. Program developers Jana Echevarria, MaryEllen Vogt, and Deborah Short and their team devised a framework under which teachers are encouraged to organize their best teaching practices. (See Figure 1).

Figure 1: The SIOP® Model Framework for Organizing Best Practices
Echevarria, Jana, et al. Making Content Comprehensible for English Learners: the SIOP Model. 5th ed., (23) Pearson, 2017.

The SIOP® Model consists of 30 features that are to be incorporated in each lesson to ensure comprehensibility in content areas. The features are sorted into the following 8 components: Lesson Preparation, Building Background, Comprehensible Input, Strategies, Interaction, Practice & Application, Lesson Delivery, and Review & Assessment. The features are as follows:

Lesson Preparation
1. Content objectives clearly defined, displayed, and reviewed with students
2. Language objectives clearly defined, displayed, and reviewed with students
3. Content concepts appropriate for age and educational background level of students
4. Supplementary materials are used to a high degree, making the lesson clear and meaningful
5. Adaptation of content to all levels of student proficiency
6. Meaningful activities that integrate lesson concepts with language practice opportunities for reading, writing, listening, and/or speaking
Building Background
7. Concepts explicitly linked to students' background experiences
8. Links explicitly made between past learning and new concepts
9. Key vocabulary emphasized (e.g., introduced, written, repeated, and highlighted for students to see)
Comprehensible Input
10. Speech appropriate for students' proficiency levels
11. Clear explanation of academic tasks
12. A variety of techniques used to make content concepts clear
Strategies
13. Ample opportunities provided for students to use learning strategies
14. Scaffolding techniques consistently used, assisting and supporting student understanding
15. A variety of questions or tasks that promote higher-order thinking skills
Interaction
16. Frequent opportunities for interaction and discussion between teacher/student and among students, which
encourage elaborated responses about lesson concepts.
17. Grouping configurations support language and content objectives of the lesson
18. Sufficient wait time for student responses consistently provided
19. Ample opportunity for students to clarify key concepts in L1 as needed with aide, peer, or L1 text
Practice & Application
20. Hands-on materials and/or manipulatives provided for students to practice using new content knowledge in the
classroom
21. Activities provided for students to apply content and language knowledge
22. Activities integrate all language skills
Lesson Delivery
23. Content objectives clearly supported by lesson delivery
24. Language objectives clearly supported by lesson delivery
25. Students engaged approximately 90% to 100% of the period
26. Pacing of the lesson appropriate to students' ability levels
Review & Assessment
27. Comprehensive review of key vocabulary
28. Comprehensive review of key content concepts
29. Regular feedback provided to students on their output
30. Assessment of student comprehension and learning of all lesson objectives throughout the lesson.
Echevarria, Jana, et al. Making Content Comprehensible for English Learners: the SIOP Model. 5th ed., (23) Pearson, 2017.

In order for the students to better access content and achieve higher success, our school recognizes that all teachers must be language teachers. Initial teacher education and experience differ greatly between staff members, and must ensure that all teachers receive language training. We have chosen the SIOP® Model as the framework for our staff to use for a number of reasons. First, it will provide a common language for staff and students. Second, it outlines clear expectations for best practices. Third, in-house Professional Development training will provide staff with practical activities which they will be able to adapt for their lessons. Fourth, the specific features will aid in teacher observations and evaluations.

For more information on the SIOP® Model, please visit siop.pearson.com.

Purpose of the Evaluation

The purpose of this evaluation is to establish the impact of the new EAL program on accessibility of content for ELLs in content subject areas. As the program is in its infancy, it needs constant monitoring to improve implementation and management, ensure accountability of participants, and to plan for the future. Process evaluation will be used initially to drive the implementation of the program and will focus on the operations, implementation, and service delivery.

Some focused questions to consider include:
1. How is the program being implemented?
2. Are the human resources being used as intended?

What is the involvement of the EAL Coordinator and MYP Coordinator?
Have the TESOL Trainer and ELL Coach provided training as intended?
Have all EAL Teachers completed SIOP® Training for Teachers?
How many content teachers have completed SIOP® Training for Teachers?
Are students are receiving EAL support as intended?
In which aspects of the program is administration involved?
Have facility and maintenance workers prepared and maintained facilities as required?

3. Have facilities been used as intended?
4. How effectively are the program supplies being used?

How many teachers have accessed and utilized the Pearson SIOP® textbook?
How many teachers have accessed and utilized a SIOP® Model lesson plan?
Which monitoring tools are being used by coordinators, coaches, teachers, students, and administration?

5. Is the budget for program implementation appropriate?

What is the cost of running the program (including training and resources)?
How are costs divided between the MSHS and EAL budgets?

6. To what extent are program activities being implemented as intended?
SIOP® Model In-house PD Session:

How and when has feedback from trained staff members been received?
Which SIOP® features are teachers most comfortable using?
Which SIOP® features do teachers find most challenging to use?
How comfortable are content teachers teaching language objectives?

Comprehensive online SIOP® Model course:

How many staff members have completed SIOP® Training for Teachers (prior to August 2018)?
How many staff members have completed SIOP® Training for Teachers (post-August 2018)?
How many staff members are currently enrolled in SIOP® Training for Teachers?

EAL Support:

How many English Language Learners (ELL) attend the school?
How many ELLs currently receive EAL support?
In what capacity is EAL support currently serving students?
In what capacity is EAL support currently serving teachers?

Future outcome evaluation questions could include:

What are the year-end academic results of ELLs in the following subject areas: Arts (Drama, Music, Visual art), Design, Individual and Societies, Mathematics, Physical and Health Education, and Science? (for the past three-five years)
How many former EAL students (who have reached the WIDA score of 5.0 or above) entered the DP in the past three-five years?
How many former EAL students successfully completed the DP? (for the past three-five years)
Which SIOP® features do ELL students feel have helped make content more accessible?
How have non-ELL students benefited from the program?
What unforeseen outcomes have occurred as a result of implementing this program?

PROGRAM THEORY

If content-based teachers receive SIOP® Model training and are equipped with supporting resources, and if we provide ongoing, comprehensive SIOP® Model training to all EAL teachers, then they will be able to deliver content lessons that include the 30 SIOP® features. If teachers deliver SIOP®-based lessons, then content will be more comprehensible to English Language Learners (ELLs). If content is comprehensible to ELLs, then there will be an increase in academic achievement. If there is an increase in academic achievement, then the school will reach an IB score of 35 which will result in an increase of enrolment.

LOGIC APPROACH

This logic model approach outlines the inputs needed (what is invested) to run specific activities for the SIOP® Model program to be successful in the international school setting. Immediate and intermediate outcomes (what we want to happen) and their outputs (how they can be measured) are listed, as well as long-term outcomes and outputs.

EVALUATION APPROACH

To evaluate the merits of the implementation of the SIOP® Model program, I have decided to use Utilization-Focused Evaluation (U-FE), developed by Michael Quinn Patton in 1978. Although beneficiary stakeholders may prefer to use a Participatory Evaluation to focus on the impact of the program in quantifiable terms (i.e. financial gains), I would like to keep the focus of the evaluation on the usefulness to as many of the intended users as possible, including Administration, MYP and EAL Coordinators and EAL support teachers. Since the SIOP® Model program is in its infancy, both process and design evaluation are essential to ensuring that the program is a success. As U-FE is a "continual examination of and adaptation to how real people in the real world apply evaluation findings and how they experience the evaluation process," (Our Approach), we will be able to better adapt the implementation of the program by adjusting inputs and remodelling activities as necessary.
Challenges with U-FE to evaluate this particular program would include proper utilization of time and human resources. U-FE is a lengthy evaluation as it carefully considers how everything that is done, from beginning to end, will affect use (Our Approach). In addition, there is a possibility that intended users, while aiding the evaluator in designing the evaluation, may be narrow-minded in their approach as they may feel bias towards particular outcomes of the program due to external or internal political philosophies. Finally, it is advised to have "active and skilled guidance from and facilitation by an evaluation facilitator," (Utilization-Focused Evaluation). It may be difficult to find an experienced, skilled, and unbiased evaluator to facilitate the evaluation.

EVALUATION FOCUS

In order to design an appropriate evaluation of the SIOP® Model program in our international school, all intended users of the evaluation must first be determined and consulted. These include, but not limited to, all human resources needed to implement activities (EAL/MYP coordinators, TESOL trainer/ELL coach, EAL teachers, subject/content teachers, students, administration, facility and maintenance workers), as well as stakeholders (CEO and Board members). The intended use of the evaluation for each user or group of users needs to be identified to properly frame survey questions and determine alternate methods of collecting data. In addition, a clear timeframe for the implementation of the SIOP® Model program and evaluation process needs to be determined.

DATA COLLECTION METHODS

There is a wide range of sources and methods with which to gather information pertinent to this evaluation. These methods will respond to the evaluation questions as determined by the intended users and stakeholders, in consultation with the evaluator. The following tables outline initial chosen collection methods and sources:

To properly analyze collected data, evaluation objectives and the needs of the beneficiaries must be considered. As a plethora of data will be gathered, a number of analysis procedures and devices will be required:

Quantitative Data:
Date must be reviewed, interpreted and summarized. The following procedures will be considered:
Effect parameter analysis - to analyze changes before and after the implementation of the program
Cost analysis - to analyze financial costs associated with the program
Trend analysis - to analyze academic changes
Qualitative Data:
Data must be reviewed, organized, coded and interpreted. The following procedures will be considered:
Content analysis - to analyze survey results
Summaries - to analyze interviews and survey results
In addition, qualitative information will be used to elaborate upon quantitive findings.

Quantitative and qualitative will be compared and contrasted to analyze findings that are corroborative or contradictory. Should contradictory information be found, an analysis of the limitations of the program must occur. After analysis is complete, an action plan will be created to address the needs of identified problems. A follow-up evaluation is recommended to assess this course of action.

APPROACH TO EVALUATION ENHANCEMENT

To garner support and enhance the evaluation, timely distribution of the report is critical. Reports should be easy to comprehend, succinct enough to draw interest, and yet detailed enough to sustain enthusiasm. In order to report to the intended users of the evaluation and to provide feedback to those who participated in data collection, consideration must be made towards the language and style of report. As many of the users are native Chinese speakers, all reports will be made available in Mandarin and English.
Audiences of different user groups may receive the findings in a different format depending on the intended use of the findings. For example, board members would receive summaries of Program Antecedents and Program Implementation reports, but a more detailed report on the analysis of the Program Results. They would receive printed and electronic copies of the report, that focus on the fiscal analysis, enrolment levels, and overall academic achievement.
On the other hand, teachers would receive electronic reports of the three subreports (Program Antecedents, Program Implementation, and Program Results) that focus more on qualitative data summarizing issues that became evident during the evaluation and how those issues were addressed (Stufflebeam, 4).
A final example would be the manner in which students received the report. An oral presentation in student-friendly language presented in an assembly would be an ideal reporting strategy for this audience. For every method used, each report should conclude with appropriate recommendations.

COMMITMENT TO STANDARDS OF PRACTICE

This evaluation intends to meet the Standards for Program Evaluation in the following ways:
Utility Standards
This evaluation will serve the needs of the intended users as outlined above.
At the onset of the evaluation, all stakeholders will be identified and their needs and values identified. A trustworthy and competent evaluator will be chosen to oversee the evaluation. Data will be collected in response to the needs and interests of the SIOP® Model program users and stakeholders. Reports will be clear and outline the context, purpose, procedure and findings of the evaluation in a timely manner. Follow-through will by stakeholders will be encouraged through appropriate recommendations.
Feasibility Standards
This evaluation will be realistic, prudent, diplomatic, and frugal.
The design and conduct of the evaluation will consider the interests of various interest groups. Procedures will be kept practical, avoiding disruption, and cost effective; expended resources will be justified.
Propriety Standards
This evaluation will be conducted legally, ethically and with due regard to those involved in the evaluation and those affected by the results.
The SIOP® Model program evaluation will be designed to assist my school to address the needs of our ELLs. Obligations of those involved with the evaluation will be agreed upon in writing. Respect towards the rights, welfare, and dignity of all participants will be upheld. The evaluation will be complete and fair to ensure that all strengths and weaknesses of the SIOP® Model program are addressed. All conflicts of interest will be dealt with openly and honestly to ensure the evaluation is not compromised. All expenditures will be accounted for and will reflect accountability. The evaluation findings will be presented or made accessible to all users and persons affected by the evaluation.
Accuracy Standards
This evaluation will be reveal and convey adequate information that determine the worth or merit of the SIOP® Model program.
The SIOP® Model program evaluation will be designed and documented clearly and accurately, with the program and context can be clearly determined. The purpose, procedures, and sources of information as outlined above will be described in detail so that they are also easily noted. Procedures used to gather information will be developed and implemented so that information can be gathered, analyzed, reported, and reviewed systematically to ensure reliability and errors can be corrected in a timely manner. Quantitative and qualitative data will be appropriately analyzed to ensure that the evaluation questions have been answered efficaciously. All conclusions will be justified and reports will remain impartial. Overall, the evaluation of the SIOP® Model program will be evaluated in formative and summative manners following the Standards for Program Evaluation so that the strengths and weaknesses of the program can be appropriately addressed.

(Adapted from: Standards for Program Evaluation, 1-3)

RESOURCES

“AEA365 | A Tip-a-Day by and for Evaluators.” AEA365 | A Tip-a-Day by and for Evaluators, American Evaluation Association,
aea365.org/blog/.
Alkin, M.C., & Taut, S. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1-12.
“Analyzing Qualitative Data for Evaluation.” Adolescent and School Health: Program Evaluation, CDC, April 2009,
www.cdc.gov/healthyyouth/evaluation/pdf/brief19.pdf.
“Analyzing Quantitative Data for Evaluation.” Adolescent and School Health: Program Evaluation, CDC, July 2009,
www.cdc.gov/healthyyouth/evaluation/pdf/brief20.pdf.
Australian Institute of Management, 8 Nov. 2017, www.aim.com.au/blog/how-embed-purpose-your-organisation.
"Disseminating Program Achievements and Evaluation Findings to Garner Support .” Adolescent and School Health:
Program Evaluation, CDC, February 2009, www.cdc.gov/healthyyouth/evaluation/pdf/brief9.pdf.
Families Commission, Government of New Zealand (2017). Making sense of evaluation: A handbook for everyone. Retrieved
from: http://www.superu.govt.nz/sites/default/files/Publications/Evaluation%20Handbook%20Dec%202017.pdf
Huey-Tsyh, C. (2005). Practical program evaluation: Assessing and improving planning, implementation, and
effectiveness. Thousand Oaks, CA: Sage Publications.
Key Evaluation Checklist:
https://www.wmich.edu/sites/default/files/attachments/u350/2014/key%20evaluation%20checklist.pdf
Kirkhart, K.E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. In V. Caracelli and H. Preskill
(Eds.), The expanding scope of evaluation use. New Directions for Evaluation, 88 (pp.5-23). San Francisco; Jossey-Bass.
“Our Approach.” Utilizaton-Focused Evaluation, www.utilization-focusedevaluation.org/our-approach/.
Patton, Michael Quinn, director. Utliization-Focused Evaluation. YouTube, MyM AndE, 7 June 2013,
www.youtube.com/watch?v=jQP1FGhxloY.
Patton, Michael Quinn. Utilization-Focused Evaluation (U-FE) Checklist. Western Michigan University, Jan. 2013,
wmich.edu/sites/default/files/attachments/u350/2014/UFE_checklist_2013.pdf.
Saunders, M. (2012). The use and usability of evaluation outputs: A social practical approach. Evaluation, 18(4), 421-436.
Scriven, Michael. “Key Evaluation Checklist.” Key Evaluation Checklist, Western Michigan University, Feb. 2007,
www.wmich.edu/sites/default/files/attachments/u350/2014/key%20evaluation%20checklist.pdf.
Shulha, L., & Cousins, B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-
208.
Standards for Program Evaluation: http://www.oecd.org/dev/pgd/38406354.pdf. Retrieved from: The Joint Committee on Standards for Educational Evaluation, James R. Sanders, Chair (ed.): The Program Evaluation Standards, 2nd edition. Sage Publication, Thousand Oaks, USA, p.23-24; 63; 81-82,125-126 (see www.wmich.edu/evalctr/jc/)
Stufflebeam, Daniel (2017) Evaluation Design Checklist. The Evaluation Center. Western Michigan University, p.1-5.
Retrieved from: https://www.wmich.edu/sites/default/files/attachments/u350/2014/evaldesign.pdf
Surbhi, S. “Evaluation.” Difference Between Assessment and Evaluation, Key Differences, 7 July 2016,
keydifferences.com/difference-between-assessment-and-evaluation.html.
Taut, S. (2007). Methodological and conceptual challenges in studying evaluation process use. Canadian Journal of
Program Evaluation, 22(2), 1-20.
“The SIOP® Model Helps Every Teacher Become a Language Teacher.” Environmental Science Program | Pearson High
School Science Curriculum, Pearson Education, Inc. , 2018, www.pearsonschool.com/index.cfm?locator=PS2rWo.
“Types of Evaluation.” YouTube, PACE MySPH, 11 Dec. 2012, www.youtube.com/watch?v=IgwvRiGm424.
“Utilization-Focused Evaluation.” Impact Evaluation | Better Evaluation,
www.betterevaluation.org/plan/approach/utilization_focused_evaluation.
Weiss, C.H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19, 21-33.
“What Is Evaluation?” YouTube, PACE MySPH, 11 Dec. 2012, www.youtube.com/watch?=Qk8SGU2iWWg.

PHOTO and image CREDITS

“Approach.” Approach, Agnihotri Immigration Consultant , 2018, www.agnihotriimmigration.com/company-overview/our-
approach/approach-2/.
Clarke, Susan. “Word-Of-the-Week #619: Enhance.” Word-Of-the-Week #619: Enhance, FUN-Damentals, 16 June 2016, fun-
damentals.com/tag/enhance/.
“Data Collection.” GDPR: Data Collection, DMA, 16 Mar. 2016, dma.org.uk/article/gdpr-data-collection.
“Handshake.” Is Commitment a Good Thing or Not?, Coral, 21 Sept. 2001, www.mindjet.com/blog/2012/09/is-commitment-
a-good-thing-or-not/.
Hart, Carrie. “ Welcome Honoring Yourself through the Opening Door (i-Am-This.com).” Welcome Honoring Yourself through
the Opening Door (i-Am-This.com), 24 Apr. 2015, i-am-this.com/honoring-yourself-through-the-opening-door-i-am-this-
com/.
"Purpose." Australian Institute of Management, 8 Nov. 2017, www.aim.com.au/blog/how-embed-purpose-your-
organisation.
“Theory.” Theory of Knowledge IB Guide | Part 4, Lanterna Education, www.lanternaeducation.com/ib-blog/theory-of-
knowledge-ib-guide-part-4/. Retrieved from: www.google.com/hk.
“The SIOP® Model Helps Every Teacher Become a Language Teacher.” Environmental Science Program | Pearson High
School Science Curriculum, Pearson Education, Inc. , 2018, www.pearsonschool.com/index.cfm?locator=PS2rWo.
Uwaoma, Eizu. “The Law of Focus.” The Law of Focus, Hexavian Business Club (HBC), www.hbc.org.ng/the-law-of-focus/.

HOME PME 800 CONTACT