Common Questions about Value-Added Modeling

May 2014

Introduction

This document provides a non-technical overview of value-added modeling. Because this is the beginning of a pilot year for the Texas Teacher Evaluation and Support System (T-TESS), the overview provides some generalized information about value-added modeling and decisions that are subject to change based on further analysis and validation.

Common Questions

1. What is a value-added model?
Value-added modeling looks at how much value is added to students’ learning over the course of the year. In other words, value-added modeling looks at how much academic progress students make within a tested grade or subject (i.e. science, reading, math, etc.) over time.

2. Why is value-added important? Why is it important to measure growth?
Education policy has historically focused on proficiency, which had an unintended consequence of focusing on students who are on the cusp of proficient scores on state assessments rather than all students. However, we believe that all students are important, that all students can learn, and that educators should focus on all students. When we focus on growth, we focus on all students.

3. How does a value-added model work? Why is value-added important? Why is it important to measure growth?
A value-added model measures how much progress a group of students made relative to the expectation of progress. This group is defined as all the students with a particular district, school or teacher in a specific subject, grade and year. The expectation of progress is defined by the individual students within the group and how other students just like them performed across the state on average.

In other words, we follow each student over time and that student serves as his or her own control. The student’s expectation of growth is defined by how similar students performed across the state.

A value-added model then compares each measure of growth to the expectation of growth and uses statistical precision to see if there is evidence that group made more than, less than, or about the same progress as expected.

4. How is a value-added reporting system useful?

The information available from value-added modeling can help educators in a multitude of ways, including the following:

• Provides a robust tool for educators to reflect on their practices and have a positive impact on all students’ achievement.
• Identify strengths and weaknesses of curriculum or instructional practices in specific grades and subjects (i.e. science, reading, math, etc.).
• Detect the effectiveness of programs with special student populations.
• Pinpoint incoming students’ academic needs before those students set foot in your classroom.

5. What are the limitations of value-added modeling?

Value-added models measure teaching effectiveness through student performance on tests, which is only one aspect of effective teaching. For that reason, the T-TESS includes other aspects of effective teaching, and there is local flexibility for some of these measures.

That said, recent studies show that value-added modeling also captures the long-term effects that teachers have on their students’ future earnings, retirement savings, and educational attainment (Source: Chetty et al., The Long-Term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood).

6. What if my student has a bad test day? Does value-added modeling hold me accountable based on one test given on one day?

Value-added modeling is not about one student on one day. It’s about looking at the growth of an entire group of students over time.

In fact, value-added modeling does not provide growth measures for individual students. It only provides a growth measure for groups of students. Growth models look well beyond what students do on just one day by looking for a pattern across multiple years of growth estimates and multiple years of student test scores to see whether students, on average, made expected growth.

7. With so much emphasis on standardized testing, does value-added modeling encourage teaching to the test?

No, this is not about teaching to the test. Effective teaching is about helping each student make academic progress regardless of his/her entering achievement level, and that is what value-added modeling captures. By teaching a narrowed curriculum, very low and high achieving students will have limited opportunities to make appropriate academic growth. In this case, the teacher or school may actually be less likely to be highly effective from a value-added perspective.

8. If I’m worried about the quality of the state assessments, which is the basis of value-added modeling, should I also be worried about the quality of the value-added results?

All assessments used in the value-added modeling are reviewed and analyzed to ensure that they are appropriate for value-added analysis. These protections are put in place to ensure that valid and reliable results are provided to educators to make accurately informed decisions.

9. Value-added modeling is just a number, so how can it help me improve my instruction or tell me what to do with my students?

The value-added reporting is part of a comprehensive web application, which provides teachers with interactive diagnostic data for additional insight, school/district reporting for context, and student-level projections for proactive planning with current or future students. This information, combined with teacher knowledge of their own students and schools, can greatly assist in school improvement.

10. Is value-added modeling fair even though my students are…

11. How does value-added modeling accommodate Spanish test takers or first-time English test takers?

TxVAAS can accommodate Spanish versions of the STAAR test. Spanish test scores are converted to the English scale so that it is as if the students have been testing in English all along. This ensures that there is no drop in scores when the student takes the English test for the first time.

12. Value-added modeling seems like a black box; how do I know it is accurate?

You can rest assured that the TxVAAS modeling will use standard, vetted statistical methods to produce valid and reliable results. Non-partisan, independent experts in value-added analysis have replicated models similar to the TxVAAS approach over the years, and their most recent findings suggest that these models are among the most reliable, least biased, and most suitable for teacher value-added modeling (Source: McCaffrey, D.F. and J.R. Lockwood (2008). Value-Added Models: Analytic Issues. Presented at the Workshop on Value-Added Modeling sponsored by the National Research Council and the National Academy of Education, Board on Testing and Accountability, Washington, DC, Nov. 13-14, 2008).

13. Is value-added modeling reliable?

Yes. Independent researchers have noted that the reliability of value-added modeling is similar to what is used elsewhere for high-stakes decision-making (Source: Evaluating Teachers: The Important Role of Value- Added). Furthermore, the growth rating does not rely on rankings of teacher estimates, which can be rather unstable and uninformative; rather, the growth rating uses standard errors to determine effectiveness levels.

14. Value-added modeling is just an estimate, so how can I be held accountable for an estimate?

The growth rating takes this into account. The growth rating is based on the value-added estimate as well as its statistical precision (known as standard error). This means there must be significant evidence that students have made more than or less than the expected progress before receiving the highest or lowest ratings. Requiring significant evidence protects teachers from misclassification (identifying an effective teacher as ineffective or vice versa).

15. How can I have a low value-added rating when all my students passed the test?

Value-added is not about students passing or failing. This is about how much progress students made. Value-added modeling is based on growth, not proficiency or achievement. The growth expectation for each student is based on each student’s own previous achievement. Regardless of whether students are proficient or not, the value- added estimates whether, on average, students met the growth expectation for a class or school.

16. Will tested teachers be held accountable differently than non-tested teachers?

There is different data available for teachers that have value-added scores than for teachers who don’t have scores. Both groups of teachers, however, will use measures of student growth as a part of their evaluation process.
The purpose of this data, whether value-added data or alternative growth measures, is to provide teachers and appraisers with information that captures how students progressed during their time with a particular teacher, whether or not there are students that tended to progress more than others, and, most importantly, to provide insight into how educators could improve instructionally to better reach all students in the future.

17. How can value-added modeling use TAKS data now that we have STAAR?

TxVAAS can accommodate tests on different scales. It is not required for the prior test scores to be on the same scale as the current test. In the same way that height can predict weight, even though they are based on two different metrics, TAKS scores can be used to assess growth on STAAR tests. The important thing is that the prior TAKS scores have some predictive relationship with the STAAR test and how the students’ place in the statewide distribution have changed from one year to the next.

18. How does value-added modeling accommodate team teaching?

Teachers in tested subjects will have the opportunity to enter the percentage of instructional responsibility they had for each student. This is incorporated into the model to account for scenarios where more than one instructor had responsibility for a student’s learning in a given subject/grade.

19. How will I learn more about value-added modeling? How do I get support about value-added modeling?

Day 3 of appraiser training will include a module on value-added modeling. In addition, TEA will provide resources and guidance on its website related to the use of value-added modeling in the Texas Evaluation and Support System. Also, any teacher can submit technical or reporting questions directly to the value-added provider through the web application.

20. How is TxVAAS different from Index 2?

Both systems aim to provide a measure of student progress independent of student achievement. However, because their uses are different, the modeling is different.
Index 2 provides a measure of student progress for district and school accountability. It is an indicator of whether each individual student has met a particular growth target. Very low-achieving students are automatically considered “Did Not Meet Progress” and very high-achieving students are automatically considered “Exceeded Progress.” The individual student-level information is aggregated to provide points and a subsequent score to a district or school.
TxVAAS is a student progress measure for teacher and principal evaluation. It is an indicator of whether, on average, there is sufficient evidence to determine that students linked to a particular teacher or school made more than, less than or about the expected progress. The growth measure takes into account all students’ growth over time, even if they are very low or very high achieving.