NYSAPE Objects to PermaNent Adoption of APPR Regulations
August 23, 2015
Commissioner of Education Mary Ellen Elia
Board of Regents
NYS Education Department Building
89 Washington Street
Albany, NY 12234
Re: 3012-d Public Comment EDU-27-15-00019-P
Dear Commissioner Elia and Board of Regents:
Please accept this correspondence as NYS Allies for Public Education’s (NYSAPE) formal submission of comments in opposition to the proposed rule-making entitled EDU-27-15-00019-P, published in the NY Register on July 8, 2015. It is anticipated that the proposed rule will be presented for adoption at the Regents September 16 & 17, 2015 meeting. NYSAPE opposes permanent adoption of the APPR regulation and submits this paper in objection to permanent adoption of the proposed regulations.
NYSAPE Objects to Permanent Adoption of Proposed APPR Scheme
NYSAPE, an organization that consists of over fifty parent and educator advocacy groups across New York State, firmly believes in the power of public education and its fundamental link to the success of a thriving community and democratic government.
NYSAPE has long advocated for public education. We believe that while meaningful assessment is an essential component of a world class education, the NYS standardized Common Core assessments are aligned with unproven reforms that are neither supported by rigorous research nor vetted by educators and parents, and are harmful to children.
As leaders of an advocacy movement who demand an immediate end to the overemphasis on harmful testing practices, we assert that the proposed APPR regulations, EDU-27-15-00019-P, should be rejected.
Parents have become increasingly frustrated with the alarming changes happening to their children and their education. Concerns lodged by parents to State and education officials about the devastating impact that tying test scores to teacher and principal evaluations have had in public schools have been ignored and rebuffed. As a result, these parents exercised their parental rights to have more than 220,000 children refuse the 2015 state assessments, and “teachers of conscience” bravely defended their profession from flawed, harmful testing practices and bad policies sanctioned by the State.
While SED takes the position that state assessments must be tied to teacher evaluations for accountability purposes, we believe that New York State’s APPR scheme is neither a reliable nor accurate way to assess New York State teachers. In its current form, the teacher and principal evaluation scheme subjects New York State public school children to an egregiously flawed testing regime that not only harms public school children and teachers but threatens the very fabric of our public schools.
We urge the Board of Regents to reject the APPR regulations proposed for permanent adoption and to vote “No” at the September meeting. We respectfully request that the Regents utilize all avenues within their power to demand that both the Legislature and the Governor amend the education law and decouple test scores from teacher and principal evaluations.
The APPR Regulation Proposed for Permanent Adoption is Patently Flawed and Should be Rejected
As part of the 2015-16 budget bill as it relates to education, the State increased the impact of the value-added measure (VAM) on teacher and principal evaluations to a whopping 50% (and, in some cases, more than 50%). This has raised grave concerns about the impact that this policy will have on teacher and principal effectiveness outcomes as tied to student assessments. For that reason, and among others, the NYSAPE Board recently met with Commissioner Mary Ellen Elia to discuss our concerns relating to testing and evaluations.
Dr. Carol Burris, the recently retired principal of South Side High School in Rockville Centre, participated in that meeting, held August 4, 2015, between the NYSAPE Board and Commissioner Elia. Dr. Burris recounted the ratings of teachers at very low-performing schools in Buffalo in order to demonstrate that many of the teachers in those schools had received high student growth ratings. Then she recounted the student growth ratings of teachers at the high-performing Scarsdale public school and demonstrated that, by contrast, an extraordinary proportion were rated “ineffective.” Commissioner Elia agreed that these results made no sense. In fact, Commissioner Elia went on to concede that the facts Dr. Burris brought to her attention are utterly “indefensible.”
This “indefensible” phenomena is easily explained by the unintended negative consequences attendant to using student assessment scores to predict student growth for the purpose of teacher evaluations -- it produces unpredictable and irrational variability in student growth scores. Indeed, the VAM-based teacher and principal evaluation scheme that is being proposed for permanent adoption at the September Regents meeting is an “indefensible” scheme in and of itself, one that greatly prejudices teachers and principals and, ultimately, the students in their charge.
Trusted assessment and education policy experts agree that there is danger in using VAM for high-stakes decisions about teachers’ jobs and pay. The American Statistical Association issued a report in 2014 on VAM and said: “VAMs are generally based on standardized test scores and do not directly measure potential teacher contributions toward other student outcomes.”
Indeed, the use of student test scores to evaluate teachers using VAM methodology has lead to some preposterous results. In New York City, an art teacher was evaluated based on math standardized test scores and saw his evaluation rating drop from “effective” to “developing.”
Meanwhile, teachers in E.M. Baker School received a score of “ineffective,” 21 percent “developing” and 57 percent were “effective.” Just the year before, not one teacher received an “ineffective” score.
In 2013, 26 percent of Rochester teachers were rated as “ineffective” based on student performance on the state tests. Just one year later, the number of ineffective teachers dropped to 4 percent! In the same year, the percent of teachers in Yonkers who got “ineffective” growth scores fell from 18 percent to 5 percent.
In the span of one year, the percentage of teachers with “ineffective” student growth scores in top-performing Scarsdale went from 0 percent to 19 percent, and it jumped from 0 percent to 13 percent in affluent Roslyn on Long Island. Those “ineffective” ratings have even slipped into top-performing and affluent Jericho, New York, where 8 percent of those teachers are now considered “ineffective,” notwithstanding that 81 percent of Jericho students were proficient on the Common Core math tests, far exceeding and more than doubling the State average of 35 percent.
There are also disparities in the scores of “highly effective” teachers. In 2014, 10 percent of the teachers in Brooklyn (Kings County) got “highly effective” state-generated growth scores. By astounding and absurd contrast, not one teacher in Roslyn or Scarsdale did. Yet one year prior, 13 percent of Scarsdale teachers got “highly effective” scores from NYSED.
Also in 2014, 44 percent of the teachers in the Fox Meadow School in Scarsdale received growth scores that said they were not “effective” teachers, with 22 percent rated “ineffective.” Yet 61 percent of the school’s students were proficient in English Language Arts, and 75 percent were proficient in math—more than double the State’s proficiency rate.
Similar results were found at the high-achieving Harbour Hill Elementary School in Roslyn, where 36 percent of its teachers received growth scores that labeled them “ineffective.”
The Lincoln School in Rochester is a school designated as a priority/failing school by the State. Its proficiency rate on the State assessments was less than 3%. In 2014, 100% of its teachers received “effective” state scores, with 7 percent being rated “highly effective.”
At the Martin Luther King, Jr. School in Utica, NY, a school facing receivership, 60 percent of the teachers received “effective” VAM scores and 40 percent were given VAM scores of “highly effective.”
The dramatic differences cited above aptly illustrate how using the VAM formula to evaluate teachers and principals manifests in irrational state-produced teacher scores. To boot, these irrational evaluation scores are based on the flawed New York Common Core tests. Such disparate outcomes should come as no surprise to SED, the Regents or State officials. Researchers and experts have long cautioned that the VAM methodology is an inaccurate and flawed measure of individual teacher effectiveness and, accordingly, have long cautioned against using VAM to measure individual teacher effectiveness.
The American Education Research Association, and the National Academy of Education, both addressed these flaws, stating that the flaws compromise the teacher evaluation process so that the formula is rendered unstable and virtually irrational. Read more about that here.
Year-to-year volatility of the scores is also well established. The National Academy of Sciences (NAS) and the American Statistical Association have both stated that the VAM methodology is not reliable enough to be used as a major component of any teacher evaluation system:
“The report strongly supports rigorous evaluations of programs funded by the Race to the Top initiative. The initiative should support research based on data that links student test scores with their teachers, but should not prematurely promote the use of value-added approaches, which evaluate teachers based on gains in their students' performance, to reward or punish teachers. The report also cautions against using the National Assessment of Educational Progress, a federal assessment that helps measure overall U.S. progress in education, to evaluate programs funded by the Race to the Top initiative.”
Likewise, the American Statistical Association (ASA) opined, in relevant part:
“VAMs should be viewed within the context of quality improvement, which distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools. Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.”
As the ASA points out, using VAM scores to improve education requires that they “provide meaningful information about a teacher’s ability to promote student learning” but this is accomplished only in districts in which “VAMs are used for low-stakes purposes.” (emphasis added).
In fact, the ASA strongly cautions against the use of VAM for high stakes policy and decision making:
“It is unknown how full implementation of an accountability system incorporating test-based indicators, such as those derived from VAMs, will affect the actions and dispositions of teachers, principals and other educators. Perceptions of transparency, fairness and credibility will be crucial in determining the degree of success of the system as a whole in achieving its goals of improving the quality of teaching. Given the unpredictability of such complex interacting forces, it is difficult to anticipate how the education system as a whole will be affected and how the educator labor market will respond.“
Professional consensus about the use of VAM in evaluating teachers and principals is that VAM is inappropriate as a primary measure for evaluating individual teachers. It stands to reason, raising the percentage of APPR plan to 50% is irrational and patently absurd. As the evidence clearly shows, the current formula has yielded a huge and irrational disparity across schools and districts; of the gain scores in the current formula which demonstrates that the proposed APPR regulation slated for permanent adoption is utterly inappropriate.
Substantive Argument Against Permanent Adoption of APPR
The Regents are about to permanently adopt a VAM methodology in which student test scores will count for 50% (or more, depending on the matrix) of an individual teacher’s or principal’s overall rating. According to the comprehensive text of the proposed rule to be promulgated by the New York State Education Department (see memo from Ken Wagner, dated June 15, 2015, to the P-12 Committee), the methodology that the New York State Education Department proposes to use to measure the student growth component of individual teacher and principal evaluations is so scientifically flawed and unsound as to be unreasonable and unreliable. It produces unintended negative consequences that will assuredly jeopardize -- and possibly end -- the careers of many outstanding and competent New York State teachers, as well as harm the learning experiences of millions of New York State school children. The State Education Department has an affirmative duty to the children of the State of New York to prevent detrimental measures such as this from being promulgated; this proposed rule must not be allowed to pass.
In support of this assertion, we refer you to all of the evidentiary expert affidavits that were filed on behalf of the Petitioner in the case of Dr. Sheri Lederman v. John B. King, Jr., et al. (Index No. 5443-14). Initial affidavits of experts Dr. Audrey Amrein-Beardsley, Dr. Carol Burris, Dr. Linda Darling-Hammond, Dr. Brad Lindell, and Dr. Aaron Pallas can be found here.
Reply affidavits of Dr. Aaron Pallas, Dr. Brad Lindell, Dr. Audrey Amrein-Beardsley, Dr. Carol Burris, and Dr. Linda Darling-Hammond; as well as initial affidavits of Dr. Jesse Rothstein and Dr. Sean Patrick Corcoran; can be found here.
Given that officers of the New York State Education Department were direct parties to this litigation, NYSED in general is already on constructive if not actual notice of the entire contents of these various affidavits -- all of which were uncontroverted by NYSED during the matter. We hereby incorporate into this comment for your review and response each and every affidavit referenced above, in their respective entireties, especially for their discussion of the student growth measure currently in use and being proposed by State Ed.
In addition, we submit a further explanation of the affidavit that was submitted by psychologist Dr. Brad Lindell, as follows:
"I am Dr. Brad Lindell, one of the affiants in the Sheri Lederman case who was present at the oral arguments on Wednesday. It was truly something to observe. You got the feeling that good was was going to come from the great work of Sheri and Bruce Lederman and from the experts’ opinions in so far as changing this broken VAM system. You got the sense that the judge was listening to the science about VAM and not just to the political rhetoric.
Just want to fill you in on something that was presented in my affidavit modified to give a clear and understandable example of the effects of poor reliability on a full-scale WISC intelligence test. If the same test-retest reliability from the teacher assigned yearly VAM scores (.40) was applied to the WISC full-scale to determine the 90% confidence interval, the range would be ridiculously large.
Examples
If a student scored a full-scale IQ of 100 (average) then the 90% confidence interval would be an 81 to 119. This indicates that there would be a wide range where the scores from repeated administrations of the WISC would be expected to fall for this student. One could not have confidence in the validity of a intelligence test with low reliability. Without adequate reliability, there can not be validity. This same holds true for VAM scores, whose reliabilities have been found to be notorious low.
The reliability of the WISC is generally in the .80 to .90 range. The 90% confidence intervals are generally in the +\- 6 range. So this same person with a 100 full-scale IQ would have a 90% confidence range of 94-106. Quite a smaller range.
This is why reliability is so important, which has repeatedly been shown to be low like .2 to .4 for year-to-year VAM scores. This is also why teachers year to year VAM score vary so considerably, like in the case of Sheri Lederman. Without reliability there cannot be adequate validity."
Dr. Lindell's explanation can be found here.
Procedural Arguments Against Permanent Adoption of APPR
The Notice of Proposed Rulemaking is fatally defective, as it fails to identify the underlying science and research to support the rules. NY State's Administrative Procedure Act (SAPA) requires that the "Needs and Benefits" section of the Notice's mandatory "Regulatory Impact Statement" specifically identify "a citation for and summary ... of each scientific or statistical study, report or analysis that served as the basis for the rule, an explanation of how it was used to determine the necessity for and benefits derived from the rule and the name of the person that produced each study, report or analysis." SAPA section 202-a(3)(b). (With this information, the public can comment in a meaningful way, with an understanding of what underlying information supports (or doesn't) the proposed rule.)
But the Notice of Proposed Rulemaking glosses over this critical requirement; the Notice's response to the "Needs and Benefits" section admits that expert input is required by law -- but then fails entirely to identify any study, report or analysis, or the producers of those reports or the citations, all in violation of the law. The Notice says only that "The statute requires the Commissioner to adopt regulations necessary to implement the new evaluation system for teachers and principals by June 30, 2015, after consulting with experts and practitioners in the fields of education, economics and psychometrics." New York Register, July 8, 2015, page 24.
Not only is this omission in plain violation of the law -- depriving the public of its statutory right to give meaningful comment -- but the omission goes to the heart of the public's concerns about the rules. The public has been deeply troubled by the apparent arbitrariness and lack of science in prior APPR plans as well as in the new one, so much so that over 25,000 New Yorkers, including our state's most respected educators, signed a petition on this point.
The need for the science and research is also imperative -- and legally required -- as some of the most controversial elements of the rules were not decided by the legislature, but instead were specifically delegated by the legislature to SED and the BOR, and their materials are essential to the rules' validity, as well as for public comment.
For example, the ETA states that "The Commissioner shall determine the weights and scoring ranges for the subcomponent or subcomponents of the student performance category that shall result in a combined category rating. The Commissioner shall also set parameters for appropriate targets for student growth for both subcomponents ...." (ETA p. 128)
SED's proposed rule includes these scoring ranges and weights, detailing what percentage of students meeting targets will equate with a particular score of 1-20. SED's proposed rules also set forth the cut scores laying out out what numbers from the 1-20 score range equate with each of the 4 rating categories, i.e. highly effective, effective, developing and ineffective. For example, to receive a rating of effective, at least 15 out of 20 points are required.
See Proposed Rule at p. 33-34. Note that some scores have significantly wider bands than others, e.g., the scores of 13 and 14, which would not yield a positive rating, have wide bands, of 6 and 7 points, but higher scores, like 18 and 19 have bands as narrow as 2 and 3 points. Id. And yet, SED fails to provide any support whatsoever to corroborate these scoring ranges and weights. The public cannot possibly submit meaningful comment on this most important aspect of the rule, delegated to SED and BOR, without being alerted as to what supports these figures and cut scores. As such, the rules should be withdrawn, and if desired, a new Notice can be issued, that meets all of the law's requirements, and a new 45-day public comment period can commence.
Alternatively, it may be that in fact there is no support in science or research for these ranges and weights. Should that be the case, then the proposed rules should withdrawn for having no supportable basis. "Standards for Educational and Psychological Testing," ("Standards") considered the gold standard in testing issues, and even referred to specifically in the proposed rules (see p.25) could not be clearer about the importance of detailed research to support cut scores: "Cut scores embody value judgments as well as technical and empirical considerations. Where the results of the standard-setting process have highly significant consequences, those involved in the standard-setting process should be concerned that the process by which cut scores are determined be clearly documented and that it be defensible." (p.101, 2014 edition)
In addition, the proposed rules set the division of weight if two tests are used for the student growth component at 50%/50%. In the original emergency rule, discussed and adopted at the June 2015 BOR meeting, the ratio was 80% state test and 20% local. Following discussion, the weights were changed to 50%/50%. (See proposed rules, p. 33) But the discussion on whether to use the new division included no references to studies or analyses supporting or dismissing either set of numbers.
Again, "Standards" is spot on: "when evaluations or accountability systems rely on measures that combine various sources of information, such as when scores on multiple forms of a test are combined ... the rules for combining the information need to be made explicit and must be justified. It is important to recognize that when multiple sources of data are collapsed into a single composite score or rating, the weights and distributional characteristics of the sources will affect the distribution of the composite scores. The effects of the weighting and distributional characteristics on the composite score should be investigated." (p. 210, 2014 edition)
Because of the failure to identify any science, research, analysis, or report, the proposed rule must be revoked. The failure is against the law; as stated above, SAPA specifically requires this information, and SED chose to ignore that law.
The failure also impedes democracy, as the public cannot meaningfully comment without the required information. The public comment is mandatory because the rule makers were appointed, not elected, and the comment is the only input that the public has on these rules.
The failure also may indicate that in fact there is no support for these rules, that no science or research supports this. If this is the case, then the rules must be revoked on that basis.
For the reasons set forth more fully herein and above, we urge the Board of Regents to reject the APPR regulations in its current form at the September meeting, refuse to permanently adopt the same and to utilize all avenues within their power to demand that both the Legislature and the Governor amend the education law and decouple test scores from teacher and principal evaluations.
Respectfully Submitted,
NYS Allies for Public Education
By: APPR Public Comment Subcommittee Members,
Anna Shah, Esq.
Deborah Abramson Brooks, Esq.
cc: Assembly Education Committee Members
Commissioner of Education Mary Ellen Elia
Board of Regents
NYS Education Department Building
89 Washington Street
Albany, NY 12234
Re: 3012-d Public Comment EDU-27-15-00019-P
Dear Commissioner Elia and Board of Regents:
Please accept this correspondence as NYS Allies for Public Education’s (NYSAPE) formal submission of comments in opposition to the proposed rule-making entitled EDU-27-15-00019-P, published in the NY Register on July 8, 2015. It is anticipated that the proposed rule will be presented for adoption at the Regents September 16 & 17, 2015 meeting. NYSAPE opposes permanent adoption of the APPR regulation and submits this paper in objection to permanent adoption of the proposed regulations.
NYSAPE Objects to Permanent Adoption of Proposed APPR Scheme
NYSAPE, an organization that consists of over fifty parent and educator advocacy groups across New York State, firmly believes in the power of public education and its fundamental link to the success of a thriving community and democratic government.
NYSAPE has long advocated for public education. We believe that while meaningful assessment is an essential component of a world class education, the NYS standardized Common Core assessments are aligned with unproven reforms that are neither supported by rigorous research nor vetted by educators and parents, and are harmful to children.
As leaders of an advocacy movement who demand an immediate end to the overemphasis on harmful testing practices, we assert that the proposed APPR regulations, EDU-27-15-00019-P, should be rejected.
Parents have become increasingly frustrated with the alarming changes happening to their children and their education. Concerns lodged by parents to State and education officials about the devastating impact that tying test scores to teacher and principal evaluations have had in public schools have been ignored and rebuffed. As a result, these parents exercised their parental rights to have more than 220,000 children refuse the 2015 state assessments, and “teachers of conscience” bravely defended their profession from flawed, harmful testing practices and bad policies sanctioned by the State.
While SED takes the position that state assessments must be tied to teacher evaluations for accountability purposes, we believe that New York State’s APPR scheme is neither a reliable nor accurate way to assess New York State teachers. In its current form, the teacher and principal evaluation scheme subjects New York State public school children to an egregiously flawed testing regime that not only harms public school children and teachers but threatens the very fabric of our public schools.
We urge the Board of Regents to reject the APPR regulations proposed for permanent adoption and to vote “No” at the September meeting. We respectfully request that the Regents utilize all avenues within their power to demand that both the Legislature and the Governor amend the education law and decouple test scores from teacher and principal evaluations.
The APPR Regulation Proposed for Permanent Adoption is Patently Flawed and Should be Rejected
As part of the 2015-16 budget bill as it relates to education, the State increased the impact of the value-added measure (VAM) on teacher and principal evaluations to a whopping 50% (and, in some cases, more than 50%). This has raised grave concerns about the impact that this policy will have on teacher and principal effectiveness outcomes as tied to student assessments. For that reason, and among others, the NYSAPE Board recently met with Commissioner Mary Ellen Elia to discuss our concerns relating to testing and evaluations.
Dr. Carol Burris, the recently retired principal of South Side High School in Rockville Centre, participated in that meeting, held August 4, 2015, between the NYSAPE Board and Commissioner Elia. Dr. Burris recounted the ratings of teachers at very low-performing schools in Buffalo in order to demonstrate that many of the teachers in those schools had received high student growth ratings. Then she recounted the student growth ratings of teachers at the high-performing Scarsdale public school and demonstrated that, by contrast, an extraordinary proportion were rated “ineffective.” Commissioner Elia agreed that these results made no sense. In fact, Commissioner Elia went on to concede that the facts Dr. Burris brought to her attention are utterly “indefensible.”
This “indefensible” phenomena is easily explained by the unintended negative consequences attendant to using student assessment scores to predict student growth for the purpose of teacher evaluations -- it produces unpredictable and irrational variability in student growth scores. Indeed, the VAM-based teacher and principal evaluation scheme that is being proposed for permanent adoption at the September Regents meeting is an “indefensible” scheme in and of itself, one that greatly prejudices teachers and principals and, ultimately, the students in their charge.
Trusted assessment and education policy experts agree that there is danger in using VAM for high-stakes decisions about teachers’ jobs and pay. The American Statistical Association issued a report in 2014 on VAM and said: “VAMs are generally based on standardized test scores and do not directly measure potential teacher contributions toward other student outcomes.”
Indeed, the use of student test scores to evaluate teachers using VAM methodology has lead to some preposterous results. In New York City, an art teacher was evaluated based on math standardized test scores and saw his evaluation rating drop from “effective” to “developing.”
Meanwhile, teachers in E.M. Baker School received a score of “ineffective,” 21 percent “developing” and 57 percent were “effective.” Just the year before, not one teacher received an “ineffective” score.
In 2013, 26 percent of Rochester teachers were rated as “ineffective” based on student performance on the state tests. Just one year later, the number of ineffective teachers dropped to 4 percent! In the same year, the percent of teachers in Yonkers who got “ineffective” growth scores fell from 18 percent to 5 percent.
In the span of one year, the percentage of teachers with “ineffective” student growth scores in top-performing Scarsdale went from 0 percent to 19 percent, and it jumped from 0 percent to 13 percent in affluent Roslyn on Long Island. Those “ineffective” ratings have even slipped into top-performing and affluent Jericho, New York, where 8 percent of those teachers are now considered “ineffective,” notwithstanding that 81 percent of Jericho students were proficient on the Common Core math tests, far exceeding and more than doubling the State average of 35 percent.
There are also disparities in the scores of “highly effective” teachers. In 2014, 10 percent of the teachers in Brooklyn (Kings County) got “highly effective” state-generated growth scores. By astounding and absurd contrast, not one teacher in Roslyn or Scarsdale did. Yet one year prior, 13 percent of Scarsdale teachers got “highly effective” scores from NYSED.
Also in 2014, 44 percent of the teachers in the Fox Meadow School in Scarsdale received growth scores that said they were not “effective” teachers, with 22 percent rated “ineffective.” Yet 61 percent of the school’s students were proficient in English Language Arts, and 75 percent were proficient in math—more than double the State’s proficiency rate.
Similar results were found at the high-achieving Harbour Hill Elementary School in Roslyn, where 36 percent of its teachers received growth scores that labeled them “ineffective.”
The Lincoln School in Rochester is a school designated as a priority/failing school by the State. Its proficiency rate on the State assessments was less than 3%. In 2014, 100% of its teachers received “effective” state scores, with 7 percent being rated “highly effective.”
At the Martin Luther King, Jr. School in Utica, NY, a school facing receivership, 60 percent of the teachers received “effective” VAM scores and 40 percent were given VAM scores of “highly effective.”
The dramatic differences cited above aptly illustrate how using the VAM formula to evaluate teachers and principals manifests in irrational state-produced teacher scores. To boot, these irrational evaluation scores are based on the flawed New York Common Core tests. Such disparate outcomes should come as no surprise to SED, the Regents or State officials. Researchers and experts have long cautioned that the VAM methodology is an inaccurate and flawed measure of individual teacher effectiveness and, accordingly, have long cautioned against using VAM to measure individual teacher effectiveness.
The American Education Research Association, and the National Academy of Education, both addressed these flaws, stating that the flaws compromise the teacher evaluation process so that the formula is rendered unstable and virtually irrational. Read more about that here.
Year-to-year volatility of the scores is also well established. The National Academy of Sciences (NAS) and the American Statistical Association have both stated that the VAM methodology is not reliable enough to be used as a major component of any teacher evaluation system:
“The report strongly supports rigorous evaluations of programs funded by the Race to the Top initiative. The initiative should support research based on data that links student test scores with their teachers, but should not prematurely promote the use of value-added approaches, which evaluate teachers based on gains in their students' performance, to reward or punish teachers. The report also cautions against using the National Assessment of Educational Progress, a federal assessment that helps measure overall U.S. progress in education, to evaluate programs funded by the Race to the Top initiative.”
Likewise, the American Statistical Association (ASA) opined, in relevant part:
“VAMs should be viewed within the context of quality improvement, which distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools. Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.”
As the ASA points out, using VAM scores to improve education requires that they “provide meaningful information about a teacher’s ability to promote student learning” but this is accomplished only in districts in which “VAMs are used for low-stakes purposes.” (emphasis added).
In fact, the ASA strongly cautions against the use of VAM for high stakes policy and decision making:
“It is unknown how full implementation of an accountability system incorporating test-based indicators, such as those derived from VAMs, will affect the actions and dispositions of teachers, principals and other educators. Perceptions of transparency, fairness and credibility will be crucial in determining the degree of success of the system as a whole in achieving its goals of improving the quality of teaching. Given the unpredictability of such complex interacting forces, it is difficult to anticipate how the education system as a whole will be affected and how the educator labor market will respond.“
Professional consensus about the use of VAM in evaluating teachers and principals is that VAM is inappropriate as a primary measure for evaluating individual teachers. It stands to reason, raising the percentage of APPR plan to 50% is irrational and patently absurd. As the evidence clearly shows, the current formula has yielded a huge and irrational disparity across schools and districts; of the gain scores in the current formula which demonstrates that the proposed APPR regulation slated for permanent adoption is utterly inappropriate.
Substantive Argument Against Permanent Adoption of APPR
The Regents are about to permanently adopt a VAM methodology in which student test scores will count for 50% (or more, depending on the matrix) of an individual teacher’s or principal’s overall rating. According to the comprehensive text of the proposed rule to be promulgated by the New York State Education Department (see memo from Ken Wagner, dated June 15, 2015, to the P-12 Committee), the methodology that the New York State Education Department proposes to use to measure the student growth component of individual teacher and principal evaluations is so scientifically flawed and unsound as to be unreasonable and unreliable. It produces unintended negative consequences that will assuredly jeopardize -- and possibly end -- the careers of many outstanding and competent New York State teachers, as well as harm the learning experiences of millions of New York State school children. The State Education Department has an affirmative duty to the children of the State of New York to prevent detrimental measures such as this from being promulgated; this proposed rule must not be allowed to pass.
In support of this assertion, we refer you to all of the evidentiary expert affidavits that were filed on behalf of the Petitioner in the case of Dr. Sheri Lederman v. John B. King, Jr., et al. (Index No. 5443-14). Initial affidavits of experts Dr. Audrey Amrein-Beardsley, Dr. Carol Burris, Dr. Linda Darling-Hammond, Dr. Brad Lindell, and Dr. Aaron Pallas can be found here.
Reply affidavits of Dr. Aaron Pallas, Dr. Brad Lindell, Dr. Audrey Amrein-Beardsley, Dr. Carol Burris, and Dr. Linda Darling-Hammond; as well as initial affidavits of Dr. Jesse Rothstein and Dr. Sean Patrick Corcoran; can be found here.
Given that officers of the New York State Education Department were direct parties to this litigation, NYSED in general is already on constructive if not actual notice of the entire contents of these various affidavits -- all of which were uncontroverted by NYSED during the matter. We hereby incorporate into this comment for your review and response each and every affidavit referenced above, in their respective entireties, especially for their discussion of the student growth measure currently in use and being proposed by State Ed.
In addition, we submit a further explanation of the affidavit that was submitted by psychologist Dr. Brad Lindell, as follows:
"I am Dr. Brad Lindell, one of the affiants in the Sheri Lederman case who was present at the oral arguments on Wednesday. It was truly something to observe. You got the feeling that good was was going to come from the great work of Sheri and Bruce Lederman and from the experts’ opinions in so far as changing this broken VAM system. You got the sense that the judge was listening to the science about VAM and not just to the political rhetoric.
Just want to fill you in on something that was presented in my affidavit modified to give a clear and understandable example of the effects of poor reliability on a full-scale WISC intelligence test. If the same test-retest reliability from the teacher assigned yearly VAM scores (.40) was applied to the WISC full-scale to determine the 90% confidence interval, the range would be ridiculously large.
Examples
If a student scored a full-scale IQ of 100 (average) then the 90% confidence interval would be an 81 to 119. This indicates that there would be a wide range where the scores from repeated administrations of the WISC would be expected to fall for this student. One could not have confidence in the validity of a intelligence test with low reliability. Without adequate reliability, there can not be validity. This same holds true for VAM scores, whose reliabilities have been found to be notorious low.
The reliability of the WISC is generally in the .80 to .90 range. The 90% confidence intervals are generally in the +\- 6 range. So this same person with a 100 full-scale IQ would have a 90% confidence range of 94-106. Quite a smaller range.
This is why reliability is so important, which has repeatedly been shown to be low like .2 to .4 for year-to-year VAM scores. This is also why teachers year to year VAM score vary so considerably, like in the case of Sheri Lederman. Without reliability there cannot be adequate validity."
Dr. Lindell's explanation can be found here.
Procedural Arguments Against Permanent Adoption of APPR
The Notice of Proposed Rulemaking is fatally defective, as it fails to identify the underlying science and research to support the rules. NY State's Administrative Procedure Act (SAPA) requires that the "Needs and Benefits" section of the Notice's mandatory "Regulatory Impact Statement" specifically identify "a citation for and summary ... of each scientific or statistical study, report or analysis that served as the basis for the rule, an explanation of how it was used to determine the necessity for and benefits derived from the rule and the name of the person that produced each study, report or analysis." SAPA section 202-a(3)(b). (With this information, the public can comment in a meaningful way, with an understanding of what underlying information supports (or doesn't) the proposed rule.)
But the Notice of Proposed Rulemaking glosses over this critical requirement; the Notice's response to the "Needs and Benefits" section admits that expert input is required by law -- but then fails entirely to identify any study, report or analysis, or the producers of those reports or the citations, all in violation of the law. The Notice says only that "The statute requires the Commissioner to adopt regulations necessary to implement the new evaluation system for teachers and principals by June 30, 2015, after consulting with experts and practitioners in the fields of education, economics and psychometrics." New York Register, July 8, 2015, page 24.
Not only is this omission in plain violation of the law -- depriving the public of its statutory right to give meaningful comment -- but the omission goes to the heart of the public's concerns about the rules. The public has been deeply troubled by the apparent arbitrariness and lack of science in prior APPR plans as well as in the new one, so much so that over 25,000 New Yorkers, including our state's most respected educators, signed a petition on this point.
The need for the science and research is also imperative -- and legally required -- as some of the most controversial elements of the rules were not decided by the legislature, but instead were specifically delegated by the legislature to SED and the BOR, and their materials are essential to the rules' validity, as well as for public comment.
For example, the ETA states that "The Commissioner shall determine the weights and scoring ranges for the subcomponent or subcomponents of the student performance category that shall result in a combined category rating. The Commissioner shall also set parameters for appropriate targets for student growth for both subcomponents ...." (ETA p. 128)
SED's proposed rule includes these scoring ranges and weights, detailing what percentage of students meeting targets will equate with a particular score of 1-20. SED's proposed rules also set forth the cut scores laying out out what numbers from the 1-20 score range equate with each of the 4 rating categories, i.e. highly effective, effective, developing and ineffective. For example, to receive a rating of effective, at least 15 out of 20 points are required.
See Proposed Rule at p. 33-34. Note that some scores have significantly wider bands than others, e.g., the scores of 13 and 14, which would not yield a positive rating, have wide bands, of 6 and 7 points, but higher scores, like 18 and 19 have bands as narrow as 2 and 3 points. Id. And yet, SED fails to provide any support whatsoever to corroborate these scoring ranges and weights. The public cannot possibly submit meaningful comment on this most important aspect of the rule, delegated to SED and BOR, without being alerted as to what supports these figures and cut scores. As such, the rules should be withdrawn, and if desired, a new Notice can be issued, that meets all of the law's requirements, and a new 45-day public comment period can commence.
Alternatively, it may be that in fact there is no support in science or research for these ranges and weights. Should that be the case, then the proposed rules should withdrawn for having no supportable basis. "Standards for Educational and Psychological Testing," ("Standards") considered the gold standard in testing issues, and even referred to specifically in the proposed rules (see p.25) could not be clearer about the importance of detailed research to support cut scores: "Cut scores embody value judgments as well as technical and empirical considerations. Where the results of the standard-setting process have highly significant consequences, those involved in the standard-setting process should be concerned that the process by which cut scores are determined be clearly documented and that it be defensible." (p.101, 2014 edition)
In addition, the proposed rules set the division of weight if two tests are used for the student growth component at 50%/50%. In the original emergency rule, discussed and adopted at the June 2015 BOR meeting, the ratio was 80% state test and 20% local. Following discussion, the weights were changed to 50%/50%. (See proposed rules, p. 33) But the discussion on whether to use the new division included no references to studies or analyses supporting or dismissing either set of numbers.
Again, "Standards" is spot on: "when evaluations or accountability systems rely on measures that combine various sources of information, such as when scores on multiple forms of a test are combined ... the rules for combining the information need to be made explicit and must be justified. It is important to recognize that when multiple sources of data are collapsed into a single composite score or rating, the weights and distributional characteristics of the sources will affect the distribution of the composite scores. The effects of the weighting and distributional characteristics on the composite score should be investigated." (p. 210, 2014 edition)
Because of the failure to identify any science, research, analysis, or report, the proposed rule must be revoked. The failure is against the law; as stated above, SAPA specifically requires this information, and SED chose to ignore that law.
The failure also impedes democracy, as the public cannot meaningfully comment without the required information. The public comment is mandatory because the rule makers were appointed, not elected, and the comment is the only input that the public has on these rules.
The failure also may indicate that in fact there is no support for these rules, that no science or research supports this. If this is the case, then the rules must be revoked on that basis.
For the reasons set forth more fully herein and above, we urge the Board of Regents to reject the APPR regulations in its current form at the September meeting, refuse to permanently adopt the same and to utilize all avenues within their power to demand that both the Legislature and the Governor amend the education law and decouple test scores from teacher and principal evaluations.
Respectfully Submitted,
NYS Allies for Public Education
By: APPR Public Comment Subcommittee Members,
Anna Shah, Esq.
Deborah Abramson Brooks, Esq.
cc: Assembly Education Committee Members
