|
TNE Virtual Library
Design Principle A:
Decisions Driven by Evidence
1. Drawing Upon Research
2. The Role of Pupil Learning
High-Stakes
Testing, Uncertainty, and Student Learning
Audrey L. Amrein,
David C. Berliner
Education Policy Analysis Archive (Vol. 10, No. 18)
March 28, 2002
“A brief history of high-stakes testing is followed by an analysis of
eighteen states with severe consequences attached to their testing
programs. These 18 states were examined to see if their high-stakes
testing programs were affecting student learning, the intended outcome
of high-stakes testing policies promoted throughout the nation…. While
a
state's high-stakes test may show increased scores, there is little
support in these data that such increases are anything but the result
of
test preparation and/or the exclusion of students from the testing
process.… Because clear evidence for increased student learning is not
found, and because there are numerous reports of unintended
consequences
associated with high-stakes testing policies (increased drop-out rates,
teachers' and schools' cheating on exams, teachers' defection from the
profession, all predicted by the uncertainty principle), it is
concluded that there is need for debate and transformation of current
high-stakes testing policies.”
Sizing Up Test Scores
Dale Ballou
Education Next (Vol. 2, No. 2)
Summer 2002
“One of the basic critiques of using test scores for accountability
purposes has always been that simple averages, except in rare
circumstances, don’t tell us much about the quality of a given school
or
teacher…. The more serious difficulties arise when value-added
assessments are used to hold schools and teachers accountable, with
high-stakes personnel decisions to follow. The danger is that such
assessments will be used to supplant local decisionmaking, rather than
to inform it.”
Measuring
the Impacts of Whole-School Reforms: Methodological Lessons from an
Evaluation of Accelerated Schools
Howard S. Bloom
Manpower Demonstration Research Corporation
U.S. Department of Education, Planning and Evaluation Service
October 2001
“The goal of the present paper is to introduce education researchers to
a new approach for measuring the impacts of whole-school reforms. The
approach is based on interrupted time-series analysis, which has been
used to evaluate programs in many fields but has not been used widely
to
study education initiatives. The application presented measures program
impacts on student performance by comparing standardized test scores
for a number of annual student cohorts in a specific grade after a
reform is launched (its follow-up period) with the scores of cohorts
from several years before the reform was launched (its baseline
period).
The approach is used to measure impacts on three facets of student
performance: (1) average (mean) test scores, which summarize impacts on
total performance; (2) the distribution of scores across specific
ranges, which helps to identify where in the distribution of student
performance impacts were experienced; and (3) the variation (standard
deviation) of scores, which indicates how the disparity in student
performance was affected. To help researchers use the approach, the
paper lays out its conceptual rationale, describes its statistical
procedures, explains how to interpret its findings, indicates its
strengths and limitations and illustrates how it was used to evaluate a
major whole-school reform model—Accelerated Schools.”
The Virtues of
Randomness:
Education Discovers Randomized Field Trials
Robert Boruch
Education Next (Vol. 2, No. 3)
Fall 2002
"Randomized trials will only become more frequent in the education
arena as time goes on. Today's reform environment demands hard evidence
on which programs and policies are effective at raising student
achievement. So far, randomized trials provide the best way of
obtaining
such evidence, and their techniques become more sophisticated by the
day. The question is whether these results will be used to effect
change or be held hostage to the ideological posturing that so often
substitutes for evidence in the education world."
The
Full Circle: Building a Coherent Teacher Preparation System.
The Report of the NASBE Study Group on Coordination and Accountability
in Teacher Education
Carla Claycomb,
Jeanne Pecori
National Association of State Boards of Education
October 2000
“More and more, teachers, policymakers, parents, and others are
realizing that student achievement need not be prescribed by
socioeconomic status, parent involvement, or race and ethnicity; on the
contrary, recent evidence makes clear that regardless of the factors
that students bring to school, good teachers measurably increase
student
learning, and good schools foster high levels of student achievement in
large part because of the quality of their teachers. As a matter of
fact, teacher quality may be one of the most significant factors in
student achievement.”
Measuring
What Matters: An Update on Educational Assessment and Accountability
Committee for
Economic Development
August 2002
"In this 11-page brief, the Committee for Economic Development (CED)
examines the No Child Left Behind Act's (NCLB) key assessment and
accountability provisions and issues surrounding their implementation.
The report is a follow-up to a CED tract (Measuring What Matters:
Using
Assessment and Accountability to Improve Student Learning) that was
released early last year - while NCLB was still being tossed around in
Congress and standards and testing systems were more matters of state
than federal policy. If NCLB is to improve student learning, the new
report says, it must meet a trio of challenges (all pretty obvious yet
critical): 1) test results must measure and report student achievement
accurately; 2) educators must teach solid content, not just 'teach to
the test'; and 3) low-performing schools must be given extra
assistance. Following an explanation of each challenge is a list of
related issues to be monitored and questions that policymakers ought to
ask when designing and reforming standards and accountability systems."
Defining
"Highly Qualified Teachers": What Does "Scientifically Based Research"
Actually Tell Us?
Linda
Darling-Hammond, Peter Youngs
Educational Researcher (Vol. 31, No. 9)
December 2002
"In this report titled 'Meeting the Highly Qualified Teachers
Challenge' the Secretary essentially argues for the dismantling of
teacher education systems and the redefinition of teacher
qualifications
to include little preparation for teaching. . . The report suggests
that its recommendations are based on 'solid research.' However, none
of these arguments has strong empirical support, and the report does
not cite the scientific literature that addresses them."
Teacher Quality and Student
Achievement: A Review of State Policy Evidence
Linda Darling-Hammond
Education Policy Analysis Archives (Vol. 8, No. 1)
January 1, 2000
“Using data from a 50-state survey of policies, state case study
analyses, the 1993-94 Schools and Staffing Surveys (SASS), and the
National Assessment of Educational Progress (NAEP), this study examines
the ways in which teacher qualifications and other school inputs are
related to student achievement across states. The findings of both the
qualitative and quantitative analyses suggest that policy investments
in
the quality of teachers may be related to improvements in student
performance. Quantitative analyses indicate that measures of teacher
preparation and certification are by far the strongest correlates of
student achievement in reading and mathematics, both before and after
controlling for student poverty and language status. State policy
surveys and case study data are used to evaluate policies that
influence
the overall level of teacher qualifications within and across states.
This analysis suggests that policies adopted by states regarding
teacher education, licensing, hiring, and professional development may
make an important difference in the qualifications and capacities that
teachers bring to their work. The implications for state efforts to
enhance quality and equity in public education are discussed.”
Legislating equity: The
distribution of emergency permit teachers in California
Laura Goe
Education Policy Analysis Archives (Vol. 10, No. 42)
October 14, 2002
“There is a significant negative relationship between the percentage of
teachers on emergency permits and student achievement at the school
level in California schools, after controlling for other student and
school characteristics. Generally, the more emergency permit teachers
there are in a school, the lower the school's achievement. This
phenomenon is examined in the context of other contributors to student
achievement such as socio-economic status and school size. The
effects of teacher distribution and school selection as contributing
factors are considered. In addition, policy and legislative
initiatives related to emergency permit teachers that have been
recently
debated in California will be discussed. Finally, a set of initiatives
is proposed that attempt to decrease the need for emergency permit
teachers and ensure that those that must be hired due to shortage
conditions have the support they need to become credentialed teachers.”
The Business Model
Jay P. Greene
Education Next (Vol. 2, No. 2)
Summer 2002
“Like the makers of hot dogs, psychometricians, economists, and other
testing experts know too well what goes into the creation of
achievement
tests. Their intimate knowledge of the technical difficulties involved
in measuring student achievement makes a number of these testing
experts some of the most vocal (and persuasive) opponents of testing.
But the flaws in techniques like value-added assessment do not
automatically lead to the conclusion that those techniques shouldn’t be
used to hold educators accountable. Testing may be imperfect, but the
alternative—the old system, which allowed us to know very little about
the performance of educators—is far, far worse.”
A
Knowledge Base for the Teaching Profession: What Would It Look Like and
How Can We Get One?
James Hiebert, Ronald
Gallimore, James W. Stigler
Educational Researcher (Vol. 31, No. 5)
June/July 2002
“To improve classroom teaching in a steady, lasting way, the teaching
profession needs a knowledge base that grows and improves. In spite of
the continuing efforts of researchers, archived research knowledge has
had little effect on the improvement of practice in the average
classroom. We explore the possibility of building a useful knowledge
base for teaching by beginning with practitioners' knowledge. We
outline
key features of this knowledge and identify the requirements for this
knowledge to be transformed into a professional knowledge base for
teaching. By reviewing educational history, we offer an incomplete
explanation for why the United States has no countrywide system that
meets these requirements. We conclude by wondering if U.S. researchers
and teachers can make different choices in the future to enable a
system
for building and sustaining a professional knowledge base for teaching.”
District
fiscal policy and student achievement: Evidence from combined NAEP-CCD
data
Gary G. Huang,
Binbing Yu
Education Policy
Analysis Archives (Vol. 10, No. 38)
September 22, 2002
“School restructuring raises questions about the role of school
districts in improving student learning. Centralization by state
governments and decentralization to individual schools as proposed in
systemic reform leave districts' role unsettled. Empirical research on
the district role in the context of ongoing reform is inadequate. This
analysis of combined data from the NAEP and the Common Core of Data
(CCD) was intended to address the issue… The null effect was consistent
in the analysis of the combined NAEP-CCD data for 1990, 1992, and 1996.
In contrast, current expenditure per pupil was found related to higher
math performance in a
modest yet fairly consistent way. Future research may be productive to
separately study individual states and integrate the findings onto the
national level.”
Class Size
Reduction, Teacher Quality, and Academic Achievement in California
Public Elementary Schools
Christopher Jepsen,
Steven Rivkin
Public Policy Institute of California
2002
“In 1996, California passed a statewide class size reduction (CSR) law
that aimed to reduce average class sizes by roughly one-third in
kindergarten through third grade. Educators and policymakers expected
CSR to lead to large gains in student achievement. However, increasing
the state’s teaching workforce by thousands of new teachers had the
potential to offset the direct benefits of smaller classes,
particularly
for schools in economically disadvantaged communities that already had
staffing difficulties. This report addresses the following questions:
What were the effects of CSR on overall teacher experience,
certification, and education? Were some schools affected more than
others?; How did CSR affect student achievement? What were the benefits
of smaller classes? What were the effects of new teachers?; Are the
benefits of smaller classes concentrated among a subset of students, or
did all schools benefit equally from CSR?”
Better
Teachers, Better Schools
Marci Kanstoroom,
Chester E. Finn, Jr., Editors
Thomas Fordham Foundation
July 1999
“This book suggests a different way of thinking about this perplexing
but important issue, a way of thinking that's grounded in common sense
rather than piety and that relies on evidence rather than supposition
or
wishful thinking. The common sense approach that we propose to boosting
teacher quality involves easing back on regulations that control entry,
devolving personnel decisions to individual schools, and then holding
those schools accountable for producing results as gauged by their
pupils' academic achievement. If this ‘tight-loose’ strategy sounds
familiar, that's because it has become the dominant paradigm for
reforming schools today. We think it warrants consideration for
teachers, too.”
The Effectiveness of "Teach
for America" and Other Under-certified Teachers on Student Academic
Achievement: A Case of Harmful Public Policy
Ildiko Laczko-Kerr,
David C. Berliner
Education Policy Analysis Archives (Vol. 10, No. 37)
September 6, 2002
“The academic achievements of students taught by under-certified
primary school teachers were compared to the academic achievements of
students taught by regularly certified primary school teachers.
This sample of under-certified teachers included three types of
under-qualified personnel: emergency, temporary and provisionally
certified teachers. One subset of these under-certified teachers
was from the national program Teach For America (TFA). Recent
college graduates are placed by TFA where other under-qualified
under-certified teachers are often called upon to work, namely,
low-income urban and rural school districts. Certified teachers in this
study were from accredited universities and all met state requirements
for receiving the regular initial certificate to teach. Recently
hired under-certified and certified teachers (N=293) from five
low-income school districts were matched on a number of variables,
resulting in 109 pairs of teachers whose students all took the mandated
state achievement test. Results indicate 1) that students of TFA
teachers did not perform significantly different from students of other
under-certified teachers, and 2) that students of certified teachers
out-performed students of teachers who were under-certified. This
was true on all three subtests of the SAT 9—reading, mathematics and
language arts. Effect sizes favoring the students of certified
teachers were substantial. In reading, mathematics, and language,
the students of certified teachers outperformed students of
under-certified teachers, including the students of the TFA teachers,
by
about 2 months on a grade equivalent scale. Students of
under-certified teachers make about 20% less academic growth per year
than do students of teachers with regular certification.
Traditional programs of teacher preparation apparently result in
positive effects on the academic achievement of low-income primary
school children. Present policies allowing under-certified
teachers, including those from the TFA program, to work with our most
difficult to teach children appear harmful. Such policies
increase
differences in achievement between the performance of poor children,
often immigrant and minority children, and those children who are more
advantaged.”
Accountability
Systems: Implications of Requirements of the No Child Left Behind Act
of 2001
Robert L. Linn, Eva
L. Baker, Damian W. Betebenner
Educational
Researcher (Vol. 31, No. 6)
August/September 2002
“The No Child Left Behind Act of 2001 substantially increases the
testing requirements for states and sets demanding accountability
standards for schools, districts, and states with measurable adequate
yearly progress (AYP) objectives for all students and subgroups of
students defined by socioeconomic background, race/ethnicity, English
language proficiency, and disability. However, states' content
standards, the rigor of their tests, and the stringency of their
performance standards vary greatly. Consequently, the percentage of
students who score at the proficient level or higher on the state
assessments varies radically from state to state. Some states have
farther to go than others to meet the mandated target of 100%
proficient
within 12 years. These differences are illustrated and the implications
for achieving AYP targets are discussed. Also addressed are possible
uses of results from the biennial state-level administrations of the
National Assessment of Educational Progress as a means of leveling the
playing field. Factors contributing to the volatility of gains in
achievement from year to year for individual schools are discussed.”
Consequences of Assessment:
What is the Evidence?
William A. Mehrens.
Education Policy Analysis Archive (Vol. 6, No. 13)
July 14, 1998
“Attention is here directed toward the prevalence of large scale
assessments (focusing primarily on state assessments). I examine the
purposes of these assessment programs; enumerate both potential dangers
and benefits of such assessments; investigate what the research
evidence
says about assessment consequences (including a discussion of the
quality of the evidence); discuss how to evaluate whether the
consequences are good or bad; present some ideas about what variables
may influence the probabilities for good or bad consequences; and
present some tentative conclusions about the whole issue of the
consequences of assessment and the amount of evidence available and
needed.”
The Class Size
Debate
Lawrence Mishel,
Richard Rothstein, editors
Economic Policy Institute
2002
“In The Class Size Debate, two eminent economists debate the merits of
smaller class sizes and the research methods used to measure the
efficacy of this education reform measure. Alan Krueger (Princeton
University) maintains that smaller class sizes can improve students’
performance and future earnings prospects. He challenges Eric
Hanushek’s
(Stanford University) widely cited analysis of the class size
literature, arguing that it gives disproportionate weight to single
studies that include a large number of estimates. An appropriate
weighting, he says, would reveal that class size is indeed a
determinant
of student achievement. Prof. Hanushek counters that Prof. Krueger’s
re-analysis achieves results different from his own by emphasizing
low-quality estimates. He argues that other policies besides class size
reduction, such as improving teacher quality, are more important.
Jennifer King Rice (University of Maryland) brings a third-party
perspective to the debate. She addresses each author’s arguments and
focuses on the policy implications of the class size literature.”
Improving
Teacher Evaluation to Improve Teaching Quality
National Governors
Association
December 2002
"Governors understand the importance of guaranteeing that every child
has an effective teacher. Research shows that teacher quality affects
student achievement more greatly than any other school based variable.
The No Child Left Behind Act requires a 'highly qualified' teacher in
every classroom by the 2005-2006 school year and achievement gains by
all students over time. These realities give policymakers a strong
incentive to focus on preparing, recruiting, and retaining quality
teachers as primary strategies to boost academic achievement."
Measuring
the Content of Instruction: Uses in Research and Practice
Andrew C. Porter
Educational Researcher (Vol. 31, No. 7)
October 2002
"This article describes tools for measuring the content of instruction,
the content of instructional materials, and the alignment between
these.
Illustrative findings about the use of these tools are reported, and
possible additional uses, both for research and practice, are
discussed. The validity of data produced through use of these tools is
found to be quite good. An agenda for future work is
sketched both for improvement of the quality and
versatility
of the tools and for use of the tools in research and practice."
Testing
the Testers: An Annual Ranking of State Accountability Systems
Princeton Review
2003
"During the Winter of
2002-2003, The Princeton Review conducted Testing the Testers 2003, its
second Annual Ranking of State Accountability Systems. Unlike other
studies, ours is not primarily concerned with the rigor of academic
standards or of the tests that measure them. Rather we focused on the
policies that determine the overall character and effectiveness of each
accountability system. Properly conceived and well-implemented, these
policies will tend to produce systems that are consistent, secure, open
to public scrutiny, and flexible enough to improve over time. We also
believe they will tend to encourage and support an evolution to better
and more effective schools. "
Teacher Quality
Initiative
Public Education
Foundation, Chattanooga, TN
Pages include a description of the project (which was discussed during
the December 2002 TNE Institute in Washington, DC) and links to
an October 2002 PowerPoint presentation of preliminary results, and
results released in February 2003 of their teacher questionnaire.
What
large-scale, survey research tells us about the effects of teachers and
teaching on student achievement.
Brian Rowan
Consortium for Policy Research in Education
2002
“This paper is about conceptual and methodological issues that arise
when educational researchers use data from large-scale, survey research
studies to investigate teacher effects on student achievement. In
the paper, I illustrate the conceptual and methodological issues that
arise in such research by reporting on a series of analyses I conducted
using data from Prospects: The Congressionally Mandated Study of
Educational Opportunity. This large-scale, survey research effort
gathered a rich store of data on instructional processes and student
achievement in a large sample of American elementary schools during the
early 1990’s as part of the federal government’s evaluation of the
Title
I program. In this paper, I use data from this study to estimate the
‘overall’ size of teacher effects on student achievement and to test
some specific hypotheses about why such effects occur. On the basis of
these analyses, I draw some substantive conclusions about both the
magnitude and sources of teacher effects on student achievement and
suggest some ways that survey-based research on teaching can be
improved.”
Educational Assessment
Reassessed: The Usefulness of Standardized and Alternative Measures of
Student Achievement as Indicators for the Assessment of Educational
Outcomes
William L. Sanders,
Sandra P. Horn
Education Policy Analysis Archive (Vol. 3 No. 7)
March 3, 1995
“For decades, the assessment of educational entities--school systems,
individual schools, and teachers--has evoked strong and sometimes
violent emotions from the educational community, the general public,
and
their legislative representatives. In spite of attempts to codify
standards for the evaluation of these entities, assessment experts
remain denominationalized--often religiously so. Methods of assessment
based on the use of standardized tests have come under intense fire in
recent years with some critics going so far as to call for their
complete elimination. Those who advocate alternative methods of
assessment have become increasingly outspoken in establishing exclusive
rights to the legitimate assessment paradigm. However, some of the most
respected advocates of alternative assessment have taken a more
moderate
view, warning against an ‘either-or’ mentality (Brandt, 1992, p. 35).
Reflecting this more moderate perspective, this paper strongly
advocates the use of multiple indicators of student learning, including
those provided by standardized tests.”
Foundations for
Success: Case Studies of How Urban School Systems Improve Student
Achievement
Jason Snipes, Fred
Doolittle, Corinne Herlihy
Council of Great City Schools
September 2002
"This report and the longer-term project of which it is a part focus on
the potential role of the school district as an initiator and sustainer
of academic improvement. While there has been much research on what
makes an effective school, there is relatively little on what makes an
effective district. In fact, many see large urban school districts as a
source of problems rather than solutions. But for school improvement to
be widespread and sustained, and for our nation to reduce racial
differences in academic achievement, large urban districts must play a
key role…. This report extends the existing research by examining the
experiences of three large urban school districts (and a portion of a
fourth) that have raised academic performance for their district as a
whole, while also reducing racial differences in achievement."
Evidence
Based Education Policies:
Transforming Educational Practice and Research
2002 Dewitt Wallace Reader's Digest Distinguished Lecture
Robert E. Slavin
Educational Researcher (Vol. 31, No. 7)
October 2002
"At the dawn of the 21st century, educational research is finally
entering the 20th century. The use of randomized experiments that
transformed medicine, agriculture, and technology in the 20th century
is
now beginning to affect educational policy. This article discusses the
promise and pitfalls of randomized and rigorously matched experiments
as a basis for policy and practice in education. It concludes that a
focus on rigorous experiments evaluating replicable programs and
practices is essential to build confidence in educational research
among policymakers and educators. However, new funding is needed for
such experiments and there is still a need for correlational,
descriptive, and other disciplined inquiry in education. Our children
deserve the best educational programs, based on the most rigorous
evidence we can provide."
Expert Measures
Anita A. Summers
Education Next (Vol. 2, No. 2)
Summer 2002
“Critics of value-added assessment tend to embrace the concept but
don’t want the results gleaned from such analysis to be used for
accountability purposes—and especially don’t want to use the results to
reward or sanction teachers…. The main concern with value-added
assessment is that the technique exacerbates the amount of random error
involved in measuring student performance. The risk is that teachers
and
schools may be wrongfully rewarded or punished because value-added
techniques either over- or underestimated their students’ learning
gains.”
How Schools Matter: The Link
Between Teacher Classroom Practices and Student Academic Performance
Harold Wenglinsky
Education Policy Analysis Archives (Vol. 10, No. 12)
February 13, 2002
“Quantitative studies of school effects have generally supported the
notion that the problems of U.S. education lie outside of the school.
Yet such studies neglect the primary venue through which students
learn,
the classroom. The current study explores the link between classroom
practices and student academic performance by applying multilevel
modeling to the 1996 National Assessment of Educational Progress in
mathematics. The study finds that the effects of classroom practices,
when added to those of other teacher characteristics, are comparable in
size to those of student background, suggesting that teachers can
contribute as much to student learning as the students themselves.”
|
|
|
|
|
