Background and Summary
Based
partly on tradition and partly on the requirements of state
and federal legislation, including that of No Child Left
Behind, achievement testing in reading and mathematics is
administered in nearly every state. They report on every
district and every school within their public education
systems. Separately, at the federal level, there is the
National Assessment of Educational Progress (NAEP)- also
known as the Nation’s Report Card. Important
characteristics of the NAEP include:
•
A long track record of 37 years, establishing itself as the
defacto national standard for achievement.
• The NAEP achievement level of
“proficient” defines what it is to be at
“grade level.”
• Content standards tested by the NAEP are drawn from
the National Assessment Governing Board.
• NAEP tests children in 4th
and
8th
grades
and reports proficiencies at the state level.
• In 12th
grade
NAEP tests and reports only at the national level.
• The use of statistical sampling techniques prevents
reporting at district and school level.
The tests administered by the states also claim to
determine which students have become
“proficient” in the tested subject areas.
Generally, the state administered achievement tests:
•
Examine sufficient numbers of children to report
statistically valid results for schools and districts.
• But measure against content standards and use
cut-scores inconsistent with the NAEP.
• Most states significantly inflate the number of
children deemed “proficient” compared to the
NAEP.
• Some grossly inflate their proficiencies to as much
as 300% or more above the NAEP results. The median state
inflates at about 100%.
For stakeholders at the school and district levels, the
well-respected NAEP results are not available and the state
generated results are unreliable, due to the inflation. To
address this lack of useful information:
•
We developed a conversion formula (mapping procedure) to
estimate what the NAEP proficiencies would have been in
these localities. It essentially removes the inflation from
the state reported scores.
• Error measurements of this procedure in simulated
environments have shown good accuracy.
• Where actual results are known independently we have
verified that they are reasonably accurate.
• The method’s errors are measured to be of the
same order as the NAEP reported sampling errors.
• In terms of proficiency percentages, the measured
errors have rarely exceeded 5%.
By applying these mapping procedures we have, among others,
constructed report cards for every public school and every
school district within Oklahoma,Pennsylvania, Rhode Island,
Ventura County in California, Bristol County in
Massachusetts, and Hudson County in New Jersey. Our
findings include:
•
In the worst performing schools, less than 5% of children
are proficient in both reading and math.
• Among the “best” performing schools, it
is rare to find more than 70% of the students proficient.
• Mathematics proficiency percentages are generally
lower than those for reading during the high school years.
Our
evaluation of the analysis conducted thus far suggests:
•
In all jurisdictions studied, social promotion is the key
dysfunction leading to the low proficiencies.
• Age based group instruction hinders the ability to
properly “place” students consistent with skill
levels.
• Strict retention policies, such as used in India,
could correct and eliminate social promotion, but we think
such methods are too draconian.
• However, replacing group instruction with self-paced
(probably online) instruction would be effective.
• Reform the math curriculum first and extend it to
all math courses. A few such efforts are underway,
including our own Stellar Schools
project.
Asora®
Report Card Services:
We are
ready to work with interested clients who seek to obtain
more realistic estimates of student proficiencies in the
K-12 public schools within their states or in other client
defined regions.
To conduct its analysis, Asora Consulting uses as its input
data:
•
Recently published results from the Nation’s Report
Card- also known as the National Assessment of Educational
Progress (NAEP), which it obtains from trusted data
repositories or from the U.S. Department of Education
directly.
•
Recently published results from the state’s
achievement test as reported by the states directly or
through Standard & Poor’s data repository
at
SchoolMatters.Com.
District Proficiencies
After
applying our mapping procedure to the input data, we can
generate estimates as to how students would have performed
on the Nation’s Report Card on a district-by-district
basis. We generally provide estimates at the primary school
level (4th
grade),
the middle school level (8th
grade)
and at a high school grade level (which varies from state
to state according to the level tested by the state’s
examination). Results are presented in terms of the
percentages of children measured to be proficient or
better. According to the definition of proficiency used by
the NAEP it also denotes the percentage of students who are
at or above grade level. Given the fact that proficiency
levels are nearly always “inflated” by the
state administered examinations, the proficiency estimates
we generate are low and troubling. President Bush’s
remark about educators being engaged in “the soft
bigotry of low expectations” is made quite evident in
these estimates.
School
Proficiencies
In a
similar manner we can produce estimates for individual
schools. These estimates are useful in showing the degree
to which schools are dysfunctional. It is not uncommon to
find schools with proficiency percentages well less than
5%. We believe that a properly operated school will
maintain student proficiencies above 90%. However, among
the public schools we have checked none has ever reached
that level.
Private
School Proficiency Estimates
In most
states, private school students do not take the
state’s achievement tests. Where and if they do, we
can generate NAEP scale estimates for them in the same way
we generated the public school estimates. For those schools
that do not take the state’s achievement tests but do
take the SAT we can make a rough estimate of the
12th
grade
proficiency levels of those schools. For example, we
performed such an analysis in Rhode Island and found only
two private schools in which we estimate proficiency
percentages above 90%. Many private schools do not release
their SAT scores. In those cases we are unable to make
estimates.
Commissioned
Reports
Depending
on the client’s needs and wishes we can provide
reports based on the performance estimates generated. The
report could range in length and complexity from:
• A
short description of the findings with selective
presentation of the estimates.
• To a more in-depth analysis of the findings with a
complete presentation of the estimates generated.
We will work with clients to produce reports suitable for
their needs.
Clients will hold the copyright to reports generated for
them and will be permitted to revise and extend them
subject to the requirement that Asora Consulting will be
cited for its contributions.
Some
Examples
We have
analyzed student proficiencies in Oklahoma, Pennsylvania
and Rhode Island and provided NAEP estimates for every
public school in these states. We also looked at a small
number of private schools for which SAT scores were
available.
The Rhode Island results are contained in a rather
expansive report, RIProfNumsTechReport.doc,
that can be
downloaded from our website. Our client for this report was
the Ocean
State Policy Research Institute.
A shorter version of this report, RIProfNumsBasicReport.doc,
was also
prepared for presentation to the Governor’s office.
It is also available from our website.
In the case of Ventura County, California we looked at
every public school in that county but did not look at any
private schools. Here the report, VenturaProfNums.doc,
was of
moderate length. Our client for this report was the
Coalition
for Accountability in Education.
The reports for Oklahoma and Pennsylvania are expected to
be released soon by the contract patron organizations.
Some of them can be downloaded from our Reform Reports
page.
Methodology
The
derivations of our mapping method can be reviewed in the
report MapToNAEP.doc,
which is also downloadable from our Reform Reports
page.
__________________
If you are interested in reviewing the "big picture" of
Asora's full breadth of activities, click here to reach our
home page.