While each EdReports review process has unique characteristics that address the needs of specific content areas and review tools, there are common features that extend across all of our review processes.
Our review tools are developed by expert educators from across the country with deep content knowledge and expertise in conducting materials reviews. They absorb the findings from a national learning tour, examine existing tools and rubrics, collaborate to develop the review tool, and then finalize the review tool with input from experts in the field, including teacher membership organizations, state departments of education, researchers and leading policy voices.
These educators also develop the evidence guides that accompany the tools and can support you in both reading our reports and using the review tools as part of your own review process.
Each report found on EdReports.org represents hundreds of hours of work by educator reviewers. We select reviewers after a competitive application and review process. These reviewers, in teams of 4-5, use the review tools, evidence guides, and key documents to thoroughly examine their sets of materials. Each team is comprised of a team lead, reviewers and a calibrator. After receiving over 25 hours of training on the EdReports.org review tool and process, teams meet weekly over the course of several months to share evidence, come to consensus on scoring, and write the evidence that ultimately is shared on the website. All team members look at every grade and indicator, ensuring that the entire team considers the program in full. The team lead and calibrator also meet in cross-team PLCs to ensure that the tool is being applied consistently among review teams.
Each EdReports.org review process includes a gateway system to guide reviewers through the process. The EdReports.org’s tool supports a sequential review process through three gateways that reflect the importance of alignment to the fundamental design elements of the standards and then considers other high-quality attributes of curriculum as recommended by educators. Reviewers evaluate at least a year’s worth of core instructional materials.
A lot of research and effort goes into the creation of textbooks and digital products. EdReports.org’s tool evaluates these products from the end user’s perspective. We know when materials enter a classroom, teachers are looking for two key features:
First, is the instructional material aligned to the standards? Are all standards present and treated with appropriate depth and quality required to support students? Access to high-quality instructional materials helps students meet higher standards, enhances educator learning and saves teachers time. Aligned materials allow students to engage with texts, problems and assessments that build logically from year to year so that they graduate college- and career-ready.
Second, are the instructional materials usable for students and educators? Materials must be well designed to facilitate student learning and enhance a teacher’s ability to differentiate and build knowledge within the classroom. It also means that the materials offer assessments that help educators and parents focus on mathematical strengths and areas of growth, and incorporate technology effectively as a means to helping students learn.
The chart below illustrates how an overall alignment rating is determined for individual grade-level sets of instructional material. In order to be reviewed and attain a rating for usability (Gateway 3), the instructional materials must first meet the expectations for alignment (Gateways 1 and 2).
|Step 1||Step 2||Result: Alignment Rating|
|The reviewed materials Meet Expectations for Gateway 1||AND Meet Expectations for
|BUT either Partially Meet or Do Not Meet the expectations for Gateway 2||Partially Meet Expectations|
|The reviewed materials Partially Meet Expectations for Gateway 1||AND Meet Expectations for Gateway 2||Partially Meet Expectations|
|BUT Partially Meets expectations for Gateway 2||Partially Meet Expectations|
|BUT Do Not Meet the expectations for Gateway 2||Do Not Meet Expectations|
|The reviewed materials Do Not Meet the expectations for Gateway 1||If it Does Not Meet Expectations for Gateway 1 the material is not reviewed for Gateway 2||Do Not Meet Expectations|
From time to time, educator review teams encounter examples within the instructional materials where they need other experts to weigh in on alignment or quality. Each content area has a dedicated set of volunteer advisors – researchers and professors who are leaders in the field – that provide guidance to the teams.
All publishers are invited to provide an hour-long orientation to the review team that will be examining their materials. The review teams can also ask the publishers clarifying questions about their program as they get deeper into the materials. Publishers also have the opportunity to post two sets of information on the EdReports.org website: a 1500-word response to the educator report and a 1500-word document that includes any background information or research on the instructional materials.
Indicator: Specific item that reviewers look for in materials.
Criterion: Combination of all of the individual indicators for a single focus area.
Gateway: Organizing feature of the evaluation tool that combines criteria and prioritizes order for sequential review.
Alignment Rating: Degree to which materials meet expectations for alignment, including that all standards are present and treated with the appropriate depth to support students in learning the skills and knowledge that they need to be ready for college and career.
Usability: Degree to which materials are consistent with effective practices (as outlined in the evaluation tool) for use and design, teacher planning and learning, assessment, differentiated instruction, and effective technology use.