K-2 ELA Methodology
Our English Language Arts (ELA) review teams spend approximately 5-10 hours per week over the course of several months to create the detailed reports on the site. Reviewers collect evidence from materials for identified indicators independently and meet weekly to share findings and decide on which evidence best represents the indicator. The review rubric that teams use to identify quality and information is the EdReports.org Quality Instructional Materials Tool for Grades K-2 ELA.
The EdReports.org Quality Instructional Materials Tool for Grades K-2 ELA was initially developed by a team of anchor educators who met in the spring of 2016. This anchor educator group took into consideration information, feedback, and suggestions from the “listening tour” that EdReports staff conducted with literacy experts and educators from across the country. The group reviewed the Common Core State Standards for English Language Arts (CCSS ELA), research around literacy instruction, materials review rubrics and tools currently in use in various states, the Revised Publishers’ Criteria for the Common Core State Standards in ELA and Literacy for Grades K-2 as well as other resources such as the Instructional Materials Evaluation Tool for ELA/Literacy K-2 (IMET), the ELA/Literacy Grade-Level Instructional Materials Evaluation Tool (GIMET), and the Educators Evaluating the Quality of Instructional Products rubric for ELA/literacy K-2 (EQuIP). From this study and collaboration with educators at many levels and roles, the Quality Instructional Materials Tool for Grades K-2 ELA was created. The Evidence Guides for Kindergarten, Grade 1, and Grade 2 were developed to support reviewers’ understanding of how to identify specific grade- level evidence using this rubric.
Review teams are comprised of educators with early ELA expertise from across the country. Teams collect evidence and determine scores based on instructional materials for year-long core instruction. Review teams begin at the indicator level and identify evidence for the indicators from each grade in the series. Teams meet weekly online (audio and video meetings) to decide on scores and identify the most representative evidence to support that indicator. Then they move to the next indicator, continuing to work within and across multiple grades. The lead reviewers from each team meet separately each week to discuss indicators and calibrate how evidence is being collected. Calibrators work across teams to ensure that definitions and scoring is done consistently, and to support the application of the Evidence Guides in all review team work.
Materials reviewed include teacher editions, student editions, and any related texts that are used commonly for students and identified as core products within a year-long comprehensive program.
A key component of the ELA review tool is how it addresses the whole of the ELA CCSS to include reading, writing, speaking and listening, and language. The K-2 review tool places an even brighter spotlight on the foundational skills taught within K-2 programs. The indicators listed on Rating Sheet 3 in Gateway 1 identify quality instructional components for foundational skills. Any program that does not pass this rating sheet will not move on for review in Gateway 2. Teams review the materials to ensure that all standards are covered coherently over the course of a school year’s worth of instructional materials. The tool identifies both the presence and the integration of the standards to support students as they build knowledge and literacy skills.
Using the EdReports.org Quality Instructional Materials Tool for Grades K-2 ELA and related Evidence Guides, reviewers consider the following to create the high-quality, evidence-rich reports:
Text Quality and Complexity, and Alignment to Standards with Tasks and Questions Grounded in Evidence
If material "meets" or "partially meets" move to Gateway 2.
If materials “do not meet” the requirements for Rating Sheets 1 or 3 of Gateway 1, they do not move to Gateway 2.
Building Knowledge with Texts, Vocabulary, and Tasks
If material "meets" for Gateways 1 and 2 move to Gateway 3.
Instructional Supports and Other Usability Indicators
Text Quality and Complexity and Alignment to the Standards with Tasks and Questions Grounded in Evidence (Gateway 1)
The criteria in this first gateway support reviewers to determine if high-quality texts are the central focus of lessons, are at the appropriate grade level text complexity, and are accompanied by quality tasks aligned to the standards of reading, writing, speaking, listening, and language in service to grow literacy skills. Texts identified is specific to the demands of the grade level; text complexity is assessed for read-alouds in this grade band.
Criterion 1: Reviewers first consider whether the texts are worthy of students’ time and attention (of quality, rigorous, and at the right text complexity for grade level, student, and task).
One of the most frequent questions we receive is how reviewers determine text complexity. Text complexity is defined in CCSS Appendix A and Supplemental Information for Appendix A as having three components: quantitative dimensions, qualitative dimensions, and reader and task considerations. Our tool also measures these components through a multi-step process. First, reviewers take stock of the quantitative range of the texts and compare these to the grade-band range provided in the standards. Then, review teams consider the qualitative features of the text. Reviewers also work with the reader and task considerations, particularly if the texts fall outside the grade band. A variety of text-complexity guidance tools are used during the review process, including the K-2 Text Complexity Guide.
Criterion 2: Then reviewers look to see if materials provide opportunities for rich and rigorous evidence-based discussions and writing about texts to build strong literacy skills.
In this set of indicators, reviewers identify high quality sequences of text-focused question and tasks that support students’ reading, writing, speaking and listening, and language. The reviewers identify the specific lessons, activities, practice, and assessments that meet the grade level standards. For example, when looking at the writing within materials, reviewers seek to identify if the writing tasks are grounded in evidence (1m), provide students opportunity to practice the writing types required for each grade (1l), provide practice in both on-demand and process writing (1k), and are text-dependent (1g and 1h).
Criterion 3: Finally, reviewers consider the how well the instructional materials support foundational skills development. Are the materials in reading, writing, speaking, listening, and language targeted to support foundational literacy development and are they closely aligned to the standards for each grade level?
In order for instructional materials to receive a designation of “meets” or “partially meets” expectations for Gateway 1, materials must at least partially meet expectations for all three Criteria in Gateway 1. Materials cannot receive a designation of “does not meet” on any of these criteria and be reviewed for Gateway 2.
If materials meet or partially meet expectations for Gateway 1, reviewers then proceed to the second gateway, which covers how well the materials support students to build their knowledge and academic vocabulary and identifies the integration of the standards and the shifts. Indicators in this gateway support reviewers to evaluate whether materials build students’ knowledge across topics and content areas (and/or themes, where appropriate), if academic vocabulary instruction is intentionally and coherently sequenced to build vocabulary, and if questions and tasks build in rigor and complexity to culminating tasks that demonstrate students’ ability to analyze components of texts and topics. This gateway also includes identifying the presence of coherent, year-long plans for academic vocabulary, writing development, research skills practice, and independent reading. The Evidence Guides for each grade level highlight the developmentally appropriate and grade level differences in materials to guide reviewers as they identify evidence and assign score points.
This gateway has one criterion and eight indicators:
Only materials that fully meet the expectations for the first two gateways will be reviewed for Instructional Supports and Usability (Gateway 3). This last set of indicators that reviewers examine identify how well materials support student learning and engagement and support teacher learning and understanding of the standards. The indicators identify how materials support differentiation of instruction for diverse learners and enrich instruction through technology. There are four scored criteria and one non-scored criterion. EdReports.org considers technology use to be an important element of usability, but since printed and online materials vary widely in technology implementation, ‘Effective Technology Use,’ indicators are not currently rated.