| Title | Conference | Author | Date |
|---|---|---|---|
| Evaluating Usability and Utility of a Teacher Dashboard to Support Instructional Decision Making | National Council on Measurement in Education | Dolan, R.P., Gilbert, S.M., Ducharme, K., Wojcik, C., Posey, A. | June 2021 |
| The I-SMART Project: Empirical Map Validation | National Council on Measurement in Education | Hoover, J.C., Thompson, W.J., Nash, B., Kobrin, J.L. | June 2021 |
| Supporting Diverse Students and Teachers in Effective Classroom Assessment Through UDL | The 2020 UDL-IRN International Summit | Dolan, R., & Ducharme, K. | June 2020 |
| The I-SMART Project: Creating Multi-Dimensional Science Assessments Using Principles of Universal Design for Learning and Fine-Grained Learning Maps | National Conference on Student Assessment, Orlando, FL | Shipman, S., Tiemann, G.C., Dolan, R, & Swinburne Romine, R. | June, 2019 |
| Design Features Supporting Teachers' Use of a Dashboard for Diagnostic Assessment Results | AERA 2019 Annual Meeting, Toronto, ON | Starr, E., Dolan, R., Wojcik, C., Ducharme, K., & Blackorby, J. | April, 2019 |
| Teacher-Centric Design Process for a Dashboard to Support Formative Assessment | National Council on Measurement in Education, Toronto, ON | Dolan, R., Starr, E., Wojcik, C., Ducharme, K., & Blackorby, J. | April, 2019 |
| Blending Evidence-Centered Design and Univeral Design for Learning in Next-Generation Science Assessment | National Council on Measurement in Education, Toronto, ON | Karvonen, M., Andersen, L., Swinburne Romine, R., & Tiemann, G. | April, 2019 |
| Using Learning Map Models to Design Accessible Assessments | NARST 2019 Conference, Baltimore, MD | Andersen, L., Bechard, S., Swinburne Romine, R., Ruhter, L., & Shipman, M. | April, 2019 |
| I-SMART! Learn about UDL Applied in Assessments | 2019 UDL-IRN International Summit, Orlando, FL | Blackorby, J., Swinburne Romine, R., Karvonen, M., Dolan, R., & Wojcik, C. | March, 2019 |
| Design and Use of Innovative Science Testlets for Struggling Learners | NCME Special Conference on Classroom Assessment, Lawrence, KS | Andersen, L., Shipman, M., & Tiemann, G. | September, 2018 |
| Innovations in Science Map, Assessment, and Report Technologies (I-SMART) | National Conference on Student Assessment, San Diego, CA | Torchon, M. & Tiemann, G. | June, 2018 |
| The Development of Universally Designed, Fine-Grained Science Learning Map Models | 2018 NARST International Conference, Atlanta, GA | Andersen, L. & Swinburne Romine, R. | March, 2018 |
| Innovations in Science Map, Assessment, and Report Technologies (I-SMART) | 2018 SCILLSS & I-SMART Introductory Meeting | Karvonen, M. | January, 2018 |
| Innovations in Science Map, Assessment, and Report Technologies (I-SMART) | Technical Issues in Large Scale Assessment (TILSA) State Collaborative on Assessment and Student Standards (SCASS), Dallas, TX | Karvonen, M. | October, 2017 |
Resources
Presentations
Publications
Essential Element Concept Maps (EECMs)
Essential Element Concept Maps (EECMs) incorporate important features of evidence-centered design (ECD), learning map models (LMs), and Universal Design for Learning (UDL) in a framework that was appropriate for educators serving as item writers. The EECMs for this project are extensions of prior work for the DLM ELA, math, and science assessments. In the DLM assessment, the five assessment targets (linkage levels) derived from LMs each have a unique test design pattern. The EECM serves as guide for test developers that contains the specifications for assessment items and testlets in a format that integrates the structure of the learning map model.
Each EECM has a separate section for each linkage level that expands upon the description of the assessment targets. Here is an example EECM that describe the target linkage level test design for EE.5.LS2-1. Vocabulary words and example questions are specified in the EECMs so that all students, regardless of their communication mode, can respond to items. Misconceptions are possible student errors related to the concepts assessed by the node. Progression information summarizes the science concepts and SEP-related knowledge, skills, and understandings within the LL. The phenomenon section describes the general mechanism behind the science concepts being assessed, as well as an example phenomenon that could be used in an assessment that is appropriate for the LL. The science and engineering practices section describes how the students use the SEP at this linkage level. The four nodes assessed at this LL are presented in a table with detailed descriptions of the skills and examples of how the skills can be observed or assessed. The skills and connections in the progression of learning within a LL are shown in the Linkage Level Mini-Map.
UDL options to be used in the LL are organized in the EECM by the three principles of multiple means of representation, engagement, and action and expression, as well as the corresponding guidelines under each principle. The UDL options are specific to each LL and include options provided through testlet content, such as options for language and symbols, comprehension, recruiting interest, self-regulation, expression and communication, and executive functions. For example, at the target level, a wonder question and a “How did you do?” reflection question are self-regulation options that helps students develop self-assessment and reflection skills.
The EECM also includes an excerpt from the research narrative that was prepared during LM construction, which describes the progression of knowledge for the DCI and SEP in more detail. The narrative highlights specific connections among the nodes in the LLs.
Learning Map Neighborhoods
Learning map models are a type of cognitive model that represent the knowledge, skills, and understandings (KSUs) within a domain. I-SMART science learning maps were developed in neighborhoods that focus on a single content standard. I-SMART researchers created 11 new science learning map neighborhoods with accompanying research narratives. Neighborhood maps included 32 to 52 skills and 36 to 83 connections. As an example of this work, the research narrative for Physical Science 1, Structure and Properties of Matter, and its associated neighborhood maps are posted below.
For more information about the components of learning map models and their development for I-SMART, please see the technical report, Developing and Evaluating Learning Map Models in Science: Evidence from the I-SMART Project or the project brief, Developing & Evaluating Science Learning Map Models (I-SMART Project Brief No. 19.1).
PS1 Learning Map Model Neighborhood Research Narratives (pdf) summarizes the research that was used to create the learning map model neighborhoods for the four Physical Science 1 Essential Elements. First, the science core idea and science and engineering practice components for the Essential Element are described. Then, the research on how students learn this content is presented, organized by grade span and topic.
Learning Map Model Neighborhood EE.5.PS1-2 (pdf) is associated with the Essential Element, "Measure and compare weights of substances before and after heating, cooling, or mixing substances to show that weight of matter is conserved." The neighborhood includes the DCI, "PS1.A Structure and Properties of Matter," and the SEP, "Using Mathematics and Computational Thinking."
Learning Map Model Neighborhood EE.5.PS1-3 (pdf) is associated with the essential element, "Make observations and measurements to identify materials based on their physical properties." The neighborhood includes the DCI, "PS1.A Structure and Properties of Matter," and the SEP, "Planning and Carrying Out Investigations."
Learning Map Model Neighborhood EE.MS-PS1-2 (pdf) is associated with the Essential Element, "Interpret and analyze the data on the properties of substances before and after chemical changes have occurred." The neighborhood includes the DCI, "PS1.A Structure and Properties of Matter," and the SEP, "Analyzing and Interpreting Data."
Learning Map Model Neighborhood EE.HS.PS1-2 (pdf) is associated with the Essential Element, "Make a claim supported by evidence to explain patterns of chemical properties that occur in a substance during a common chemical reaction." The neighborhood inclues the DCI, "PS1.A Structure and Properties of Matter," and the SEP, "Constructing Explanations and Designing Solutions."
Test Design Overviews & Exemplars
I-SMART researchers tried-out new ideas in assessment design that could improve measurement of the newer and more complex performance expectations from the Next Generation Science Standards. Our plan was to assess more complex science standards without introducing new barriers for students, especially for students with disabilities and struggling learners. Infusing UDL principles into our assessment designs helped us imagine new ways to increase access to the science content and increase interest and engagement.
Two resources highlight I-SMART's testlet design innovations. A testlet is a collection of items that assess a specific linkage level in the I-SMART learning map models.
Test Design Overview and Exemplars from an Initial Linkage Level Testlet (pdf) highlights a testlet that measures a life science at the middle school grade band. The testlet addresses the disciplinary core idea of Ecosystems: Interactions, Energy, and Dynamics and the Science and Engineering Practice of Constructing and Designing Solutions. The testlet incorporates UDL features intended to increase student interest and engagement throughout the testlet.
Test Design Overview and Exemplars from an Precursor Linkage Level Testlet (pdf) highlights a testlet that measures a earth and space science at the high school grade band. The testlet addresses the disciplinary core idea of Earth and Human Activity and the Science and Engineering Practice of Using Mathematics and Computational Thinking. This testlet also incorporates UDL features intended to increase student interest and engagement throughout the testlet.
Universally Designed Assessment Tool
ATLAS staff designed an interactive tool to show how the UDL framework can be applied in the design and delivery of large-scale assessments. Our hope is that assessment designers, developers, researchers, and teachers can use this tool in planning how to incorporate and self-evaluate their implementation of UDL principles in large-scale assessments. Lessons learned from I-SMART helped inform the design of the tool.
I-SMART Scenario-Based Tasks
This series of brief videos describe how I-SMART researchers designed and evaluated a set of scenario-based science tasks with deep application of Universal Design for Learning (UDL) principles.