Q. How is evaluating arts growth different from measuring academic achievement?
A. Measuring arts performance can be challenging, because there are concerns about whether an evaluator can be objective. Yet, if you think about it, teachers have always assessed student writing and essay tests, which is much more difficult than grading multiple-choice questions. In all areas where subjectivity is at play, inter-rater reliability (ensuring that various “judges” assess performance in the same way) is crucial. Assessments in the arts often examine students’ background knowledge as well as their performance skills.
It’s widely thought by researchers that arts education not only strengthens students’ art skills specifically but helps students acquire a set of mental habits that can be transferred to academic subjects such as English language arts (ELA) and math. A number of the projects Metis is evaluating aim to impact on these so-called “pro-cognitive skills,” so we have developed a range of tools to measure those related to “habits of mind” as well as actual arts learning.
Q. Let’s talk about measuring actual arts learning. How do you go about evaluating art?
A. Our work with Arts Achieve offers some examples of how it can be done. Arts Achieve is a large project funded by two U.S. Department of Education grants—one from the Arts in Education Model Development and Dissemination program (AEMDD) and one from the Investing in Innovation (i3) Fund. It brings together the New York City Department of Education (NYCDOE) Office of Arts and Special Projects with five premier cultural organizations whose consultants and coaches partner with arts teachers in more than 40 elementary, middle, and high schools. Metis is overseeing a rigorous evaluation of this program, which involves assessments in visual art, music, dance, and theater. The assessments align with grade-level benchmarks for learning that were established by the NYCDOE and published in the Blueprints for Teaching and Learning in the Arts. In addition to direct arts learning, we’re examining whether there is a transfer of skills to students’ academic performance.
During the first school year of this five-year project (2010–11), teams from the NYCDOE and arts organizations collaborated to develop the assessments and the rubrics that adjudicators use to assess students’ performance on a variety of tasks. Metis is contributing to this process by assisting with the establishment of inter-rater reliability. To ensure agreement in the ratings, adjudicators participate in trainings where they watch videotapes of student performances and then rate them together using the rubrics. We help them to talk about what to look for in the work and discuss any disagreements. At times, the rubrics are changed so that everyone can agree on how to use them. These methods have attracted the interest of administrators at the New York State Education Department who are interested in issues of inter-rater reliability in performance assessments in all academic areas and of arts educators who are eager for valid and reliable tools that can be used in future evaluations of arts-education programs.
Q. The issue of assessing the transfer of pro-cognitive, arts-related skills to academic subjects is controversial. Why?
A. Some arts professionals believe that learning in the arts should be a goal in itself—that we should not have to prove any carryover benefit beyond these disciplines themselves. But, in the past few decades, there has been growing recognition that development of certain mental habits—which are often taught through arts subjects—is crucial for success in higher education and careers. Although the exact list of habits varies, the common denominator is that students need to think independently, to critique and value evidence, to collaborate, and to be creative—all skills that are reflected in the Common Core State Standards. Metis is looking at these skills in the four evaluations it is conducting for AEMDD-funded programs and in other studies to understand whether the “soft” or “pro-cognitive” skills conveyed in arts instruction have an impact on academic learning, and if so, how?
Q. How do you go about assessing this transfer from the arts to academic achievement?
A. One example can be seen in Metis’s evaluation of Framing Student Success: Connecting Rigorous Visual Arts, Math, and Literacy Learning, which linked New York City public schools with local artists to help schools integrate high-quality visual arts education into math and ELA curricula. The program was provided to students in grades 3 to 5 in three high-poverty, low-performing public schools. Metis collaborated with Studio in a School to develop a Studio Habits of Mind rubric based on the work of Project Zero, an educational research group at the Harvard Graduate School of Education that promotes arts as a central part of education. The group has broken down the kinds of thinking that art teachers can promote. Metis’s assessment of Framing Student Success used a five-point rubric to measure progress in five of these habits, including students’ ability to engage and persist in a project for a sustained period of time; envision images that will help guide their work; observe closely their own work and that of others; reflect on their process, intentions, and decisions; and stretch and explore— trying new things, taking risks, and learning from their mistakes. Twice a year, teachers in this program are using this rubric to assess their students along with these indicators.
Metis also collaborated with the Eastern Suffolk Boards of Cooperative Educational Services (BOCES) to develop a similar checklist for the Creative Classroom Collaboratives (C3) project, a four-year arts integration program for students in second through fourth grade that is funded through an AEMDD grant. This program involves teacher collaboration with outside artists and training in “aesthetic education” and “21st century learning models.” The students participate in a number of activities, including viewing dance, music, storytelling, and theater performances and engaging in performing arts activities linked to ELA and 21st century skills. The checklist that Metis developed enables teachers to assess their students’ 21st century skills, such as creativity and innovation, collaboration, critical thinking and problem solving, and communication. These are the types of skills that have been theorized to transfer to academic areas. The first participants are currently in third grade, and Metis plans to examine their state test scores this year in relation to the scores of students in similar schools. Metis also will examine whether there is a link between improvements in pro-cognitive skills and academic progress.
Q. What do you think are the most important next steps for the field?
A. Metis is working on a number of arts-education projects, and we’ve developed some really good tools that address key questions. We will definitely be applying these tools to future undertakings. One of the central questions is establishing inter-rater reliability. Multiple-choice tests are easy because there are right and wrong answers. However, if you’re assessing a piece of art or writing, it becomes very important to build solid rubrics and to establish inter-rater reliability.
We also need to clarify how arts skills are connected to pro-cognitive or soft skills and measure them reliably. Overall, we need more data to document the importance of arts education in children’s development. Should we concern ourselves with a carryover of skills to the academic context, or should we be more concerned with showing the importance of arts education in its own right? And, does arts education support college and career readiness? Arts education is more than making art; it is about being a good audience member, building confidence, exploring creativity, and trying new things—in short, being a well-rounded person.