Upon reviewing the results of a new district-wide exam designed for underclassmen, a department head found that the exam and its results contained errors.

On March 8, English department head Bryan Ritter was in “disbelief and disappointment” when he received a report indicating that Lowell freshmen performed below the district average on the February 21-March 2 Common Learning Assessment. According to Ritter, the report that indicated “only three tenth graders and none of the ninth graders had passed within the district benchmark” caused him to look more closely at the test, where he noticed that several questions were obviously marked with wrong answers.

According to John Burke, a supervisor in the San Francisco Unified School District Achievement Assessments office, the incorrect answer key was due to human error. Later that day, the district fed in the corrected answer key and ran new results. This time the Lowell CLA-takers scored above the district average.

However, even with the updated results, Ritter noticed that 6 of the 25 questions, or approximately 24 percent of the ninth grade English exam, did not have clear answers, making more than one choice a possible response. “There was so much ambiguity in the questions and their passages that two answers were acceptable,” Ritter said. In addition, Ritter said that 2 of the 26 questions on the tenth grade English CLA had clearly incorrect answers, leaving that assessment’s data with a result error of 7.6 percent.

This is the first year the CLA was administered across the district, as before the district piloted a test titled MAP (Measuring Academic Performance). The intent of the CLA is to help schools plan a curriculum that meets California educational standards and measures student progress. “The CLA covers Math and English Language Arts for grades 2-12 and Spanish Language Arts for grades 2-8,” Burke said. “These assessments are intended to be one of many measures teachers use to assess student performance.” According to Burke, students can access results from the CLA to help them develop lesson plans. “The results are available via DataDirector, the district’s online data management system, allowing teachers to analyze class performance and individual student responses by utilizing the available reports,” he said.

While the CLA was intended to give teachers an idea of their students’ academic performance, Ritter said that “it has not provided [him] with any information on how to better teach [his] students; it has simply caused frustration to correct the exams. We’ve also lost class time by giving them.” According to Ritter, a “properly designed” test would be helpful. “[The quick turnaround] is nice because the test is evaluated in two weeks, so we can use that information in the classroom. If any students didn’t understand a concept, I could focus a little more on that,” he said.

While the English assessment, a multiple-choice test, has reduced its usefulness due to content errors, it was also missing a key component — a writing sample. Ritter pointed out “a writing sample would have made the CLA a more ‘authentic’ assessment.” That assessment of writing ability was not part of the CLA because the company had difficulties coming up with a grading rubric, according to Ritter.

The CLA was created by a private company called Intel-Assess, whose goal is to “provide results that drive student improvement.” According to Burke, the exam is created by teams of educators from SFUSD in partnership with Intel-Assess to assess the progress of students towards mastering California standards. “SFUSD contracted with Intel-Assess based on their ability to meet our assessment needs, including working with teachers and content specialists to customize the assessment content, their ability to provide assessments translated into Spanish, their ability to provide modified assessments for special education students and the overall quality of their product,” he said. Although some districts, such as Los Angeles, utilize test results in judging their staff, the CLA is not used for teacher evaluations.

According to Burke, the 2011-12 contract with Intel-Assess cost $210,000. “This is approximately $5.00 per student for 42,000 students in grades 2-11,” he stated.

However, Ritter expressed the belief that the district should not be paying a private company in light of SFUSD’s current budget crisis. He deemed the CLA “poorly spent district funds” that should be spent on books instead. “What the companies are producing is substandard; it’s corporate education and it’s not effective,” he said.

Instead, Ritter advocates for a test created by teachers, or at least a review by department heads before the test is administered to students. “The district can do a lot better,” Ritter said. “There are only 25 or 26 questions on the CLA. A good teacher can do that in a weekend; it could be done quite easily.”

However, Burke stated in an interview that the district does accept input from teachers. “SFUSD department heads set their own agendas. We welcome feedback from them. We are open to having them more involved in the development process if that is something they are interested in,” he said.

English teacher Jennifer Moffitt commented on the first version of the CLA (which was not administered to students because it had plagiarism issues). “I was really shocked by the test including a passage that any of us from the department would call plagiarized because the test company hadn’t cited its sources of information,” she said. “I think it’s really hard to teach the importance of citing sources correctly when the district’s materials model something like that. I’m glad that they did correct that, but the fact that their initial version had that sort of material is a red flag to me.”

Although according to Burke, the CLA did “undergo a continuous improvement process which includes feedback from teachers,” several other English teachers were not impressed with the English Language Arts portion of the exam. Ninth grade teacher Timothy Lamarre, who administered the exam to his freshmen English class, thought the CLA “could’ve been reviewed and refined more before it was actually distributed to the students. It seems like the test was rushed and wasn’t properly prepared.” Another ninth grade teacher, Anne Freeland, does not plan on using the CLA’s results. “I don’t feel like it’ll inform my instruction enough to utilize it; the test does not align with my classroom standards,” she said.

Similarly, ninth grade teacher Nicole Henares will focus on other assessments she has in place. “The test was so poorly constructed that I don’t feel like it’s an accurate assessment,” Henares said. “I seriously question the validity of a test that had so many mistakes, was so poorly constructed, so hastily thrown together that the first test results were a mistake.” Henares added that she prefers portfolio assessments, which include compare-and-contrast essays, etc., to exams such as the CLA to give students the feel of a “real world” test.

For the 2012-2013 CLAs, Burke said that the “SFUSD will continue to incorporate feedback from teachers as well as have teachers collaborate with Intel-Assess to design the assessment.” According to Burke, there are several methods for teachers to provide input on the exam. “We have a feedback form for teachers to provide specific feedback, an email address for anyone to correspond with us, several focus groups to collect information about both the administration and content of the CLAs and an online survey at the end of the year. In addition we worked with teams from the curriculum department to design/redesign the tests during the summer. The nature of the feedback affects how and when any changes or edits are made to the tests,” Burke said.

The math department administered the math CLA this year for the first time, having given the MAP test last year. Fortunately, math department head Thomas Chambers has not noticed any errors on the math CLA and its data. However, he does think that district funds and time spent on assessments such as the CLAs could be spent more wisely. “With the furlough days, it is really a shame that we need to give up additional days to district assessments,” Chambers said. “I do believe the time could be better used for instruction and the funds be better used to prevent teacher layoffs and/or reduce class size.”

According to Ritter, the state is currently designing a standardized test called Smarter Balanced. CTB, a publisher of educational assessment solutions, “and its partners will research innovative item and task formats to ensure that the assessments provide authentic information that can help teaching and student learning,” according to the Smarter Balanced website. According to the Smarter Balanced Assessment Consortium website, CTB/McGraw Hill will lead a team of assessment experts from the American Institute for Research, Data Recognition Corporation, The Council for Aids to Education, HumRRO and the College Board in developing the first set of exams. The California Department of Education’s website, states that the intent of the exam is to “meet federal and state-level accountability requirements and provide teachers and parents with timely and accurate information to measure and track individual student growth.”

The new assessment will replace the CLA and certain subjects, such as English, in the California Standardized Test, where, unlike the CST, a writing sample will be included. The vision for Smarter Balanced includes it being taken with a computer program that will adjust test questions according to a student’s responses to minimize the amount of time spent on the test.

With the plethora of assessments and standardized tests the district and state impose upon its students, Ritter advocates that sufficient information would be gathered if a simple exam were administered once each semester. “Teachers are already continuously evaluating students in a variety of ways,” Ritter said.

According to Burke, the CLA is going to be administered two times a year at the high school level. Smarter Balanced will be administered to SFUSD students beginning in spring of 2015.

Ilustrations by Hoi Leung.