|
21st Century
Schools
|
Rings Around Saturn
The story of evaluating St. Paul's Saturn School -- told here from three perspectives -- is a lesson in the difficulty of school change
Hallie Preskill, D. Thomas King, and J. Michael Hopkins
condensed from The Executive Educator
May 1994
Shortly after the Saturn School of Tomorrow opened in St. Paul in 1989, then President George Bush congratulated us on reinventing the concept of public education. He visited our school, invited one of us to the White House, and talked about our efforts in his speeches.
All of the attention was great fun, but it also made us a little nervous. We knew, from the beginning, that Saturn was an experiment -- a work in progress. We expected mistakes, but we wanted to learn from them. So we set up a system of recording our journey -- stumbles and all.
Although many people have watched, copied, or rejected some of the education approaches we used in the Saturn School, little attention has been paid to our evaluation system. We believe what we learned in this area is important to educators considering any sort of sweeping change.
What follows is a record of our experience from three different perspectives: that of the planner (Tom King, who envisioned the broad concepts of the Saturn School): that of the implementer (Mike Hopkins, who was on-site lead teacher when the school opened); and that of the evaluator (Hallie Preskill, the evaluation consultant and education professor who recorded the progress and pains of the school in its first three years).
King: The planner's perspective
As the person responsible for the initial planning of the venture, I was excited about the prospects for change. The Saturn School of Tomorrow was named for the challenge offered by Al Shanker, president of the AFT. In the late 1980's, this plain-speaking union leader traveled around the country calling not for more money or better tests, but for fundamental change in America's schools. Unable to find any school in this country that came close to his vision, he called on educators to change the way children were taught in the same way the Saturn automobile manufacturers had changed the way cars were made.
As a former math teacher, I shared Shanker's outrage that four-fifth of our students are not successful: One-fifth drop out before graduation, and three-fifths seem to get through school by the seat of their pants. Inspired by one of Shanker's Saturn speeches and armed with some of my own ideas for change, I secured permission from David Bennett, then superintendent in St. Paul, to create a Saturn School that would personalize learning, professionalize teaching, involve the community and the parents, and take advantage of the cutting-edge technologies available. We formed a planning committee that included the superintendent, the local union president, the deans from local universities, and several speicialists in educational technology. The planning group envisioned a very different school, one that would be committed to the success of each student. To achieve this, we planned to use teacher teams, cooperative and project-based learning, site-based decision making, parent involvement, personal development plans, portfolios for each student, and extensive technologies for students and staff. We also planned to use community resources (museums, libraries, local businesses, and so on) for off-site learning and mentoring activities.
The planning team drew up blueprints for a city-wide downtown middle school, ungraded but serving children who normally would be in the fourth to eighth grades. The plan called for an extended school year and the latest in technological equipment -- all in an eight-story building that had been gutted and redesigned to meet Saturn's needs.
From the beginning, the team believed a comprehensive evaluation had to be an essential part of the Saturn project. We planned to document carefull y what we were trying to accomplish and make necessary adjustments along the way. We wanted to create a record of what worked and what didn't, to inform ourselves as well as other educators contemplating change. That reasoning appealed to a local foundation, which granted us funding for a three-year formative evaluation.
The thorough, systematic evaluation would turn the
Hopkins: The implementer's view
As the person charged with implementing the Saturn project, I was in for a lesson in the difficulties of turning theory into practice. The original plan called for a year of staff preparation before the opening of the Saturn School. But parents were so eager to enroll their children that the district agreed to skip the year of preparation. I was hired six weeks before the school opened.
This led to predictable problems. Although the planning team had spent a great deal of time mapping out Saturn's future, the teaching team did not have enough time to formulate how these grand plans would be carried out. Many of the planning team's ideas were given short shrift as we scurried about, getting ready for the first day of school. (Lesson learned: The people who are to implement a program should be in on planning the program.)
Time continued to be a problem after the school opened. We had four teachers and four interns working with 162 students. Because wehad promised every child personal counseling on a continuing basis, that meant each team teacher served as adviser to 40 kids. Add to that the challenge of creating an entirely new, personalized learning plan for each child. We were often at the school until 9 at night, planning activities for the next day.
But we understood the importance of evaluation and made time for it. Hallie Preskill led an evaluation team that took on the job of observing, appraising, and reporting progress and problems at Saturn. We knew the staff, other educators, and the St. Paul community would be awaiting the results.
During the first year, the Saturn staff met regularly with Hallie, who was patient with staff members and actively sought everyone's ideas on evaluation issues. She attended the weekly leadership team meetings, which usually had full agendas. With all the pressing issues of planning and implementing a new school, though, we often found it difficult to schedule discussions of the evaluation process.
I met with Hallie by myself every other week for the first two years of the evaluation. Looking back, I realize these meetings were useful in terms of conveying information, but they painted a picture from a single perspective. This had the unintended result of reducing the staff's trust in the evaluation.
The Saturn community naturally awaited Hallie's first annual report with both fear and anticipation. Although she occasionally had given us a little feedback, we didn't know quite what to expect. When the report came out, we found she had followed the traditional path of an evaluator: she had collected data and compiled it into a report without offering any judgment.
But a local newspaper certainly offered judgment. Based on that first year's report, the editor assigned a letter grade of C+ to the Saturn School.
This, along with reports of falling scores on standardized math tests, made some parents anxious. Some pulled their children out of the school, and others needed considerable reassurance.
Staff members felt enormous stress. They had signed up for a grand experiment in education, and now they were finding themselves the objects of public criticism. Unfortunately, the staff had no established way of responding to the criticism.
Reacting to the pressure, some staff members began to doubt themselves and each other. After that, the staff had trouble forming the strong community we had hoped for. It is significant that every one of the original teaching team left the school. (Another lesson learned: Prepare staff members of any new venture for criticism, and give them constructive ways to cope with it.)
Looking back, I understand how important it is for organizational structures to support the development of trusting human relationships. As the outside observer who knew us best, Hallie had excellent insight into problems developing at Saturn School. But the traditional role of evaluator forced her to maintain the detachment expected of an objective observer. That is a shame because we could have used her input in our decision making.
Perhaps the most important lesson learned: New organizations need "critical friends" even more than they need objective observers. A critical friend could remind program leaders of their mission, point out areas of strength, and offer suggestions for improvement.
Preskill: The evaluator's vantage point
As the person responsible for evaluating the project, I approached my assignment with relish. Evaluating Saturn was a evaluator's dream -- a chance to put to use everything I knew about evaluation and to grow alongside a group of people seeking to transform education. I had been a program evaluator for 12 years, but my involvement with Saturn was the catalyst for believing the evaluation practice itself also needs to be transformed.
One incident set the stage for the way my role at Saturn was to develop. Early in the first year of the Saturn evaluation, an intern heard another staff member make a racist remark. The staff was already under a great deal of stress, and I knew someone should step forward who was skilled in conflict resolution. No one with those skills worked in the school, and for a brief moment, I thought I might be called on to deal with the situation.
Later, one of the team teachers called and we decided, after much discussion, that I should remain the "neutral observer" of the school and not become "part of the problem." My initial reaction was relief: I knew this was a difficult situation and I was glad I wouldn't have to "fix it."
But as time went on and more issues came up, I came to see that the evaluator's role needed to develop alongside the school's. The school did indeed need, as Mike says, not just a neutral observer but a critical friend.
In schools and other types of organizations that are building new structures and processes, I've decided, evaluators should interact continuously with the people who are developing the program. This view redefines evaluation and puts it at the core of the development effort.
If I were evaluating Saturn again, I would collect much of the same data -- thousands of pages of transcribed interviews, field notes from my weekly observations, and about 400 photographs -- but I would approach the project differently. I would try to make myself the "evaluation-facilitator" rather than the "evaluator".
I would begin by engaging staff members in a conversation about their previous experiences with evaluation, their expectations for this evaluation, their theories about evaluation, and how this evaluation could serve their particular needs. We would talk about using the evaluation to create a true learning organization, and we would develop a process for accomplishing this goal through reflection, dialogue, and action planning. In addition, we would meet regularly to discuss the evaluation findings.
This process would help the staff develop a common language and shared assumptions about the school. Just because the staff members seem to agree on the core mission of the school does not guarantee they will agree on how to achieve their goals. Indeed, the Saturn staff's varied assumptions and beliefs about these goals became a major issue of controversy during school's first three years. Evaluation should raise these critical issues and provide the opportunity for discussion and resolution.
Because of the way my role had been defined, I saved most of my observations about Saturn for the annual reports, which ran between 70 and 90 pages and were made available to the entire community. Even though I emphasized I was reporting on a program in progress, many people -- and, in particular, the local press -- took my reports as definitive conclusions. The result was unfair treatment and premature judgment of the school.
If I were doing this again, I would write periodic executive summaries of no more than 20 pages. I would give the staff much more feedback through discussion, photographic essays, brief memos, debriefing sessions, and, perhaps, quarterly reports of no more than five pages. This would allow the staff to refine the program without the glare of publicity.
Evaluation, I've decided, should be an ongoing process that engages all key stakeholders, improves a school's chances for success, and empowers its staff members to build trust and respect, positive interdependence, and personal responsibility.
Final Thoughts
With barely a compass and a few charts describing this new planet of learning possibilities, a small staff of brave and bright educators launched out to discover Saturn. Often feeling like a goldfish in a bowl with the world peering in, we found some observers wished us well and others hoped we would fail.
We believe both wishes came true. Saturn is still serving students in St. Paul, but not quite in the way we had envisioned. Many of our good ideas remain in place at Saturn or have been adopted in other schools around the country.
Progress comes from both successes and failures. The challenge of evaluation is to use the latter to increase the former. If an evaluation is well conceived, implemented, and reported, greater success will come to those who study, adopt, or adapt the findings. We recommend educators rethink the role of evaluation so it serves those closest to the program. Used properly, evaluation can stimulate dialogue and reflection. It might even help us create better schools -- schools that give each student a better chance to succeed.