08.17.15About that Mirage: Reading TNTP’s Report on Teacher Development (Part I)
Note:This is the first of three posts i wrote after reflecting upon TNTP’s outstanding Mirage Report…with apparently dispiriting news about the field’s capacity to help teachers get better. You can find the other two posts here and here.
Part I:
Last week, an organization I admire immensely, TNTP, came out with a report on Professional Development called The Mirage. The gist of it was that PD doesn’t work very well at making people better. The study analyzed the outcomes of PD programs across three large districts and one charter management organization (CMO). (The CMO was not Uncommon, where I work; beyond that, I don’t know which organization it was). In total, researchers surveyed 10,000 teachers and 500 school leaders.
The findings in the three districts were that 3 out of 10 teachers improved as a result of PD and nothing the analysts at TNTP could identify was a reliable cause of (or even correlated with) that improvement. Improvement appeared to be unreliable at best and totally unpredictable, despite millions of dollars in direct and indirect expenses. And improvement became very rare indeed after a teacher’s first few years in the profession. Teachers corroborated these findings: PD, they said, didn’t help them very much. The first twenty-nine pages of the report document and analyze these findings.
Only on about page 30, where the report examined the CMO, did things really start to make sense. By the time I’d finished the report, I had the opposite reaction from the authors: Despite their assertion that they were unable to learn much of anything about what worked in developing teachers, the report laid out a pretty viable analysis of what probably works in helping teachers to improve. (What you might say honors them for their effort by making them better for the rest of their careers). The gist of it is this: workshops don’t develop people; organizations do. Before I tell you more about what they found, and why I think they found it, I want to be clear that in my eyes this is not a charter-versus-district schools argument even though the CMO happened to be the positive example in this study. So please hear me out.
TNTP found that 7 out of 10 teachers improved in the CMO each year. Again, the authors were unable to isolate drivers of growth, but at the bottom of page 30 they wrote: “We did find some differences on an institutional level [in comparing the successful organization to the three unsuccessful ones], specifically a more disciplined and coherent system for organizing themselves around teacher development, and a network-wide culture of high-expectations and continuous growth.” Later they summed it up this way: “Having a meaningful impact on teacher performance over time depends as much on the conditions in which development takes place as on the nature of the development itself.” I might add to that, in my opinion, the nature of the development also matters deeply, but it’s hard to tell if there isn’t a culture of excellence and growth within the organization.
What does that mean? What are the all-important “conditions in which development takes place”?
For starters, I’ve identified some of the findings about the successful organization in the bulleted list below. Again, it is more important to me that this was the one successful organization out of four studied than the fact that it was the charter organization in the bunch. In fact, I would be willing to wager that, if you disaggregated the data by school, within the three generally ineffective districts, you would also find single schools and school leaders who were able to create the sort of conditions and build a vibrant learning culture that led to reliable growth among the adults. It might be easier to do in the CMO, but leaders in districts do it as well. However, that fact that it appears to be very hard to execute that sort of culture across an entire school district is perhaps part of the point. Decisions about PD should be made at the level within an organization where decisions can be made and executed to build positive culture around it.
Here are the author’s findings about the organization (where more than twice as many teachers improved annually):
- A culture of excellence among the adults: The report noted that at the successful organization, “everyone in (the) school community was constantly working towards better instruction and pushing each other to do their best work.” “What’s unique about being at [my school] is that there is always… someone to push you,” the report quoted one teacher saying. “I don’t think I’ll ever be able to stagnate here.” At the successful organization, people pushed one another constantly. There was a clear sense across the organization that the goal was to be great at the most important work in the world.
- Development is embedded throughout the school culture: Workshops are great, but as someone who runs a lot of them I am sure they are insufficient by themselves. They require constant follow-up, conversation and support, especially at the peer-to-peer level. At the successful organization, the report notes, “Each teacher receives weekly observations from his or her coach, followed by a 30-45 minute debrief.” Further, compared to teachers in our other three organizations, teachers in the high performing organization “are far more likely to report opportunities to practice teaching outside the classroom (82 percent reporting “sometimes” or “often” practicing, compared to 17 to 38 percent elsewhere). What’s more, teachers at the successful organization “spend two to three hours every week with other teachers, reflecting on instructional practices… practicing new skills or reflecting on changes to be made next.” Professional development, in short, isn’t something you do outside your daily job; it is the daily job.
- It’s primarily an improvement culture (rather than an evaluation culture): Because of their frequency, the observations at the successful organization were formative… they were specifically tied to getting better rather than evaluation-oriented. Perhaps as a result, the report’s authors found that teachers in the successful organization–were by a wide margin–more likely to believe that observations and feedback were “effective for their improvement.” The report calls this a “culture of continuous learning” and notes that it is distinguished by its vibrancy and rigor.
- Alignment and efficiency: At the successful organization, the report notes, the central office focuses on developing leaders’ coaching skills, providing data to school leaders so they understand what’s happening in the school and supporting them with trainings. But it is school leaders and their leadership teams that do most of the teacher development. And they do lots of it—it’s perhaps their primary job. But one outcome of this is coordination. There aren’t people dropping in from all over the system to give teachers more and more advice. It’s coordinated and streamlined so one or two people give consistent advice on consistent, important and limited topics.
- Investment in Teachers: It’s also worth noting that the successful organization makes an even bigger investment in developing teachers about twice as much according to the report. The best organizations are all about investing in people development.
So for me what the report suggests is this: Developing people is a function of organizational culture, of good training in an organization that’s built to make its people better. If that last part doesn’t happen, it’s not likely that people will get better. To paraphrase my colleague Paul Bambrick, PD has to be embedded in the life of a school.
Workshops are great but most PD happens when we don’t realize it’s PD—when teachers are talking; when they are pushing one another to be rigorous about improving; when they are getting feedback and practicing constantly. And again, while it is the CMO that is doing that best in this study, there’s not much reason all schools couldn’t. In fact, as I mentioned above, I bet that within those districts, many individual schools do.
To drive this point home, tomorrow I am going to share Part II of this post, a video of powerful teacher development happening in a highly successful school. In the meantime, a final caveat: if organizational culture is a primary driver of PD results, then I think we have to consider all of this–both TNTP’s report and my interpretation of it–as conjectural since the number of organizations in the report is actually just four, which is too small of a sample size to draw firm conclusions from.
Hi Doug,
Good post. Pondering your words here. Glad you’re thinking about this one.
You write: ‘Despite their assertion that they were unable to learn much of anything about what worked in developing teachers, the report laid out a pretty viable analysis of what probably works in helping teachers to improve.’
I read the CMO section a little differently. It’s true they choose to describe but not emphasize (i.e., not on Page 1) your point — that it’s about “a more disciplined and coherent system for organizing themselves around teacher development, and a network-wide culture of high expectations and continuous growth.”
However, when they say they don’t know “what worked”….I think it’s at least relevant that they couldn’t find any input patterns in the CMO.
“We wondered if there would be dramatic differences between improvers and non-improvers within the CMO that would point to particular strategies that seem to be having a marked effect on CMO teachers who make greater strides than their peers. But when we compared improvers and non-improvers within the CMO, we found very few
distinguishing features.”
School leaders and teachers in CMOs make choices about cash and time. Some teachers would get X and others would not. Common planning time, Japanese lesson study, additional observations in the form of a “Department head,” visits to other schools, consulting from A-Net, weekly staff meetings to talk about stuff, etc.
I think it’s useful that TNTP points out that even here, they found no signal on what types of choices pay off even WITHIN a healthy organization like that CMO. Don’t agree that this is helpful to shout out, or don’t agree with their narrative?
P.S. FWIW – I struggled a little with the “? minus 2 = two” test that I needed to prove to your blog I wasn’t a robot. Which maybe means you should take my comment with a grain of salt.
Hi, Mike. Thanks for your comment. Supremely non-robotic. Excellent point about the drivers even in the successful organization still being unclear. Possible that studying further successful organizations at teacher dev’t might reveal more… or that in a vibrant culture different things work for different people but, again, the overall ethos of “this [trying to all get a little better everyday] is what we’re doing here together” is still the key… or that the overall ethos is in fact built by the specific types of study and action. that “Common planning time, Japanese lesson study, additional observations in the form of a “Department head,” visits to other schools, consulting from A-Net, weekly staff meetings to talk about stuff, etc.” build the broader culture and the culture makes people better rather than the other way around (the culture makes all those actions and strategies more effective). or… well there are a lot of possible readings. Would love ot see TNTP take the next step and identify a selection set of schools that are successful at making people better and then study what they do that distinguishes them from others.
Hi Doug
Thanks for this summary and analysis, which I found helpful for clarifying my own thinking. As a former teacher and someone who now works closely in teacher development projects here in the UK, I too read the Mirage report with considerable interest.
The point that struck me from the report – and one which chimed with my own research on school-based mentoring in Teach First – was that PD is a highly individualised process which is difficult (or impossible) to reduce to generic formulae or structures. Rather, it is heavily dependent on the nature of the relationship between coach and teacher – and particularly the level of empathy and trust between them. Principals and system leaders should be focused on creating a culture which allows professional relationships with these qualities to thrive – as demonstrated in your video of Juliana and Ashley.
This doesn’t mean that relationships between teachers and their managers needs to woolly or accepting of mediocrity – another important point I picked out of the Mirage report was the implicit indictment of how performance management systems are being used, with teachers all being rated as Good or better – but that coaches and managers need to define their terms when it comes to PD: clear expected outcomes, based on stated actions and assumptions (I’ve found logic models to be helpful in this); measuring progress against these outcomes; and evaluating the impact of PD on what is meaningful – students’ results, numbers progressing to higher study, career progression, retention/wastage rates. I’m naturally sceptical of attitudinal surveys being used to evaluate impact.
Here at the Institute of Physics we do a lot of work supporting local and national networks of physics teachers, and we run a lot of PD activities in schools via a team of external experts (Teaching and Learning Coaches, or TLCs). Recently, we have been developing a coherent articulation of what effective coaching for physics teachers looks like and how to manage coaching relationships. This report, and your comments, will help us to build on this work and also look at how we can engage and mobilise the whole-school culture – an ‘enlightened professionalism’ – required to facilitate these relationships. As we’ve learned with the issue of female participation in physics, supporting good practice in physics teaching is an issue which reaches beyond the science labs.
So, thank you – I wanted you to know how important I found your post, and that it will contribute to internal discussions at the IOP. We will be turning to our partners – high five – and…
All the best
David
Thank you, David. Nothing could make me happier than knowing that the post was useful to you in your important work. And can I say that several points you make are really critical… especially this one:
“This doesn’t mean that relationships between teachers and their managers needs to woolly or accepting of mediocrity – another important point I picked out of the Mirage report was the implicit indictment of how performance management systems are being used, with teachers all being rated as Good or better.”
You point out, rightly, that the work is too important for there not to be an ethic of excellence behind it all… but I think great schools (and organizations) manage to communicate that it’s our goal together to make sure you are 1) your best version of yourself in the classroom and 2) successful in a way that aligns to our organization’s ethos and approach and that while in the long run you share responsibility for your response to the support you receive, WE have a responsibility to make that support warm and positive and of high quality … and to make it safe for you to struggle along the way. That to me is key. If you are afraid of getting it wrong in the short run, you will never get it right in the long run… so all those weekly interactions have to assume that teachers will be imperfect and allow them to be comfortable with that part of growth.
Anyway thanks for your insightful comment.
–Doug
Thanks for this, Doug.
A few days ago, I was working with a district in MI. Our team was leading a ‘workshop’ for school leaders in which we shared best practices for providing specific, observable, and bite-sized feedback to teachers. We shared some videos followed by small group discussions and role plays, and folks were pretty engaged.
Mid-way through the training, a gentleman who we had not seen before raised his hand and said: “do you have videos of teachers in public school districts? For example, a veteran teacher and a young school leader? Because this is not comparing apples to apples”. We said we didn’t. The examples were from Paul Bambrick’s book. With that, he stood up and left.
It turns out the gentleman was the district’s Director of Culture. A while later, one of the school leaders said she had to speak up. She challenged the gentleman’s statement and invited her colleagues to look beyond the ‘us vs them’ approach between traditional schools and CMOs. She said that good teaching and good coaching looked the same, regardless of where you taught.
Your reflection about how it’s not workshops, but organizations that develop people couldn’t be more true. It’s not really about the technical skills, it’s about people’s mindsets, about what it means to grow and to be comfortable with the struggles inherent in becoming a better teacher, a better school leader.
thanks, JP. We’re very comfortable talking about growth mindsets for the kids. but your point–which i agree with–seems to be that growth mindset among the adults matters more. and it doesn’t happen by accident.