An Interview with Dr. Gretchen Suess: Data Collection and Evaluation for School-Based Programming in Philadelphia

Posted by on June 30, 2011

By Liz Shriver and Gretchen Suess

Pursuing the gathering of “good data”, that can help a non-profit or higher education institution to analyze and evaluate the students affected by a program or project within in a Philadelphia School District school can be challenging. Attempting to create a collaborative data collection effort with several projects or programs operating within a school in order to have a more holistic view of how each student within that school is being served (or not served), is even more challenging.

Logistically and ethically, collecting data about school programs requires time and dedication specifically to that effort. Dr. Gretchen Suess, Director of Evaluation for the Netter Center for Community Partnerships at the University of Pennsylvania, and a team of undergraduate and graduate school research students have been working on this process. Their ultimate goal is to be able to easily identify how all students are served in each of the schools they work in regardless of whether they are part of a Netter Center program or any other special program provided in the school.  By working towards this goal each program coordinator within a school will be able to improve collaboration in order to serve teachers, students, and families better. This aligns with the goals of the Netter Center to help create schools that are truly “community based” and with the School District’s stated goal of serving all children equally.

Q: Gretchen, what is your role at the Netter Center as a data collector?

A: To work nationally with other researchers and locally with our staff and partners. To focus on the institutional gaps that would allow for our programs to be sustainable. To make sure all of our data is “cleaned up” and holistic. I work with program staff to complete reports to funders, though it’s all internal for partners now. Our project coordinators (the school-based staff) complete that piece. I also manage the student teams that collect program data.

Q: How do you build a collaborative effort in collecting data that involves other programs as well as school investment?

A: The emphasis on our team is to value evaluation over research. The goal of our evaluation is to help students in the schools we work in overall. This means that it is helpful for us to know which students in our programs are also involved in other programs; if particular students gravitate towards certain programs or are served well by them, and if there are a group of students that aren’t being served at all, then we should all know that.  That kind of evaluation requires good research, but our primary goal is to put the data to use for quality improvements.

The feedback from principals was that often these (Netter Center) programs are perceived as only for “some kids”. This phenomenon is inevitable when we are doing programs without doing any kind of targeted recruitment using data.

As an example, at Sayre High School there are three different college-prep programs involved with various students in the school in addition to ours. We are trying to ask how all of us can set out to serve targeted groups of students in the school using all of our data. Ie: using our data strategically. The meaning of a “University-assisted community school” is not that Netter provides programs to all students or that any one non-profit provides programs for all students at a school, because we know that’s not logistically possible. Our goal is for each organization and program to provide services to a targeted group of students to their capacity so that all students are served as equally as possible with the resources available.

In order to reach that goal we are creating a “relational database”, so that we can have a space to show exactly how students are being served by programs.

We’re lucky because the Netter Center has staff at all of our schools. So my initial goal coming in wasn’t to start gathering data from scratch, but to figure out what we have and what our short and long term goals are in making our data collection and evaluation processes more thorough.  I also needed to know if there were any conflicting goals before we starting evaluating anything.

Q: How would a University go about building a data collection and evaluation process from scratch?

A: First, there needs to be a collective vision and a clear plan on how you’re going to get there. You have to create a logic model, think about all the factors. What would help to sustain your program and what would throw it off track? For example: Is the program grant dependent? If so, how can you build sustainability within the school so the program will continue when the grant is gone? You also have to have a back up plan. Can you run things differently if any of the factors that help the program function go away?

At the Netter Center we needed a better needs assessment for students and a better baseline for the school and program to work with. Also, we often have faculty with specific resources that a school could use, and the school has specific needs that have to be met. However, the resources given and the needs met don’t often match up. One of reasons this happens is that we still often use surface-level data.

One way that we can prevent a mismatch of resources given vs. resources needed is by asking (at a given school), if there are other evaluation efforts that are already happening, and what we can do to work together better.

Q: How can “data collection” be a school or community effort?

A: This year we worked with four high school interns who were in our programs at University City High School and Penn students in my anthropology course (ANTH318) to help consolidate and administer multiple school-wide surveys so that students and teachers weren’t over-burdened with data collection. To do this, we worked with two other institutions/funders that needed data collected. We managed to merge all of our needs in to a common survey so that not only could we collect data efficiently, but we could also consolidate our data so we had the same findings. Everyone has to give and concede a little in a situation like this. No group can dictate, or be in charge. This worked because students were receiving only one survey, instead of three, and we could pool our resources to give them incentives to take it etc.  As a thank you, we served a healthy breakfast of granola bars and honey tangerines (partnering with our Agatston Urban Nutrition Initiative) to all the teachers and the Academy that returned the greatest number of surveys.  We then also gave students de-identified data sets to work with so they could help generate findings to share back with our staff and the school. This gave students and teachers a chance to see and talk about the data coming from their school. In this way our data collection became highly collaborative, and without breaching confidentiality. It was a village approach to get the survey done. The teacher leaders, administrators, program staff and students were all invested.

There are still huge challenges to gathering good data. The rate of change at the district is fast and constant. Every school needs different things, so they have to be different and every partnership is unique and needs to be flexible. We know this, but data collection can also be another way to build support for our programs and to help sustain them; however, we need to all work together.

Q: How can high school students and college students be involved in creating and gathering better data at school-based programs?

A: Classes (like a statistics class) with a dedicated professor could start by putting together a data collection map at each school. Meaning that they could find out what all the different programs and funders are at a school and what their data collection needs look like. The students could then bring these people together, and attempt to have them work collaboratively. For example, even a simple issue can be huge, like using the same variable names. We should all default to what the school district calls each variable name. Also, we always work hard to be good partners with the district. When collecting data, we do so for evaluation not for research to be published. That isn’t the particular purpose of our office, which is another reason we are able to work collaboratively. We still have a lot of work to do and a very long way to go, but are working with the district because eventually we would like to publish. But we’ve seen what amazing things can be accomplished when we truly work together!  The process work much better and ends up being useful not just to us, but hopefully to everyone for the benefit of our West Philadelphia schools and families.