The Minnesota Department of Education should be more consistent in its oversight and administration of Q Comp.Key Recommendations:
Implemented in 2005, Q Comp is an alternative teacher compensation and professional development system. Q Comp includes performance-based pay, alternative teacher salary structures, on-the-job professional development, teacher evaluations, and additional career opportunities for teachers.
Participating school districts and charter schools are eligible to receive additional revenue of up to $260 per student. In fiscal year 2009, Q Comp participants will receive an estimated $49 million in state Q Comp funds.
In the 2008-09 school year, the 72 school districts and charter schools participating in Q Comp will collectively receive an estimated $49 million in state Q Comp funds.
Participation in the program is dominated by large school districts in the Twin Cities Metropolitan Area. Out of almost 500 school districts and charter schools in the state, only 72 are participating in Q Comp during the 2008-09 school year. Barriers to participation include resistance from teachers and concerns that the Legislature will stop funding Q Comp in the future.
We sent questionnaires to administrators and teachers across the state to learn their opinions of Q Comp. Superintendents and charter school directors in Q Comp schools expressed positive views about the program. Teachers, in contrast, expressed more mixed opinions. Over 80 percent of administrators in Q Comp settings agreed or strongly agreed that Q Comp had improved classroom teaching and will lead to increases in students’ performance on standardized tests at their school. Less than half of teachers in Q Comp settings who responded to our questionnaire felt similarly.
Teachers in Q Comp settings expressed more positive views about the program than teachers in non Q Comp settings. However, teachers who responded to our questionnaire, regardless of whether they were in Q Comp settings, had mixed views regarding the effect on their school of reforming the steps and lanes salary schedule.
There is not sufficient evidence to determine the impact of Q Comp on student achievement.
Q Comp is a relatively new initiative. Only 11 school districts and charter schools have participated in the program for three years or more. As a result, there is a limited amount of data available to assess student achievement in Q Comp districts and schools.
School districts and charter schools choose whether to participate in Q Comp. The program’s voluntary nature, however, makes it difficult to draw conclusions about the effectiveness of the program. School districts and charter schools that have implemented Q Comp are likely different than those that do not participate in the program. Even if one could show that changes in teacher retention or student achievement were attributable to Q Comp, it is not clear that similar changes would occur if Q Comp were implemented in other districts or schools.
Furthermore, it is difficult to disentangle the effects of Q Comp on student achievement from other initiatives in a school. Any changes in student standardized test scores could be due partly or entirely to the effects of other programs.
Finally, research on the relationship between student achievement and alternative teacher compensation models similar to Q Comp has not demonstrated a conclusive link.
MDE holds current applicants to different standards than earlier applicants.
MDE has revised its Q Comp application process each year since the program’s inception. Because of these changing requirements, Q Comp applicants have been held to different standards depending on when they applied to the program.
For example, one school district that has participated in Q Comp since 2005 did not have schoolwide student achievement goals in its application. However, MDE requires current Q Comp applicants to establish schoolwide goals based on standardized assessments. Teachers’ performance pay must be based, in part, on whether these schoolwide goals are met. Another school district that has participated in Q Comp since 2005 only has two (rather than the currently required three) evaluations of each teacher. MDE told us that these participants’ applications would not be approved if they were submitted today.
The Minnesota Department of Education does not require all participants to comply with current Q Comp standards.
In the meantime, some recent applicants have had to revise their applications or have had their applications rejected because they did not have sufficient schoolwide goals or they did not have at least three teacher evaluations. In cases where these applications were denied, the applicants are not currently eligible for the $260 per pupil in Q Comp funding. This is in direct contrast to those applicants who entered Q Comp early, are not complying with current MDE Q Comp requirements, and are receiving the additional Q Comp funding.
While it is reasonable for MDE to revise Q Comp requirements as it gains more experience with the program, MDE does not have a method for holding all participants to its current Q Comp standards. MDE does not systematically review previously approved applications to ensure that all participants meet the current standards. We recommend that MDE develop a process to apply changes in Q Comp requirements to all program participants.
The Q Comp application review process has several weaknesses.
Short timelines and MDE’s external review process have weakened Q Comp application reviews.
By law, MDE must convene an application review committee within 30 days of receiving a completed application. MDE has interpreted this requirement to mean that the department must respond to all submitted applications within 30 days of receipt.
The short timeline is problematic when MDE schedules sessions to review Q Comp applications. Department staff said that sometimes the application readers are not able to review the materials before they arrive at the department for an application review session. Additionally, the 30-day timeline makes it difficult for MDE to revise its application requirements because new applications could be submitted at any time.
Minnesota statutes require the use of external application readers in the application review process. We observed review sessions where application readers were not prepared or missed information presented in a Q Comp application. In one session we observed, it appeared that none of the application readers had read all of the applications being considered that day. In two of the application review sessions we observed, external readers missed key elements provided in the applications.
Different application readers review and comment on different Q Comp applications depending on schedule constraints and other factors. Having different people review Q Comp applications has led to inconsistent expectations for Q Comp applicants. Also, because the external application readers (and not MDE staff) recommend Q Comp applications for approval, technical assistance provided by department staff to Q Comp applicants has at times been limited. MDE staff must anticipate issues that the application readers will identify, which they have done with mixed success.
We recommend that the Legislature no longer require MDE to review Q Comp applications within 30 days of receipt. Instead, the Legislature should require MDE to set deadlines for Q Comp applications at two or three specific times during the year. We also recommend that the Legislature no longer require an external committee to review Q Comp applications. Instead, application review and approval decisions should rest with MDE. The department would have the option to solicit input from external application readers, if it wishes.
Some Q Comp participants can choose the amount of oversight they receive.
MDE does not provide regular oversight of all Q Comp participants.
MDE conducts annual program reviews of most Q Comp participants. These program reviews consist of a visit to the participating school district or charter school by a team of outside evaluators and a document review by MDE staff.
The evaluation team typically consists of an MDE representative and two or more representatives from other Q Comp programs. Using a standard protocol developed by MDE, the team evaluates each component of the participant’s Q Comp program.
Once a participating school district or charter school has passed two program reviews, it may choose whether to participate in the program review process. These participants essentially customize their review—they can choose to have just a site visit, just a document review, or no review at all. None of the five Q Comp participants that had reached this level chose to undergo any part of the program review process in 2008. MDE also does not conduct program reviews for schools that use the Teacher Advancement Program (TAP) as their Q Comp model. Instead, the department relies on the oversight conducted by the National Institute for Excellence in Teaching, which sponsors the TAP program.
We recommend that MDE conduct periodic program reviews of all Q Comp participants. The department should extend some oversight to all program participants.
Summary of Agency Response
In a letter dated January 16, 2009, Minnesota Department of Education Commissioner Alice Seagren said that the department agreed with most of the recommendations made in OLA’s report. In particular, Commissioner Seagren said, “MDE, with input from the Advisory Committee, will work to develop a process that can be used to apply consistent expectations to all districts and charter schools participating in the program,” as recommended by OLA. Commissioner Seagren also said, “MDE will work to add clarity to the identification of minimum requirements.” In response to another OLA recommendation, Commissioner Seagren said, “MDE agrees there should be periodic reviews of all Q Comp participants and we will strive to develop a means to do this work in an efficient manner.”
The Program Evaluation Division was directed to conduct this study by the Legislative Audit Commission in March 2008. For a copy of the full report, entitled "Q Comp: Quality Compensation for Teachers," 85 pp., published in February 2009, please call 651/296-4708, e-mail Legislative.Auditor@state.mn.us, write to Office of the Legislative Auditor, Room 140, 658 Cedar St., St. Paul, MN 55155, or go to the web page featuring the report. Staff who worked on this project were Judy Randall (evaluation manager), David Kirchner, and Mary Robison.