Posted:
03/06/2006
Announcement
SEDL Issues RFP to Support Evaluation of Afterschool Programs
SEDL in Austin, TX has issued a Request for Proposal (RFP) to support a randomized control study to determine the effectiveness of afterschool programs on student academic outcomes. This proposed work is supported through SEDL’s US Department of Education (ED) Contract ED-03-CO-0048. Prospective bidders may download the final RFP. Submission deadline is 5:00 pm (CST), Thursday, April 6, 2006. We recognize that time is very short for the development of the RCT proposals; therefore, we are posting answers to questions as quickly as possible to help you in your development process. Additional questions may be directed to afterschoolrct@sedl.org. Question 1 We are interested in using some of the money associated with the grant to fund services in the schools. The idea rests on the proposition that such money would enable us to engage the schools in a randomized trial. Moreover, such an investment would enable us to clean up the treatment environment so that we have a better measure of the hypothesis we are attempting to assess. For example, we are interested in comparing differences in curricular reading/math materials so as to assess what works best with different populations of students. Neither the schools nor the district have the resources to create a "clean" treatment environments. So can we use the money to buy materials, buy teacher time for after school instruction, buy training, and equipment related to and necessary for the project evaluation. We are not talking about using grant money to buy services that we would otherwise purchase. We are asking about the latitude we have to spend the money to ensure the creation of a "clean" treatment environment, so as to minimize construct and measurement error in our scientific randomized trials. Response: The RFP states that potential subcontractors may use up to 30% of the funds for materials, training, and other implementation activities. However, we would give preference to a proposal which minimizes or eliminates these costs through matching funds. In either case, the budget proposal must make clear that the fund will be used in a reasonable and necessary manner to support a rigorous evaluation effort. Question 2 Is there an opportunity to broaden the reach of this RFP to K-12 students? Response: For practical reasons, the Department of Education and SEDL have agreed that programs serving the elementary grades will be more conducive to RCT studies. For instance, elementary level after-school programs tend to have much higher participation rates than programs for older children, as well as readily available assessment data at least in reading and mathematics. Question 3 Are only organizations located within SEDL territory eligible to respond to the RFP? Response: SEDL provides programs and services nationwide. Anyone in the U.S. who meets the eligibility requirements listed in the RFP is eligible to apply. Question 4 Is this a one-time RFP or is there a possibility to apply next year? Response: This is a one-time opportunity; the studies will be conducted over two years. Question 5 We are concerned that many schools will not want to maintain two conditions for two years. Would the grant cover two linked one-year studies, as opposed to one two-year study? Response: If what is proposed amounts to two separate analyses (one in each year), it could possibly be considered, but the question would be whether the treatment would be powerful enough to detect a difference in one year. The preference is to fund designs in which—at least for
some subset of participants—two-year longitudinal data would be available. Question 6 Does the focus of the RCT have to be on a curriculum that spans all levels 2-5, or can it focus on specific levels within that span, such as 4 and 5? Response: It can do either, span grade levels 2-5 or a specific level with that span. Question 7 If indirect costs are included in my proposal should they be included in the $300,000 or are in addition to the $300,000. Response: The total budget should not exceed $200,000-300,000. Any indirect costs should be included in those figures, not in addition to the total. Question 8 Do you have a list of curricula that you recommend to use for reading and math instruction? The districts have several research-based curricula in mind, such as After-school Achievers: Reading club (from Great Source) and the after-school component of Success for All. Response: We do not have a particular curricula in mind. We are looking for mature afterschool programs with evaluation data that suggests success with student achievement that can now be confirmed through a randomized control study. Question 9 What would be an adequate sample size for the study? Right now, we are thinking of using two intervention programs with a total of 400 students. Response: Each proposal should submit an estimate of the adequate sample size based on a statistical power analysis. As stated on page 3 of the RFP: Applicants should demonstrate whether their research design includes sufficient number of settings and/or participants to provide adequate statistical power for detecting meaningful effect sizes for improvements in academic achievement. Preference will be given to proposals that can demonstrate a range of minimal detectable effect sizes supported in the literature and their impact on the study sample size as a justification for their decision regarding sample size. However, given the capability to develop a common protocol of outcome measures and aggregate data across projects, studies will not be judged solely on their ability to have adequate statistical power to detect meaningful effect sizes.
Click here to return to the list of Announcements.
|
