SEDL home ADVANCING RESEARCH, IMPROVING EDUCATION

The National Center for Quality Afterschool

Science After School (SAS) Consumers Guide  

Evaluation Criteria

Materials in the Consumers Guide are evaluated at three levels.

  1. First, as items are nominated, staff at the Lawrence Hall of Science review and rate them according to the following criteria:
    1. Science quality and accuracy of this material
    2. Social & entertainment value of this material
    3. Outstanding features
    4. Ease of use
    5. Value to user (dollar cost is not the sole criterion here, --e.g., a material may cost $100, but be the basis for multiple highly productive activities or adaptable for a variety of useful purposes within the afterschool science context)
    6. Suitable for repeated usage (e.g., the materials are designed for use numerous times, possibly lasting for more than one year)

  2. Second, LHS contacts the developer and obtains a copy of the materials. LHS then conducts a preliminary professional and common sense review to make sure that the materials are "in the ballpark" from both a science and an afterschool perspective.

    For science content, the main criterion is content in the National Science Education Standards (National Academy of Science, 1996). Most after school materials make use of relatively common science content (e.g., inquiry skills, properties of matter, life cycles of organisms, phases of the moon,) because they are prepared with the assumption that the instructors may not have broad or deep science backgrounds. Most state standards follow the National Science Education Standards, which are presented in grade bands (K-4, 5-8, 9-12); some add more specificity, such as California's standards which provide grade-by-grade guidelines. Given that the Consumers Guide is a national resource and most after school programs work with students from multiple grade levels, LHS believes that a grade band match is sufficient and realistic. LHS also checks the materials to make sure that no pseudo-science (e.g., astrology instead of astronomy, alchemy instead of chemistry, etc.) is given consideration.

    For afterschool, the main criterion is reasonableness. If a person likes what he or she reads in the reviews, would he or she be able to obtain the program for his or her use? By this criterion, "Uncle Bob's Science Show in Small Town, New Jersey," which is a pseudonym for a program that was actually nominated by "Uncle Bob" and three or four of his satisfied customers (or relatives?) was ruled out. LHS understands that it is not uncommon for afterschool programs to hire people to come in and give an entertaining live show to satisfy their goals of "doing something in science," but LHS did not feel that this was the kind of "something" that fit the criteria for academic enrichment in the afterschool arena, nor could we give endorsements to individuals.

  3. Third, after LHS staff screening, LHS sends the materials to a science and an afterschool reviewer with the instructions below. LHS field-tested the instructions with members of our Expert Panel to make sure that the instructions were sufficiently directive that they would cover all of the points that LHS wanted in the reviews, but open enough that the reviewers could bring up additional points that they thought were worth noting. The reviews that were received based on the field test of the review instructions were detailed enough that LHS was satisfied of coverage, while varied enough that LHS was satisfied that they would be interesting and tailored to the particular program.

Below is the guidance given by LHS to reviewers.

Instructions for Science Reviewers

Please review this material/curriculum for recognition in the Consumers Guide for Science Afterschool Resources.

Rather than provide a checklist or structured instructions for your review of these nominations, we ask you to examine the materials from your perspective as an experienced, knowledgeable, consumer or manager of users of science educational materials. This guide is loosely based on the highly accessible Zagats Restaurant guides that offer local reviews submitted by actual consumers who provide candid observations and practical answers to questions about the specific categories that comprise the guide.

Please examine the nominated material we have sent to you and comment on its quality as a hands-on, inquiry-based tool for engaging students in science; its scientific accuracy; its adaptability and appropriateness for the grade levels/ages it targets; and the likelihood that it could be implemented as a component of an afterschool science program staffed by instructors with a wide range of experience and/or expertise in science and limited preparation time (e.g., many afterschool leaders will not have time to participate in extensive professional development or have much time to prepare each day prior to their sessions with students). Finally, based on your review, is this consistent with the notion of a research-based material?

We ask that you prepare a 1 or up to 2 page narrative addressing the elements noted in the set of bullets below.

Instructions for Afterschool Reviewers

Please review this material/curriculum for recognition in the Consumers Guide for Science Afterschool Resources.

Rather than provide a checklist or structured instructions for your review of these nominations, we ask you to examine the materials from your perspective as an experienced, knowledgeable, consumer or manager of users of science educational materials. This guide is loosely based on the highly accessible Zagats Restaurant guides that offer local reviews submitted by actual consumers who provide candid observations and practical answers to questions about the specific categories that comprise the guide.

Please examine the nominated material we've sent you and comment on the quality, adaptability, and likelihood for implementation as a component of an afterschool science program. We seek your insights to rate the utility that these nominated materials would offer afterschool instructors, given the high degree of variability in their familiarity with science, capacity to implement materials like these, limited amount of preparation time, and their focus on supporting the psychosocial needs of program participants. Your perspective as an expert on the range of issues that impact the success afterschool programs is critical, since our purpose is to identify programs and materials that not only offer high quality science-oriented content, but are particularly welN-Suited for the afterschool the environment.

We ask that you prepare a 1 or up to 2 page narrative addressing the elements noted in the set of bullets below and the issues noted above.