Annual Report Overviews

Service of Standards-Based Science Teaching Annual Report Overview 2000-01

Goals and Objectives of the CRSEP Initiative
The overarching goal of the Capital Region Science Education Partnership LSC Initiative is to provide professional development for science teachers that expands and deepens their content and pedagogical knowledge. The primary focus of this professional development is assessment in the service of standards based teaching. Each of the initiatives projects, Professional Development, Materials Design, High Stakes Testing Study and the Student Achievement Study, contributes to the overarching goal of Local Systemic Change and contributes to the science education fields understanding of the complexity of providing high quality professional development in sufficient quantity to make a difference in student achievement.

Activities
Professional Development Project Activities-Education.
Professional Development Project Activities-Education. Through 31 October 2001, the CRSEP project personnel have designed, implemented, evaluated, and modified approximately 202 professional development activities.

Workshops topics have included:

1. Introduction to STC units for teachers new to CRSEP districts and new to grades within the units;
2. Tested implementation strategies for STC units for teachers experienced with STC units;
3. Strategies for the design of end-of-year and end-of-STC Unit assessments.
4. Analysis of and strategies for administration and grading of the New York State Mandated ESPET and Middle-Level Science examinations.
5. Alignment of STC Units with New York State Mathematics, Science and Technology Standards and New York State Core Elementary and Middle Level Curricula.
6. The Language of Science- Reading, speaking, and writing in science.

The design, implementation and evaluation of these activities has been facilitated by the development of a planning form for professional development, extensive use of power point presentations, and reflection by the Professional Development and Materials Development Coordinators with Professional Development Specialists about the CRSEP Initiative philosophy and strategies for professional development and how these are different from those underlying the professional development experienced by the Professional Development Specialists.

Activities-Research
Because attracting teachers to CRSEP professional development activities has been a challenge, a questionnaire was designed to get information about CRSEP District Teachers; reasons for choosing not to participate in CRSEP professional development activities and conditions under which teachers would chose to participate in CRSEP professional development activities. The results of this research activity have guided the project's design and implementation of professional development. The results also signal the challenges science professional development providers nationwide face in an education system in which mathematics and English-language arts receive the greatest emphasis.

Materials Development Activities-Education
In collaboration with the Professional Development Project personnel, Materials Development Project Personnel have participated in the design of professional development materials aimed at providing workshops addressing narrow topics relevant to science assessment while struggling to fit out developing knowledge into a coherent structure for professional development. While much of project personnel time has been spent on the development of concept maps, understanding of the principles of pedagogy as applied to teachers has been a focus of the project development efforts. Personnel on the Professional Development and Materials Development Projects have spent considerable time learning from the research literature and from each others experience about how to work effectively as teachers. Simply stated, professional development materials and strategies must convey the philosophy that teachers are active partners in the design of their professional development. Conveying this notion is difficult when teachers experience is that professional development is something that is done to them.

Under the guidance of the University at Albany (UA) faculty and research assistants, the PDSs have been developing concept maps representing the science content contained in the Science and Technology for Children (STC) units. In the process of constructing concept maps, the PDSs have come to recognize gaps in their own knowledge of the science content contained in the units and have seen how certain key ideas in the units are conflated, leading to the possibility that students might develop misconceptions without skilled teachers who can help them make essential distinctions. Despite the fact that the development of concept maps has been difficult work and has challenged their science understanding, the PDSs have found that this activity has improved their content knowledge. Correlating the principles contained in their maps with the New York State Science standards has helped the PDSs identify science required by the Standards that is not contained in STC units. They will use this information to help teachers add the principles to STC Units at appropriate places.

The concept maps have been presented to Partnership district teachers, who have found them useful as representations of science content that they can use as references to the science content in the STC units. These teachers are excited at the prospect of receiving more concept maps from the Project. The teachers who have been introduced to the process of making concept maps to date see great potential in map development as a learning tool for their students.

Of special interest to teachers is the use of concept mapping as a tool for student writing. Constructed response items on the NYS 8th Grade Science Assessment have proved challenging to students, and teachers are most receptive to any and all strategies that will improve student writing.

High Stakes Testing Study
During the first year of the project we have been involved in research activities that will lay the groundwork for the upcoming fieldwork.

First, we have conducted a thorough review of the existing literature on high stakes testing. This will allow us to put the experiences in upstate New York school districts in the larger policy and practice context of the country. What we have found is that there is a great deal of literature which presents serious challenges to high stakes testing and much less evidence that HST has had a positive effect on schools and educational reform. Still though we find there remains a need for more empirical evidence of both broad and specific effects of high stakes testing. Also, this project will look at the broad array of testing demands especially in elementary schools that may well effect the attention teachers are able to pay to science education reform. We have found very little in the literature that addresses this systemic perspective but with the possibility of testing all children in grades 3 through 8 it will become even more salient.

Second, we have been searching documents that will allow us to tell the story of the testing policy context in New York State. These documents have been collected from the NYSED website, the NYS Library, the New York City Board of Education, the New York State United Teachers (NYSUT) library, and newspaper accounts statewide. We are also undertaking a review of historical documents related to the creation of the New York State Regents Exams as they are the foundation for at least some of the current testing policy. We are still in the process of collecting and categorizing documents and so do not have any findings at this point. Additionally, we have been meeting with school district and building staff to plan for the fieldwork which will begin in the fall 2001. This has involved meetings with superintendents, CRSEP staff, and principals.

Additional High Stakes Testing Study Activities of High Stakes Testing Study include:

1. Preparation of an annotated bibliography of research on high stakes testing
2. Collecting and analyzing documents regarding New York State policies on high stakes testing
3. Interviewing New York State Education staff and policy makers, including members of the Board of Regents on New York state policies and practices in student assessment
4. Negotiating with districts and schools for access to begin the field work component of the study
5. Preparation of summary school report cards to track student assessment information

Activities include:

• Working with the schools, we designed the data collection of existing test scores in a way that met the schools interpretation of the requirements of the Belmont Report.
• Working on a double coding of ID numbers as a means to insure that the names of the students do not leave the district offices and that the ID numbers used in the research cannot be matched to students if the numbers inadvertently get back to the school.
• Set up the categories of variables and the format for the database that will be used in the tracking and analysis of data.

1. Differences in rules of intellectual engagement: Ideas are often perceived as mandates by teachers, and as matters for discussion by school administrators and individuals from higher education.
2. Differences in administration structures across higher education and K - 12 education: Other than budgetary considerations, higher ed participants are free to follow diverse paths regarding strategies for pedagogical and content when proposing directions for K-8 science. The primary administrative constraint is resources. K-8 participants (teachers and administrators) are constrained by state curricular and testing mandates, community values, and internal administrate structures that dictate curriculum (including educational materials selection, time allotments to different subject areas, and importance placed on student performance on state tests, admission to college, union contracts, and the degree to which individual parents support the school in the education of their children
3. Differences in individuals perceptions of autonomy: Teachers perceive that they have little autonomy except in their classrooms and protect that autonomy.
4. Differences in the allocation of time to doing planning, and reflection: Teachers have little time for planning or reflection on their professional practices. Consequently, they do not take the time to plan or reflect even when professional development time is provided for planning and reflection. In contrast, individuals from higher education, plan and reflect often to excess in terms of the consequences of the matter under consideration.
5. Collaboration across autonomous school districts is challenging: Each school district has a unique science curriculum. Despite the fact that all the districts in the CRSEP collaborative are using STC Units, each district uses different units, they are using units at different grade levels, and units are used at different times of the year. These individual choices and schedules make planning a coherent program of staff development an impossible task.

1. Getting teachers to engage in CRSEP Initiative professional development is a challenge.
2. Teachers are highly selective about participation in professional development activities.
3. Teachers prefer professional development activities designed and implemented by their colleagues.
4. Teachers are proud and confident of their ability to teach science and see projects such as the CRSEP project a threat to their pride and confidence. The perspective is that NSF support was sought because of the science program and by implication of teachers was weak, rather than the perspective that NSF resources provide the opportunity to make good programs better.
5. Teachers participation in professional development is defined in union contracts.

Materials Development
The CRSEP approach is based on the principle that development and implementation of formative and summative assessment strategies requires an understanding of science content.

• With few notable exceptions, K-8 teachers do not have sufficient understanding of the science content contained in STC units to effectively implement the units or to design instruments and strategies for the formative and summative assessment of the units.
• With few notable exceptions, teachers have only limited use of the language of science, its vocabulary and syntax. This limitation makes the design of items and scoring standards for formative and summative assessments challenging.
• Teachers know little about the form and function of assessment or principles of assessment design.
• Teachers in New York State, take direction for curriculum emphasis from state mandated tests rather than from the New York State Mathematics, Science and Technology Standards or the New York State Core Curriculum documents.

At the foundation of all of the design and implementation of professional development materials is the principle that teachers must be responsible for their own professional development. Consequently, the themes of reflection and responsibility pervades all the CRSEP workshop design and implementation strategies.

High Stakes Testing Study
We are too early in the research process to have formal findings from our research activities. The review of research has demonstrated a preponderance of research that presents real challenges to hst as a viable practice but also some evidence that supports HST as a positive strategy for educational reform. This document, which is in draft form now, will be a valuable resource for educational researchers, policy makers, and the public.

We are just beginning the analysis of the data we have collected to portray the testing policy context and practices in New York. New York State has a long history of testing with the Regents Examinations and that has been an important element in the creation of current policies. In New York, as elsewhere in the country, policy makers see student testing as the key to providing advantage to those who have not received adequate educational services, the most powerful means to hold schools and their personnel accountable, and down play any technical issues or concerns with the testing, scoring, and reporting.

Student Achievement Study
Our findings to date are that the lead-time to get IRB approval can be half a year or longer when several Institutional Review Boards and several interpretations of rules and regulations are involved. This can be the case even for research using data that are routinely collected by school districts and there is not new data being generated.

Training and Development:
Professional Development Activities-Education

Without exception, CRSEP Initiative personnel are developing new understanding and abilities. Some personnel are coming to understand the programmatic and administrative expectations of the National Science Foundation. Others are developing enhanced understanding of ways of institutions of higher education and K-12 school districts and how differences enhance the project without compromising the values of either institutional type.

Professional Development Specialists are learning to function in a new role, that of a professional development provider, and which of their previous experiences in the classroom and with professional development apply in their new role.

We have learned that professional development materials and strategies must demonstrate to teachers that they are active partners in the design of their professional development. Conveying this notion is difficult when teachers experience is that professional development is something that is done to them.

Doctoral students are learning about standards-based reform not only from books but also by doing it. The project provides them the opportunity to apply the theories they have studied in advanced courses and evaluate those theories in light of their experience in schools. A faculty member, not associated with the CRSEP Initiative, reported that her seminar on reflection as a strategy for teacher development was enhanced by reports from the from the field by CRSEP project personnel about the engagement of Professional Development Specialists in the process of reflection.

The undergraduate and graduate students working on the High Stakes Testing Study have gained substantial research skills, have developed a more sophisticated understanding of the complexity of educational reform, and especially the use of high stakes testing as a tool of reform. In addition to students who are employed to work on this research project, one doctoral student is currently completing a research internship working with the project and there are several others planning to do so in the future.

Outreach Activities:
The CRSEP project has begun to provide and participate in outreach activities as follows:

1. Attendance of Professional Development Specialists and Professional Development Associates at the K - 20 Re-thinking Science Education in New York State Conference
2. Presenters at the Eastern Section Science Teachers Association of New York State, Inc. on Understanding the Big Ideas of Science with Concept Maps, Curriculum Alignment, Why Is It Necessary?
3. Presenters at Spencer-Van Etten Central School District conference day on Language and Science to 45 elementary teachers and 30 middle school/high school teachers
4. Attendance at the STEM (Science, Technology, Engineering and Mathematics, Inc.) reception for science educators in the capital district
5. Presentation at Albany Consortium for Research in Instructional Design and Theory (ACRIDAT) on the use of concept maps to SUNY doctoral students and faculty

The Relational Concept Maps and Diagrams contain relationships among all the scientific and technical concepts (terms) contained within the unit for which they were designed. In addition to the terms contained in the STC Units, scientific and technical concepts that are a part of the New York State Science Standards or that help a teacher develop a better overall understanding of the content are included in Relational Concept Maps and Diagrams. The maps and diagrams also provide detailed parallel examples. For example, in the Animal Studies Relational Concept Map and Diagram, the Dwarf African Frog is compared to the Fiddler Crab, promoting an understanding of the form and function of living things in meeting basic needs. The maps and diagrams are designed to summarize the big ideas of the units to elementary teachers in a non-threatening way. Knowing the big ideas (major principles) that the lessons and units are designed to teach is necessary for the design of formative and summative assessment.

The principles are stated in ways that are appropriate for the grade level for which the units are designed. Ultimately, the purpose of the maps and diagrams is to promote all students' learning of science, enabling them to meet New York State and National Standards.

High Stakes Testing Study:
While there has not been an opportunity for the school districts to use the school report card profiles, the districts have expressed a great deal of interest in the data in this form. We anticipate school districts will have an opportunity to understand the meaning of the test scores in new ways as this data base builds over the life of the project. For example, we will show the almost immediate increase in the number of students passing the test after the second administration, the diminishing improvement after that, as well as the differences in different types of schools.

The staff working on the supplemental research grant have contributed to the planning for the research components of the main project, especially with regard to the Student Performance Study. The Director of this project has also participated in numerous discussions about teacher development project issues at a general level and has leant substantial experience in evaluating such programs to the CRSEP staff.

The work to date on this project has been very helpful to the American Evaluation Association, its Public Affairs Committee and the Task Force on High Stakes Testing. The AEA Task Force is preparing a public statement and documentation on the effects of high stakes testing and the knowledge and experience of our research project has been a major influence on this activity. It is likely that AEA will join with other professional associations in an effort to raise awareness, encourage critical analyses, and promote best practices in student assessment and educational evaluation. There is every reason to believe this project will continue to provide evidence and materials in these efforts.