Innovating a Promising Practice in High Poverty Rural School Districts

This article gives meaning to innovating promising practices in high poverty rural contexts, as experienced by the Rural Math Excel Partnership (RMEP). The project sought to develop a model of shared school-family-community responsibility to support student success in foundational math courses as preparation for science, technology, engineering, mathematics and health (STEM-H) careers. RMEP was one of the two rural development grant awards in the 2012 federal fiscal year, the first year for awards in the rural priority area of the U. S. Department of Education’s Investing in Innovation (i3) program. High poverty rural areas may have major implications for fidelity of implementation and measurement of intended impacts that raise important questions about project organizational structures, capacities and evaluation needs. If significant external funding and a partnership approach are key catalysts for innovating solutions to educational challenges, the answers to 10 questions of readiness could have major implications for project success.

(see, e.g., Fullan, 2001Fullan, , 2010Hargreaves & Shirley, 2009;Robinson, 2015;Wagner & Dintersmith, 2015;Zhao, 2012). In recent years, innovation labs, hubs, centers and projects have evolved rapidly at all levels of the education system. At the federal level, the U.S. Department of Education (USED) established an Office of Innovation and Improvement (OII) in 2002, becoming the Office of Innovation and Early Learning under the Office of Elementary and Secondary Education in USED's 2019 reorganization.
In 2010 OII began operating the Investing in Innovation (i3) program and made awards in the 2010-2016 fiscal years before being replaced by the Education and Innovation Research (EIR) program authorized in the Every Student Succeeds Act of 2015. The i3 competitive grant program sought to expand the implementation of, and investment in, innovative practices. The program enabled eligible entities to (1) expand and develop innovative practices that could serve as models of best practices, (2) work in partnership with the private sector and the philanthropic community, and (3) identify and document best practices that could be shared and taken to scale based on demonstrated success (Investing in Innovation, n.d.).
The i3 program awarded three types of grants: development, validation, and scale-up. Development grants supported developing and testing interventions with limited or no prior evidence of effectiveness. Interventions with moderate evidence of effectiveness received validation grants to support implementing and testing the intervention in a broader population or in new contexts. Scale-up grants supported implementing and testing interventions with strong prior evidence of effectiveness on a much larger scale. All projects were required to have an independent evaluation conducted as part of the project. Over the 2010 to 2016 period, the i3 program awarded 172 grants: 115 development, 46 validation, and 11 scale-up grants. The i3 program invested almost $170 mil. in 23 rural priority area projects (Investing in Education i3 Grant Awards, n.d.). RMEP was one of the two rural development grant awards in the 2012 federal fiscal year, the first year for awards in the rural priority area.

Innovations for Rural Contexts
Education reforms and subsequent innovations have been designed most often for urban school settings (Zuckerman et al., 2018). Few large-scale efforts have focused on schools in rural contexts. Noteworthy exceptions include the 30 National Science Foundation's rural systemic initiatives (Harmon & Smith, 2012) and The Rural Challenge (Annenberg Rural Challenge, 2000;News from Brown, 1995;The Rural Challenge, 1999). Stern (1994) noted in a USED publication entitled, The Condition of Education in Rural Schools, that in 1975 the National Institute of Education (NIE) documented the education innovation process in 10 rural communities participating in its Experimental Schools program. Stern (1975) also pointed out: Many co-called 'innovations' being championed today were born of necessity long ago in the rural schoolhouse. Cooperative learning, multigrade classrooms, intimate links between school and community, interdisciplinary studies, peer tutoring, block scheduling, the community as the focus of study, older students teaching younger ones, site-based management, and close relationships between students and teachers-all characterize rural and small school practices. (p. 1) In the late 1980s, Congress charged USED's Office of Educational Research and Improvement (OERI) to carry out a rural education program in each of the regional education laboratories (RELs). Though today the RELs are not required to operate a rural education program, they are required to devote a portion of their budgets to serving rural schools. Several of the RELs have formal research-practice partnerships or alliances that focus specifically on needs of rural schools. Rural-specific publications of the REL Program are available at https://ies.ed.gov/ncee/edlabs/projects/rural.asp.

RMEP's Rural Context
The Rural Math Excel Partnership (RMEP) began on January 1, 2013. RMEP was an investment of approximately $3.1 mil., funded by an i3 grant award ($2.7 mil.) to Virginia Advanced Study Strategies, Inc. (VASS) and $420,000 in private matching funds that were committed to the project's independent evaluation. Previously, in implementing the Virginia-based model of the National Math and Science Initiative (NMSI) APTIP program over four years in 73 schools, VASS staff concluded that many middle and high school students in the high-poverty rural areas lacked the supports necessary for success in rigorous math and science Advanced Placement (AP) courses. In offering pre-AP and AP teacher training and student supports (e.g., Saturday help sessions), VASS staff observed that many students needed broad-based supports for success in foundational math courses as preparation for advanced courses and STEM-H careers. These supports, however, needed to embrace shared responsibility among teachers, families and the rural community.
Based on the NMSI project experiences and research evidence in the literature, a partnership project comprised of VASS, Inc. and six rural school districts was created to develop a promising practices model of shared responsibility that could be supported by the i3 funding opportunity. However, the model design also needed to embrace a workforce development purpose for STEM-H occupations in the rural region. Like many rural areas, the predominantly blue-collar rural communities needed schools to graduate students capable of pursuing technical-level and higher career choices (Alliance for Excellent Education, 2010;Beaulieu & Gibbs, 2005;Gibbs, Kusmin, & Cromartie, 2005;Thompson, 2007). Technical occupations were among the fastest growing job fields in America (Carnevale, Smith, & Strohl, 2010). Also, STEM-H technician occupations offered higher pay than traditional jobs in the region, but they also required more education and attainment of some form of postsecondary education credential.
Economic and workforce development were high priorities in the five counties of the six school  Table 1 shows key characteristics of the five counties.
All counties lost population from 2010 to 2012. Poverty levels in each county were double the state percentage. Unemployment rates exceeded the state rate. State median household income far exceeded levels in the counties. The five-year average (2007)(2008)(2009)(2010)(2011) high school education attainment in each county was much lower than the state average. Percentage of the population with some college was similar to the state average. Bachelor's degree or higher attainment, however, was much lower in each county compared to the state average.
With increasing poverty rates, decreasing populations and lower education levels, each of the six school districts was eligible for the federal Rural Education Achievement Program (REAP), the USED i3 program's definition of a "rural" local education agency (LEA). Five of the six districts were public countywide school systems, with one district within one rural county classified as an independent city public school system. RMEP included seven high schools and seven middle schools. One county school system had two high schools and two middle schools. Table 2 shows key characteristics of the 14 schools when the RMEP project began.

Designing the Promising Practice
RMEP aimed to develop a sense of shared responsibility among teachers, students' families and the rural community as collective support for student success in foundational math courses required for advanced high school and postsecondary study. Foundational math courses included Algebra I, Algebra II, Algebra Functions and Data Analysis, and Geometry. As an applied research project RMEP pursued a solution to an immediate real-world problem rather than basic research "conducted for the purpose of extending the boundaries of our collective body of understanding…" (Jaccard & Jacoby, 2020, p. 31). The guiding conceptual framework was that (a) math teachers, families of rural students, and community organizations can each perform specific support functions for students; (b) these supports collectively could enable students in Algebra I, Algebra II, Algebra Functions and Data Analysis, and Geometry courses to acquire foundational math knowledge and skills; (c) such foundational math knowledge and skills are necessary for success in advanced high school courses; and (d) advanced high school courses serve as preparation for postsecondary education and training for STEM-H careers.
This conceptualization was grounded in the grand theory of functionalizism (Jaccard & Jacoby, 2020); for example, how a school functions in society to serve the individual and community. Achievement motivation, attribution and expectancy-value theories (Low & Jin, 2012;Singh, 2011) informed RMEP's design, particularly self-efficacy theory whereby "learning develops from multiple sources, including perceptions of one's past performance, vicarious experiences, performance feedback, affective/physiological states, and social influences" (National Academies of Sciences, Engineering, and Medicine, 2018, p. 112).
RMEP's conceptual framework also drew heavily on relevant research evidence. In a study of 10 rural high schools Hardre, Sullivan and Crowson (2009) found school interest, effort, and intentions to graduate and go on to postsecondary opportunities increased if students could see usefulness and value of what was learned in school as contributing to achieving their goals. Moreover, the researchers found students who feel competent and believe they could learn and develop skills in a school subject (e.g., mathematics) were more likely to demonstrate course-related interest, intend to graduate, and pursue postsecondary education.
Also important to the conceptual framework, McFarland (1999) found in a study of 108 small rural school districts in 12 states that parents are the single most important influence in the career planning and decision-making of their children. In a national study of more than 8,000 rural youth, Griffin, Hutchins, and Meece (2011) found the most widely-used career counseling information sources for students in rural schools were parents, friends, teachers, and school counselors. Lastly, in a study of five rural high schools Singh and Dika (2003) found adult social network processes explained between 13% and 15% of the variance in educational and psychosocial outcomes, particularly for academic effort, academic orientation, and trust. Academic support was statistically significant for educational aspirations, academic effort, academic orientation, and selfconcept. Emotional support was statistically significant for academic effort, academic orientation, and trust.
In the RMEP shared responsibility model each role group had specific functions. Teachers of students in the four foundational math courses were to (1) integrate web-based Khan Academy videos into lessons as student homework assignments, and (2) conduct a family math night/forum at the beginning of each semester. Parents of students were to (1) reinforce child completion of the videos and assessments in the home environment, (2) participate with their child in family math nights, and (3) reinforce child/family participation in the community STEM-H careers event. A community team was to organize and (1) plan a STEM-H careers event that reinforced the need to learn math for success in technician-level and higher careers important to the regional rural economy, and (2) conduct a STEM-H careers event at the county fair or some other county/community venue that parents considered highly accessible and or welcoming, particularly if certain segments of the population had negative views of attending in a public school setting.
The logic model (program theory) in Figure 1 illustrates how project inputs, activities and outputs aligned with intended outcomes. Evaluators refined the logic model to show a theory of change with mediating variables, which is found in the final RMEP evaluation report (Nagle et al., 2016, p. 3). A project data base was established to aid project reporting, implementation decisions and evaluators' assessment of model potential as a "promising practice."

Inputs
Activities Outputs Outcomes Impact

Evolving the Promising Practice
Funding and flexibility of the four-year i3 development grant (and extended fifth year) enabled project staff and the project's Advisory Leadership Team (ALT) to address numerous challenges. Evaluators focused on assessing implementation fidelity and intended outcomes consistent with USED-required evaluation assistance that sought to align evaluation design with standards of the USED What Works Clearinghouse. Developing the shared responsibility model was an evolutionary journey in each of the six rural school districts.

Project Year 1 (Jan.-Dec. 2013)
RMEP leadership met individually with district superintendents to review expectations of the grant award. Project orientation meetings were held with school principals in three separate locations of the rural region. A contract was finalized with the external evaluation organization. The cooperative agreement with the USED i3 office, the year-one management plan, and the revised evaluation plan were all completed on time as required. RMEP held a meeting of the ALT. RMEP staff meetings routinely occurred as two face-to-face or conference call meetings per month. An evaluator participated at least once per month in staff meetings. External  evaluators also attended the ALT meeting, the orientation meetings with principals, and a special RMEP staff meeting to explain the purpose of the external evaluation and evaluation plan.
Superintendents and principals nominated a teacher to serve as the district's member on the RMEP's teacher development team. Lead by the RMEP math specialist, the team of six teachers (one per district), with each teacher supported by a project stipend, was responsible to develop the Math Advanced Studies (MAS) guide and plan the training of teachers in their respective districts. The first unanticipated challenge occurred when the project math specialist could not be hired until June, after completion of the public school teaching contract. This delayed project implementation. District superintendents wanted all teachers of the four courses included in the project. This would ensure all teachers received the high-quality professional development offered in the project and all students were equitably served. Project leadership agreed; however, this decision caused major implementation challenges and changes in year three.
To connect content of math courses with workforce needs, RMEP staff held two modified DACUM sessions with persons working in 35 STEM-H technician occupations. RMEP used state databases to identify occupations defined as "bright outlook" or important to the future workforce development of the region (see Table 3).
DACUM results revealed an unanticipated finding: content in the targeted math courses may not align with math competencies used by the STEM-H technicians in the workplace. The majority of technicians used middle school math primarily and some Algebra. Project staff published the findings to reveal the four types of math gaps discovered in the project (Harmon & Wilborn, 2016). Teachers on the development team acknowledged attendance at the DACUM sessions was one of the most rewarding professional learning experiences of their careers. It was their first opportunity to learn "how" math competencies were used from those who performed them on the job. The team concluded that telling students to take all the math they can in high school to be "college ready" was not the appropriate answer for working in the rural region as a STEM-H technician.
This impacted development of the Math Advanced Studies (MAS) guide. The team had to design lesson plans and select Khan Academy videos as student homework for 39 competencies. Each lesson also included at least two examples of how the math competency was used by STEM-H technicians. Consequently, staff soon faced a new challenge, as many teachers began to perceive that RMEP was asking teachers to implement some new kind of curriculum, one not aligned with the state math standards for their courses.
Teachers also were concerned some students had limited home access to the Internet for accessing the Khan Academy videos in the MAS guide. RMEP planned to provide students with access to the Internet either by paying for Internet connection and or providing a tablet for students who did not have a computer at home. By the end of year one, many technology-related issues were discovered, often unique to a district's student security policies and procedures or related to a student's home Internet signal.
Schools surveyed students to determine home Internet access. This step proved very time consuming and complicated. And one school district board of education expressed equity concerns in identifying "those in need," preferring instead that all students receive a tablet. Budget limits prohibited this option. But this technology challenge also revealed that RMEP needed to take a site-specific implementation strategy for each component of the shared responsibility model. No longer was it feasible to expect the shared responsibility model would have consistent implementation fidelity in each district. The model would evolve as a promising practice consistent with unique challenges and adaptations necessary in each school and district.
Community teams also were in various stages of development. Place-specific issues presented challenges for identifying the lead community organization, selecting team members, and planning the community STEM-H careers event. Slow progress or lack of teachers' success in accomplishing their role functions (e.g., family math night) delayed team planning of the STEM-H careers event. Parents' performance of their role functions also depended greatly on teachers' ability to make student homework assignments and conduct the first family math night.

Project Year 2 (Jan.-Dec. 2014)
Project staff worked diligently to solve Internet access issues in student homes. VASS, Inc. negotiated a contract with Verizon Wireless to purchase the 10-inch Samsung Galaxy Tab 2 tablet with a shared monthly data plan. RMEP purchased a cloud-based mobile device management (MDM) system that included Mobicontrol software on all devices and a cloud-based dashboard set up that could be managed by VASS staff. After the first batch of 130 tablets were received in April 2014, project staff worked with Verizon to apply screen protectors, fit tablets into protective cases, activate and set up tablets, and install Mobicontrol MDM software-one tablet at a time.
Staff conducted two-day teacher training sessions in three sites of the region. An additional one-day training was held for teachers unable to attend a regional session. 79 teachers participated, more than the original plan of 70 teachers. Evaluators reported overall, quality of the teacher training sessions was high. RMEP was successful in creating teacher buy-in, demonstrating math importance in technician occupations, and in describing activities teachers would need to perform. Evaluators found most teachers understood their role and functions. Evaluators also found in survey results that a majority of teachers "agreed" they would use the MAS guide and assign the Khan Academy videos. Overall, the teachers were very receptive to the technology presented in the training sessions and understood the need to plan and conduct a family math night.
RMEP project implementation with students began in the fall semester. By mid-September, all teachers were trained and had tablets-but that did not mean that a majority of the teachers could or would perform their functions. Only 30, or 38%, of the 79 teachers assigned at least one Khan Academy video as student homework. The majority (61%) of teachers registered in the technology platform had assigned no videos.
One reason was surveying students regarding home Internet access proved time consuming, especially in five of the seven high schools operating on a 4x4 block schedule. Half or more of the semester passed before students without Internet access was determined. Moreover, some students in schools with block schedules did not enroll in any of the four math courses in the spring semester. Staff had to retrieve their tablets, another unanticipated, disruptive and time-consuming challenge. Also, in one district "hotspot" accounts with a different Internet provider had to be purchased to give students home Internet access.
Another reason for delay in teacher use of tablets occurred when district policy required delivery of tablets to district information technology (IT) staff for teacher distribution. Other delays occurred as RMEP staff also had to work directly onsite with teachers and district level personnel to set up user accounts in the project's technology platform and Khan Academy. Setting up student accounts required information from teachers about courses taught and associated periods/blocks prior to staff arrival for the fall semester. Delays occurred in scheduling school computer labs to set up user accounts. Setting up a user account required an email account. Staff found gaining parent consent for students under the age of 13 (required by law) was very tedious. Eventually, student accounts were established successfully in 13 of the 14 schools. Some students found ways around the tablet's security lockdown that filtered web content and limited data usage. A student in one high school learned to hard reset the device to get around the filtering and informed many other students on how to do it. This caused liability concerns associated with CIPA regulations for both the school district and VASS, Inc. It also presented potentially intolerable data overage costs for the project budget. Consequently, VASS, Inc. established a virtual provider network (VPN) to gain control over content filtering and data usage. This also enabled school staff to receive instant alerts of security breaches.
During year two, the project's technology platform was constantly under revision. Teachers frequently requested the technology platform provider to make tracking the homework assignments more user-friendly. The private sector technology provider was exceptionally accommodating. But sometimes RMEP staff were unaware of provider changes. This made assisting teachers and students cumbersome and time consuming.
Data reporting problems also occurred in the technology platform system. Teachers witnessed their students watch Khan Academy videos in the classroom that were not recorded in the technology system. After months of seeking a solution staff learned the Khan Academy videos, as YouTube videos, were blocked by Internet filtering systems in some schools. Filtering systems were blocking Khan Academy from transferring the data to the RMEP technology platform-because the videos had "YouTube" in the URL. Staff of Carney Labs, the private company developing the RMEP technology platform, graciously worked with RMEP staff to find solutions to technology issues. In support of the project, Carney Labs was developing the technology platform free of charge.
In year 2, some teachers came to view student access to Khan Academy videos as a viable option for students to do "skill refreshers" outside of class. This presented another challenge, as these teachers assigned videos more aligned with pressures for student success on state tests and less with lessons contained in the MAS guide. Consequently, the RMEP math specialist had to devote extensive time to visit schools and gain teacher buy-in for selecting lessons in the MAS guide and making the appropriate video homework assignments.
Staff assistance enabled teachers at 13 of the 14 schools to hold a Family Math Night (FMN) in the fall semester. Commonly, the RMEP technology and math specialists attended the FMN to explain the purpose of the project and demonstrate the project's technology platform and Khan Academy technology. STEM-H technicians often participated to discuss how they used math in the workplace. Parents learned about resources on the RMEP web site. Staff helped teachers engage parents and family members in STEM-related activities. Food and door prizes were also provided. Parent attendance varied tremendously among the family math nights held by the schools, usually lower than the teacher(s) anticipated.
RMEP staff also helped organize each community team, with five of the six countywide teams led by the youth development specialist of the Cooperative Extension Service (CES). By the end of the semester, one team had held a STEM-H careers event. One team that held an event the previous year and experienced low parent attendance decided to plan a joint STEM-H careers event with a neighboring county team. Team leaders rationalized holding the joint event in the rural area's major shopping town and where most employers were located in the two counties would likely increase parent and student attendance. A joint event also would enable the two teams to share the tasks of planning and holding the event. This was a particular concern of one team leader who strongly supported the RMEP project, but simply had little time available to commit-and little assistance capacity in the county's small Extension Office. Parent and student attendance at the event was far less than anticipated.
Making math relevant to students, parents and community members was a key goal of the STEM-H careers event. This meant creating meaningful conversations among key stakeholders was necessary in the planning of an event. Consequently, REMP staff helped one county team design a "Math at Work in Our Community" activity that teachers of the four foundational math courses could give students as a homework assignment. The activity intentionally caused students to discuss with their parent(s) or adult family member(s) the names of people in the community that used math in their jobs. The student then was to interview one such person. The team provided each student with a 4 x 6-inch card to complete during the interview. How math was used and if the person would present this information to students in the class were included as questions on the card. The county team compiled the information and used results in planning the community event. Classes of students, parents, teachers and business persons were recognized at the event. More than 170 persons attended, far exceeding attendance of any other community STEM-H event conducted by other community teams.

Project Year 3 (Jan.-Dec. 2015)
Two major changes occurred near the start of year 3: project leadership changed and the number of participating teachers was reduced from 79 to 24. The RMEP co-director became the project director when VASS, Inc. President/CEO (and RMEP project director) accepted a position as superintendent of a school district in December 2015. VASS, Inc. consequently hired a new CEO/President, who also became the RMEP co-director. Reduction in teachers occurred as a redirection to increase implementation fidelity of the model of shared responsibility.
Evaluation results showed many teachers found it difficult to establish student Internet access accounts in the technology platform system and or make the Khan Academy homework assignments. Staff routinely had to set up the accounts for the teachers with their students in computer labs at the school. Limited computer lab space in schools made scheduling difficult. Persistent technology issues early in the project decreased enthusiasm for project participation among some teachers. Planning a family math night (FMN) was a new role for most teachers. Few had sufficient time for planning in a collaborative group. Many teachers simply considered this extra work, more demand on their time, with little real benefit for improving student performance on state standards of learning tests.
Unanticipated, many teachers also were particularly challenged in making presentations to adults. For many teachers, time for project implementation took a backseat to more familiar and valued classroom instructional activities. Mastery of state standards of learning in math and student performance on the high-stakes tests were the priorities for numerous teachers in deciding investment of class time and student homework activities. For some, discovery of the Khan Academy videos was a welcome instructional aide as test preparation. Also unanticipated, numerous teachers expressed a need for permission of their principals to engage more fully in RMEP activities, particularly family math nights. Many principals were surprised to learn this was the perception of their teachers.
Increasing teacher demands for continuous assistance of RMEP staff overwhelmed capacity.
Only 1 FTE math specialist position was budgeted in the grant award. Perhaps also, certain staff members, understanding the "situation" of teachers in the high poverty rural schools, were overly accommodating in assisting teachers in performing their roles. Many teachers seemed highly dependent on RMEP staff. Staff commonly had to identify and contact STEM-H technicians to present at the FMN activity. Numerous teachers had little time, even if they saw it as their responsibility, to invest in reaching out to parents in ways other than sending a letter home to "inform" parents of the FMN. Few teachers considered holding the FMN in locations other than the school.
RMEP staff concluded that fidelity of implementation would be exceptionally low unless the project's limited human and fiscal resources could focus adequately on the more "ready" teachers. Leadership Advisory Team members agreed. Certain teachers appeared substantially more ready to implement their roles and functions. Intensifying support for these teachers was determined essential to give the model a reasonable chance of becoming a promising practice. Therefore, the number of project teachers was reduced to a cohort of 24 "high implementation-ready" teachers. The 24 teachers taught 58 different classrooms of math students, representing a total of 1,134 students in 11 schools of the six school districts. Also, all students of the teachers received a tablet, not only those "in need" of better Internet access at home as originally planned.
Staff retrieved all tablets from the schools in May and worked over the summer to reset them to factory settings. The virtual private network (VPN) created in partnership with Verizon and another company enabled internet filtering without the use of specific software on tablets. This prevented students from hard resetting tablets to get around Internet filtering. However, this solution only worked for students able to get a strong Verizon data signal at home.
The Internet access issue still needed resolved for approximately 50 students, the majority enrolled in one district's middle school and high school. Where feasible, US Cellular hotspots were provided these students to gain Internet access on tablets. Staff also assisted teachers in making video assignments a week in advance to accommodate students needing to complete the online homework while at school with access to the district's Wi-Fi network. Staff also worked with Verizon to test and use new software that limited data usage to 1 gigabyte of data per month per tablet, thus eliminating student access to unnecessary apps such as music streaming and social media. The private technology provider continued making teacher use of the technology platform more efficient and effective. Staff conducted a needsfocused training workshop in summer of 2015 for the cohort of 24 teachers. School site follow-up support intensified. Consequently, all schools of the 24 high implementation ready teachers held a more teacherfacilitated family math night activity.
Two strategies helped the community teams conduct the STEM-H careers event. One strategy helped the team leader focus team planning on a specific idea, rather than brainstorming ideas in several meetings. This facilitated the team in coalescing around a "let's try it" action-oriented attitude that fostered efficient planning and selection of team members and others that could perform specific activities for the community event. The second strategy facilitated the community team's intentional collaboration with math teachers at the school(s) and focused on the Math at Work in Our Community activity.

Project Year 4 (Jan.-Dec. 2016)
Spring semester 2016 was the last semester in the original four-year project plan. At the end of the semester, Internet access for the tablets discontinued as planned in the vendor contract. Of the 1,107 students in teachers' classes, 83.7% completed at least one Khan Academy video or exercise. At least one assignment was completed by students in 47 of the 57 math classes (82.5%). Because 23 teachers made 1,195 assignments (videos and exercises), the total possible for completion by all students in all classes was 23,962. Students completed 6,313 assignments, or 26.3%, approximately the same as in the previous year (27%). The number of videos assigned by a teacher for a class ranged from 1 to 33 videos.
By the end of the semester, all five county community teams had held at least one community STEM-H careers event. On June 28, 2016 RMEP held a special conference to share project accomplishments and recognize partner school districts, Extension Service youth leaders, and others. A panel comprised of a teacher, parent, principal, and community team member commented on their experiences in developing the shared responsibility model. A second panel, comprised of students only, shared their experiences with Khan Academy homework assignments, family math nights, and community STEM-H careers events.
Throughout the project, the external evaluators conducted evaluation activities and held annual evaluation retreats with project staff to review evaluation findings. Staff continued to hold the annual ALT meeting, monthly calls with the USED i3 project officer, quarterly conference calls with the project's TA provider, biweekly staff meetings (with evaluators participating once per month), and meetings with superintendents and other district staff. RMEP staff continued to update resources on the RMEP website. Staff produced an annual project update newsletter and participated in the annual USED i3 project directors' meeting. Staff also participated in all webinars held by the USED i3 rural community of practice. Staff routinely presented project results at state, regional and national conferences.

Evidence of Promising Practice
RMEP leadership recognized from the project's beginning that significant positive impact on state test scores was unlikely during the model's "development" phase. It was unlikely that all three components of the model would be developed and implemented long enough in any school district to have collective impact on student test scores. Evolving challenges unique to implementation in each district reinforced this belief and contributed to the focus on only the more highly "implementation ready" teachers and sites in year three. Consequently, this challenged evaluators, whose four-year design aimed to measure fidelity of implementation and key mediators of impact as illustrated in the evaluators' theory of change logic model (see Nagle, et al., 2016, page 3). Interpretations of impact would likely be compromised with one year or less of evidence.
In July 2016 RMEP leadership received the final project evaluation report (Nagle et al., 2016). Evaluators found considerable evidence in survey results that features of the shared responsibility model were providing positive results for teachers, students and parents. Evaluators reported that the RMEP team completed five of the six core implementation activities established as standards of performance by the evaluation team. Although full implementation of the model was restricted to a single semester by the end of 2015, there was evidence that the higher level of support was beginning to have positive impacts on the teachers' assignment of videos and student completion of the assignments. Though attendance was lower than expected, evaluation results showed families and students who participated in family math nights and community events perceived them useful and worth the time. Teachers and community event team members reported that organizing the activities required a large time commitment and that they needed more help in identifying ways to increase attendance.
Evaluators found that the promising practice had no statistically significant impact on students' achievement or attitudes. Evaluators provided several reasons for their findings: 1. Differences existed between the content knowledge assessed on state exams and the content emphasized by the RMEP project. 2. State data availability limited the sample to only those students whose teachers implemented the intervention in the 2015-16 school year (the high-implementing teachers). Any effect may have been diluted by the inclusion of scores from students who did not have access to the intervention because their teachers were not participating in the RMEP project in fall 2015. 3. The project was only fully implemented in the fall 2015 semester, a short time period, which may have contributed to the lack of effect on achievement and attitudes. 4. The small sample size of students may not have been large enough to detect a very small effect. Evaluators concluded the RMEP project could serve as an example for other such initiatives, as the project helped reveal how similar projects should consider level of participant buy-in, anticipate and be able to troubleshoot technology access issues, and provide enough time as well as staff support for full implementation.
In essence, evaluation evidence and staff experiences revealed four years of development effort was required to address challenges unique to each school district's circumstances. During the June 28 conference RMEP staff conducted a special training institute for math teachers from nonpartnership school districts. Staff also held training sessions for non-project teachers in four locations across the state. Because of interest expressed in these and other information dissemination activities, staff considered how to best use unspent grant funds to improve the model in an extended year. Project evaluation evidence revealed family math nights and community STEM events held "promise" of positive impact for student, families and communities. But many teachers confronted time or other barriers in planning and conducting the family math nights. Parent and family participation often was much less than anticipated in both the family math nights and community events. Therefore, staff concluded greater supports for school personnel, parents and community teams using online technology should be the focus of remaining funds in the extended year.

Extended Year (Jan.-Dec. 2017)
The USED i3 office granted a one-year, no-cost extension for the project, with an end date of December 29, 2017. RMEP contracted with an independent evaluator to hold a focus group session with teachers in the high-implementation cohort; 10 of the 24 teachers participated. The evaluator reported strengths, challenges and sustainability issues related to their roles in the project.
Staff decided teachers, parents and community team members needed web-based supports and technical assistance. In particular, videos could help communicate a consistent message about the purpose and parts of the model-the "why" and "how" of shared responsibility. Four years of innovating the promising practice had revealed that communicating a consistent message took much more time than anticipated, from the beginning to the end of the project. And we discovered at the end that district leaders knew only about "pieces" of the projectusually the piece they were most involved with as a major challenge or success in their schools.
In fall semester 2017 staff selected two REAP eligible school districts in different rural regions of the state as pilot demonstration sites to try the refined model and supports developed in the spring and summer of 2017. Among the supports were: 1. New website with a video overview of the RMEP project and password-protected portals for the three role groups of teachers, parents/family members, and community team members. 2. Occupational profiles and data for seven STEM-H jobs in rural regions where the original six development and two pilot demonstration districts were located. 3. Community demographic profiles for the three regions served by the project. pilot site implementation of the model, and the online resources. 8. 10 short videos designed to give principals strategies for supporting innovative teachers. RMEP staff worked with leaders in the two pilot sites to select four math teachers, two per district. Teachers received tablets with necessary RMEP contents for all students. Staff revised the RMEP web site to include the supports. The RMEP math specialist and technology specialist trained teachers on how to use the tablets. Training included numerous phone and electronic communications and on-site technical assistance as necessary. Staff also provided on-site and electronic communications assistance for each community team. RMEP contracted with a former coordinator of the state REAP program to serve as a parent/family liaison consultant for the pilot sites. The consultant worked with district and school personnel to facilitate ideas and activities that could encourage parent and family participation in key online and school-or community-based activities.
In the fall semester, all four teachers assigned at least one Khan Academy video or related online exercise as homework. All 264 students completed some form of homework activity using the tablet, on a home computer, or at a location with Internet access (e.g., relative, town library). Each school held a family math night. Each teacher also gave students the online option in completing the Math at Work in Our Community activity. Parents and students could also view numerous web-based videos (e.g., STEM-H technicians).
Both school districts formed a community team and held a STEM-H careers event. One team, led by the school principal, created a new option for conducting the Math at Work in Our Community activity. Rather than have students (and parent or family member) travel to conduct the interviews at the workplace (or other location), the team organized 16 interview stations as an integral part of the actual community event held in a high school gym. Students, parents and family members rotated on a time schedule among three stations of their choice to ask questions of the person at the station about how math was used. This reduced parent/student travel across the county to a business site to conduct the interview. This also reduced the possibility that a person in a business would be interviewed multiple times by different students who needed to record answers on the interview card. Also, in the scheduled rotation approach multiple families could listen to answers of questions asked by different students.
Parent and student participation rates in family math nights and community events exceeded rates commonly experienced in the original six school districts. In the pilot school district with two participating high schools, however, attendance was much higher by parents/family members of students who were enrolled at the school where the community event was held.
To congratulate public school personnel and community teams on their success, VASS, Inc. held a 30-minute recognition event via videoconference webinar for each pilot school district. Each district could also invite parents, school board members and others to participate in the recognition event from home, business or onsite district videoconference location.

Implications for Theory and Practice
Sound theory guided design of the Rural Math Excel Partnership (RMEP) innovation. Collectively, specific functions of teachers, parents/families, and community members were expected to impact student academic success and attitude about STEM-H careers. This expectation was consistent with achievement motivation, attribution and expectancyvalue theories (Low & Jin, 2012;Singh, 2011). Consistent with the theory of functionalizism (Jaccard & Jacoby, 2020), a school could function in society to serve the individual and community.
Survey data and qualitative interviews analyzed by evaluators revealed attitudes were impacted positively, but not at a statistically significant level. Student Algebra 1 test results between project and comparison sites were not statistically significant. Evaluators offered several reasons. The reasons were consistent with information the U.S. Department of Education (Seftor, 2016) provides for school districts and others to understand valid reasons when an innovation or intervention is said to have no effect on a certain intended outcome. In particular, Seftor (2016, p. 1) notes: "In practice, a statistically significant finding may not be large enough to be meaningful. In other instances, a finding that is not statically significant may still be substantively important for practitioners or policy makers." Though the theoretical framework of the innovation may have been sound, evolving a promising practice in high poverty rural areas may have major implications for fidelity of implementation and measurement of intended impacts. As Hill and Erickson (2019) point out: "Estimates of implementation fidelity also can help explain null effects, in particular distinguishing between possibility that the program was not delivered as designed and other sources of failure, such as methodological problems, flaws in program theory, or lack of fit to local contexts" (p. 590). Though implementation challenges unique to each school and rural community occurred in all years, the project met four of the five indicators of implementation fidelity set by the evaluators. In many instances, RMEP's journey emulates the experiences reported by leaders in other USED i3 rural projects (Fox et al., 2017) and the challenges of evaluating "development" projects (Patton, 2011(Patton, , 2016.
Ongoing evaluation results aided critical decision making, such as deciding to focus on the 24 implementation ready teachers only, rather than attempting to increase implementation fidelity of all teachers. After the change evaluators began to observe more video assignments by teachers and their completion by students. Evaluators noted this finding suggests that the level of supports and resources necessary to promote buy-in and adoption by participants was greater than RMEP staff could meet originally. Project staff concluded the start-up decision to include all teachers of the four math courses in each district exceeded technical assistance capacity and greatly impacted implementation fidelity. Moreover, the evaluators had limited time and remaining budget to focus intensely on the 24 implementation ready teachers and remain consistent with evaluation standards of USED's What Works Clearinghouse. Certainly unclear was how collective impact of the teachers, families and rural community could be assessed in such an evolving implementation context.
A reflection on implementation challenges raises important questions about project organizational structures, capacities and evaluation needs. Innovating a promising practice may require formation of a research-practitioner alliance, collaborative or partnership that seeks to develop a solution in ways more consistent with recommendations of Bryk et al. (2015) for using networked improvement communities; suggestions of Penuel and Gallagher (2017) for creating researchpractice partnerships in education; or other examples of collaboration for developing innovations in rural places (Reardon & Leonard, 2018). These structures particularly accommodate evolving a solution to a problem of practice that keeps stakeholders and evaluation activities closely integrated. Evaluation must closely reflect needs of "development" challenges (Patton, 2011(Patton, , 2016, as iterations of improvement address changing contextual circumstances in the school and rural community. Both project staff and school district participants should embrace a learning-for-making-better approach to evolving the promising practice. This has implications for doing educational research in rural context (White & Corbet, 2014) and using research evidence (Gitomer & Crouse, 2019) to innovate meaningful solutions for problems of practice. Moreover, high poverty rural school districts may have a high need for solutions to practice challenges, but they also must be "ready" to perform the tasks required to innovate a solution. The organization or partnership that wins a grant must be ready to lead with enthusiasm and use research evidence to address challenges. If significant external funding and a partnership or collaborative approach are key catalysts for innovating solutions to educational challenges in a high poverty rural context, the answers to these 10 questions of readiness could have major implications for project success: 1. What understandings and experiences do member organizations and their leaders in the partnership have regarding collaborative leadership? 2. Do all school districts and or schools in the project have a genuine need for the promising practice, as defined by the intended users of the innovation? 3. Do intended users of the innovation have successful experiences in developing solutions to educational challenges as a team member with others in rural school districts? 4. Do intended users understand their roles and time commitments in an iterative development process, as compared to implementation of a compliance-oriented "proven practice" supported with required training and technical assistance? 5. Do technical assistance providers have successful experiences in rural school districts and the capacity to provide the extensive on-site and online assistance required to evolve the innovation? 6. Is using research evidence and reflecting on educational practices a common culture in the partnership members, school districts and schools? 7. Does the external funding source provide the flexibility in use of funds and evaluation requirements to accommodate failure in learning how to evolve the promising practice? 8. Can project leadership engage key stakeholders in making bold decisions that are consistent with the project logic model and evolving challenges in schools and communities? 9. Are the technology platform and other technology tools necessary for project implementation user friendly and compatible with technology infrastructure and student security policies of the school districts? 10. Are there enthusiastic advocates in each school district, school and community who can serve as champions to build support necessary for evolving and sustaining the innovation?

Conclusion
Developing an innovation for a difficult educational challenge in the world of practice is anything but a series of smooth, sequential steps to success. It is a journey with few guardrails and many potholes along the rural road to success. It may be a risky endeavor for school districts and partners to undertake, especially in a high-stakes testing environment with significant portions of students and communities in poverty. Large amounts of flexibility, patience, and perseverance will be necessary as all partners learn together. This includes those charged with the responsibility to evaluate the progress and impact of the development effort.
In RMEP, results of the ongoing implementation evaluation were highly prized for making decisions and developing the innovation. In reality, project staff experienced an iterative process in helping each role group evolve their respective functions in the shared responsibility model. Developing online videos and other supports evolved as a key strategy to address important time challenges for participation of teachers and families. Teachers evolved solutions in making video-based homework assignments for students without home Internet access. Community teams struggled to organize, yet evolved to plan and hold STEM-H careers events that families and students valued highly. This is the journey of innovating a promising practice in high poverty rural settings.
The RMEP project is one example of the challenges that await those who seek to innovate promising practices with important benefits for students and communities in rural places. Planning the journey should include knowing the answer to some critical questions of readiness. The RMEP project served more than 8,000 students and garnered more than $400,000 in private sector funds. It created a repository of resources that included the Math Advanced Study Guide, STEM-H technician videos explaining math use in workplaces, Family Math Night protocols, the Math at Work in Our Community activities, webinars, PowerPoints and other important implementation supports. After five years of development effort, one conclusion is clear: innovation in public education must continue as an investment in the people, place and prosperity of rural America. And it is the learning to collaborate and innovate together that may be the greatest challenge and reward.
Funding: This material is based upon work supported by the U. S. Department of Education Investing in Innovation (i3) program under Award No. U411C120091 to Virginia Advanced Study Strategies, Inc. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author and do not necessarily reflect the views of the U. S. Department of Education.