» Download PDF Version

Background

Like many other Federal programs, the U.S. Department of Education's Program for the Education of Children and Youth Who Are Neglected, Delinquent, or At-Risk of Educational Failure is responding to Congressional calls for increases in accountability. The first critical step in responding to that call is to attain high quality data on the program's outcomes. This data brief describes the efforts to improve data quality, the Federal context in which these changes are occurring, why the changes are necessary for the success of the program, and what States and grantees may do to facilitate the changes. This brief is part of NDTAC's ongoing Data Quality Initiative, which is directed at maximizing the quality and integrity of data reported by Title I, Part D programs.

Federal Context: GPRA and PART

In 1993, in order to improve efficiency and accountability in Federal programs, Congress passed the Government Performance Results Act (GPRA), which obligated Federal agencies to develop rigorous performance measures in order to demonstrate program improvement.¹ However, GPRA in many ways did not live up to its legislative intent,² so in 2002 the Office of Management and Budget (OMB) developed "PART."

The Program Assessment Rating Tool (PART) is a review process led by OMB that assesses every program in the Federal government and assigns a score to each of them. That score is based on assessments of the program's purpose, design, management, and performance. The ratings are meant to be compared and used by Congress for appropriations. In fact, the PART score is submitted along with the budget request for each Federal program. Figure 1 illustrates the most recent PART review ratings, conducted in 2006 for the fiscal year (FY) 08 Federal budget.³

Although the Federal budget process is influenced by a variety of factors, PART is intended to provide the executive branch and Congress with at least one consistent and formalized review of program effectiveness to inform funding decisions. Programs that cannot demonstrate results or are deemed ineffective are candidates for greater scrutiny and potential funding cuts. For example, a review of transcripts from Congressional budget documents will show a few lines like this one, "The program received a rating of 'ineffective' by the 2006 PART Assessment. It is duplicative of numerous other programs and has demonstrated no measurable success."⁴

Title I, Part D and the Federal Review Process
Title I, Part D underwent its first PART review in 2005.⁵ At the time of review, the first comprehensive Part D data collection had not been completed, and no quantitative information was available to demonstrate performance. For this reason, the program was categorized as Not Performing and received a rating of "Results Not Demonstrated." In spring 2007, after 2 years of expanded data collection, it became possible to assess the impact of the Part D program, at least preliminarily, and the program was reviewed again.⁶ The program was categorized as Performing and rated as "Adequate," a significant achievement considering the relatively short period of time in which the program built up its data collection process.⁷

Title I, Part D Federal Performance Measures
Although Part D performance measures continue to evolve, the program has delineated three academic performance measures as of spring 2007. The three measures are aligned with the key aspects of the program's mission to improve educational programming for youth who are neglected (N) or delinquent (D) and to promote their successful transition from institutionalization to further schooling and employment. To reach its goals, the program as a whole must demonstrate a 5 percent increase in the number of students achieving each of the key indicators from year to year. Table 1 displays the Part D measures, the performance targets for each measure, and the results based on data collected thus far.

Table 1. Results for the Title 1, Part D Academic Performance Measures^†

Measure 1: The percentage of students who are N or D obtaining a secondary school diploma or diploma equivalent			Measure 2: The percentage of students who are N or D earning high school course credits			Measure 3: The percentage of students who are N or D who improve reading skills as quantified on approved and validated measures‡
Year	Target	Performance	Year	Target	Performance	Year	Target	Performance
2005	Set baseline	10.50	2005	Set baseline	55.95	2005	Set baseline	72.53
2006	11.03	11.23*	2006	58.75	47.68*	2006	76.16	70.11*
2007	11.58	(June 2008)	2007	61.68	(June 2008)	2007	79.96	(June 2008)
2008	12.16	(June 2009)	2008	64.77	(June 2009)	2008	83.96	(June 2009)
*Year 2 target met			*Year 2 target unmet			*Year 2 target unmet
† These indicators were calculated using Subpart 1 data from the CSPR. Baselines were set using the data from SY 2004–05. ‡ Future Federal reports will also include a fourth academic performance measure, calculating the percentage of neglected or delinquent students who improve mathematics skills, as quantified on approved and validated measures.

 

As the data in Table 1 show, the program did not meet two of its three performance targets in 2006. It is likely that the baselines set and the changes in performance reflect data quality issues rather than actual changes in student or program performance. Especially in the first year of data collection, programs were becoming familiar with the new collection instruments and States did not have data-collection systems fully up and running in advance of school year (SY) 2004-05 collection.

The remainder of this brief examines the importance of data quality and the common problems in the data reported by N or D programs over the past 2 years.

Why Is Data Quality Important to Title I, Part D Programs?

As indicated above, at the national level, the Title I, Part D program needs reliable data to be able to demonstrate student progress to Congress and taxpayers. However, the importance of collecting high-quality data extends beyond Federal reporting requirements to the State and local level. States may use data reported by the facilities and programs under their jurisdiction to understand the effectiveness of N or D education and to make educated management and funding decisions directly affecting the students in those facilities and programs. Analyzing data from multiple programs, States may identify and share "best practices," thus elevating the quality of education across the State. States can also use the data in reporting to legislators, State agencies, and nongovernmental foundations to win support for their programs.

Individual programs need accurate data in order to gauge whether their students are benefiting from the educational instruction provided and to make informed decisions about these programs. Depending on the comprehensiveness of the local data collection, analyses can help identify which students or groups of students are making the greatest gains, which students or groups of students need additional targeted intervention, or which programs of instruction most effectively raise student achievement. However, the information gleaned from data is only useful if the data are of high quality.

Real Change or Impact of Data Quality?

In June 2007 NDTAC launched a programwide Data Quality Initiative, with the goal of helping States understand how their data are used and, at the same time, encouraging data managers to examine data more critically and think more broadly about data quality. This section provides examples of changes demonstrated in the Title I, Part D data over the past 2 years as reported by States, which, depending on the quality and accuracy of the data, may or may not reflect actual changes that occurred in the field.

Programs Funded Under Title I, Part D
Although overall State funding levels have remained fairly constant-the average change in allocation to each State amounted to only 1.7 percent of the total allocation for SY 2004-05-the proportions of those dollars spent on particular program types (e.g., juvenile detention, adult corrections) appears to have shifted from SY2004-05 to SY2005-06.

The change in programming was seen both in the types and total number of programs funded. Figure 2 illustrates the changes States made in allocating Part D funds to entire program types, such as neglect programs or detention programs. It appears as though some States discontinued Title I, Part D funding to particular program types, while others offered funding to program types they had not funded in the previous year.⁸ The most notable example of this is the rise in the total number of States allocating Part D funding to "at-risk/other" programs; the number doubled from 11 States in SY 2004-05 to 22 States in SY 2005-06.⁹

Figure 3 captures the fluctuation in the total number of programs receiving Part D funds, per program type, from SY 2004-05 to SY 2005-06. Most striking is the increase in the number of neglected programs in Subpart 2 and juvenile detention programs in both Subparts, doubling or nearly doubling from the previous year.

There are a number of reasons why the funding distributions may change from year to year. The differences may reflect actual changes in funding decisions or data quality changes. States are able to determine program funding on an annual basis and distribute the funds based on need. Facilities and programs may also refuse funding or close down, and monies may be reallocated. Thus, some variablity in funding is expected from year to year. Other factors, both internal and external to the programs, may also have an impact on their ability to report accurate data (see Factors Having an Impact on Data Quality below).

Student Counts
In the SY 2004-05 data collection, States reported a total unduplicated count of 110,115 students for Subpart 1. In the SY 2005-06 data collection, the reported unduplicated count totaled 135,607 students, an increase of over 23 percent. This expansion was not isolated to the unduplicated count, but was consistent with other demographic data, such as the total number of students reported in the age subgroups,¹⁰ as depicted in Figure 4. Why did these counts increase? Student populations and the juvenile crime rate may certainly fluctuate, and it is possible that the Part D program served 25,000 more students in SY 2005-06 than in the previous year. However, closer examination of the individual State data suggests that the rise in the number of students reported may stem from data-collection issues for one or both years: 13 States indicated a 75 percent or greater increase in the population of students who are N or D. Figure 5 portrays the percentage of change in the population served for each State from the first to the second data collection.

The integrity of the student counts is especially important to any performance review process-at the Federal, State, or local level-as most calculations of student progress (e.g., percent of students obtaining a high school diploma) rely on accurate estimates of the student population served by Title I, Part D. More specifically, in the formula to calculate the percentage of students attaining an outcome, the population estimates are the denominators. Thus, as the denominators increase and the numbers of students attaining the outcomes remains the same, the percentage attaining the outcome decreases.

Because the program as a whole does not yet have historical data that would allow for predicting normal fluctuations in the population of students who are N or D, it is all the more important for States and individual programs to monitor the quality of their data. Assuming educational programming remains the same, it is expected that a true increase in the student population would be echoed by similar increases in student-outcome data. When the student population is incorrectly reported, especially if it is inflated, it becomes difficult to accurately reflect student progress and demonstrate the effectiveness of Title I, Part D programs.

Factors Having an Impact on Data Quality

The program and student-count data provide a quick look at the types of changes taking place between the first and second collections. Without long-term results, it is difficult to determine which changes are real and which changes are the result of poor data quality. Collecting high-quality data is an effort that requires diligence and commitment from parties at all levels—the local programs, which collect the student-level data; the State data managers, who compile and enter the data; and the agencies that develop the collection instruments. Collection of missing, incomplete, and/or inaccurate data for Title I, Part D may be the result of a number of factors: data collection instruments or instructions were confusing or misunderstood; the reporting questions and/or program categories¹¹ were adjusted in between data collections; individual programs were unable to provide data, or could only provide partial data; data-collection systems were not fully in place during one or both of the collection periods; data were incorrectly entered during the data entry process; or data were not reviewed for accuracy at the program or State level—either within the current reporting year or in comparison to previous years.

Alone, or in conjunction, these factors can have a serious impact on the accuracy of the data reported. Even minor data quality problems at the local level can influence the Title I, Part D data at the national level. Developing processes for ensuring data accuracy are best initiated at the individual program level, and programs should be active in communicating to their State and local education agencies if there is any confusion regarding the type of information the State and Federal reporting forms request and the types of resources necessary to help make data collection accurate and efficient.

What Steps May Be Taken To Improve Data Quality?

As a part of the Data Quality Initiative, NDTAC prepared individualized data quality reports for each State based on the Subpart 1 data it provided in the CSPR for SY 2005–06. States should ensure that they understand their scores and what data quality problems were found.

In preparing the State data quality reports and other analyses, NDTAC identified four data quality issues posing the greatest problems for Federal reporting:

• Lack of alignment between the demographic and student-count data (e.g., the total for males and females did not equal the total for all the age groups)
• Missing data on the key academic outcomes used in Federal program reviews¹²
• Reporting a greater number of students achieving outcomes than the number of students served¹³
• Discrepancies in the academic performance data¹⁴

States and programs are encouraged to conduct and expand on these reviews on their own. For example, the reviews did not provide comparisons to SY 2004–05. States should consider making direct comparisons between reporting years and noting where data points have changed by more than about 10 percent.

The importance of quality data to the Title I, Part D program cannot be stressed enough. At the Federal, State, and local levels, poor data quality can lead to unreliable results and potentially false information regarding program performance. Thus, decisions regarding program funding or effective practices may be negatively affected. Everyone involved in the Title I, Part D program—administrators, teachers, and most importantly, students—stands to gain from the effort to collect high-quality data.

NDTAC seeks to provide ongoing technical assistance to States regarding the data collection and reporting process, and welcomes feedback on any products or tools that may help States to effectively collect accurate data.

This ND Data Brief was prepared by Tal Kerem, Stephanie Lampron, Natalia Pane, and Tarek Anandan of the National Evaluation and Technical Assistance Center for the Education of Children and Youth Who Are Neglected, Delinquent, or At Risk (NDTAC).

Resources

» Learn more about GPRA and PART.

» View State-specific data for Title I, Part D on the NDTAC State Fast Facts pages.

» Find your NDTAC State liaison and request technical assistance on the Direct Assistance page.

Notes