MTC DATA QUALITY MANAGEMENT PLAN

The Metropolitan Transportation Commission (MTC) is the developer of the StreetSaver pavement management software with more than 400 users nationwide, consisting predominantly of cities and counties. In addition to making StreetSaver available to local agencies, MTC uses the software to perform regional street and road condition and maintenance needs assessments for the 109 cities and counties in the San Francisco Bay Area. Since 2008, StreetSaver has been used exclusively to perform the California local street and road statewide needs assesment. The statewide assessment is used to inform the Governor and Legislature of the critical funding shortfall for maintaining local streets and roads, and the consequences of deferring or reducing transportation funds for cities and counties.

To ensure quality data from our Pavement Management Technical Assistance Program (P-TAP) consultants for our analysis, we have developed procedures and guidelines for managing the quality of pavement data collection activities.

MTC’s Data Quality Management Plan includes three components:

1. Consultant pre-qualification
2. Quality control plan--before, during and after production
3. Data acceptance

The pre-qualification process is part of the consultant selection and is an integral part of our data quality plan. The Rater Certification Program; however, ensures that raters are capable of providing the desired level of accuracy on pavement condition ratings. Some StreetSaver users at the local agency level, within or outside the Bay Area, may require that consultants’ raters be certified or the firm is pre-qualified by MTC to perform data collection.

I. PRE-QUALIFICATION

Pre-qualification of contractors is the first component of the pavement distress data quality plan. The pre-qualification process can ensure that the contracting agencies selected to participate in the P-TAP contracts are capable of collecting distress data that is reasonably close to what would be collected by an "expert" rater. However, that does not ensure that the data collected during the network-level distress data collection is reasonably accurate at the desired resolution and precision desired. In this discussion, accuracy indicates that the distress identified is the correct distress, e.g. alligator cracking is identified as alligator cracking instead of block cracking. Resolution indicates that correct severity levels are identified, e.g. low severity longitudinal cracking is identified rather than ignored or high severity longitudinal cracking is recorded as high severity instead of low severity longitudinal cracking. Precision indicates that repeated ratings produce reasonably similar recorded distress type, severity, density combinations.

Pre-qualification Sites

The pre-qualification sites should be residential streets in a small geographic area of a city within MTC region. Specific segments should be selected so that as many of the standard MTC eight asphalt surfaced distress types at the three severity levels as possible are included in at least one of the segments. The segments must be carefully rated using a walking survey conducted by very experienced surveyors. Extensive photographs and distress maps must be completed on all segments included in the pre-qualification sites. The survey data from these sites will be used to calculate the reference or "ground truth" PCI values of the segments. These segments must be full width of the paved surface by 100 feet long for the manual survey segments and 12 feet wide by 200 feet long for the semi-automated survey segments. The beginning and ending of each segment must be marked on the street, and the corners must also be also marked for the semi-automated segments. All segments selected for the pre-qualification test will include pavement conditions in very good condition (PCI greater than 75), very poor condition (PCI less than 20), and moderate condition (PCI less than 75 and greater than 20).

Data Collection by Potential Contractors

The potential contractors should rate up to 24 segments within a period of time (generally from 8 am to 4 pm) as specified by MTC. The distress data should be provided to MTC staff immediately after the surveys are completed or at the end of the testing period so that they can be entered into the MTC StreetSaver software by MTC or contracted staff to calculate the PCI values.

Acceptance Criteria

The following acceptance criteria should be used:

1. At least 50 percent of the PCI values for the rated sections must be within +/- 8 PCI points of the reference, or "ground truth," PCI values.
2. No more than in 12 percent of the PCI values for the rated sections can be greater than +/- 18 PCI points of the reference, or "ground truth," PCI values.

MTC Certified Contractor
Potential contractors that passed the pre-qualification test will be certified by MTC for eligibility to submit Statement of Qualification (SOQ) for P-TAP contracts. This certification is valid for two (2) years and can be renewed upon passing a pre-qualification test.

II. QUALITY CONTROL PLAN

Each qualified firm is responsible for providing quality data to the project sponsor and MTC. In the Quality Control Plan (QCP), each firm will be required to describe the qualifications of each rater who will collect pavement distress data, including training and experience. If new raters will be assigned to collecting pavement distress data, the data collection contractor will need to describe the training and supervision that will be exercised during their "internship" period. The qualifications of the new raters must be provided to and approved by MTC before they will be allowed to conduct ratings without the direct supervision of experienced raters.

All firms should describe the data verification processes that they will employ to validate accuracy, resolution and precision of the data collected. "Accuracy" indicates that the distress is identified correctly, e.g., alligator cracking is identified as alligator cracking instead of block cracking. "Resolution" indicates that severity levels are identified accurately, e.g., low severity longitudinal cracking is identified and recorded rather than omitted. "Precision" indicates that repeated ratings produce reasonably similar recorded distress type, severity and density combinations.

Data verification processes must include:

1. Periodic re-survey of "control" sections by survey teams at least once every two weeks.
2. Re-survey of at least 5 percent of the sections previously rated within one month of completing surveys. The same survey team may do the re-surveys.
3. Re-survey of at least 5 percent of the rated sections by a supervisor.
4. Checks of collected data against prior survey data and checks of calculated PCI values against PCI values based on prior survey data projected to the survey date for the same section if no treatments have been applied since the prior survey. All of those outside plus or minus 15 PCI point differences should be checked by a supervisor or other person approved by MTC.

The QCP must define what results will be considered acceptable; at a minimum, results must meet the acceptance criteria as defined in the pre-qualification. The firm must also describe which remedial actions will be taken if the results of the data checks are not acceptable. The Plan must identify when, in what format, and how often the results of the quality control plan checks and corrective actions will be submitted to MTC. The Plan must also identify when the survey results will be entered into StreetSaver software, the PCI values calculated, and the results made available to MTC. Qualified firm must submit a Quality Management Report for acceptance by the project sponsor. A copy of the sample report is available here. 

Rater Certification Program

Under the P-TAP, even if a firm has pre-qualified, all of the firm’s raters must be certified by MTC through the Rater Certification Program (RCP). The RCP consists of passing a pavement distress manual survey exam, and an online written exam. These exams are designed to evaluate the skills and knowledge of raters on pavement condition rating based on the MTC’s seven-distress protocol. A rater must rate up to 24 test sites and achieve the required level of accuracy performing ratings as defined by the Acceptance Criteria (see above). Raters from non-P-TAP consulting firms and local agencies can be certified. The rater certification exam will be conducted at lease once a year or as needed. Upon successful completion of the exams, a certificate will be issued that is valid for two years. Renewal is good for another two more years upon successful completion of an online refresher course. Every four years, all certified raters must take the field and written exams to be re-certified.

Registration for the Rater Certification Exam is now ​closed.

III. QUALITY ACCEPTANCE PLAN 

MTC staff will need to communicate with the data collection contractors about issues found such as the sections for which the calculated PCI values do not match the projected PCI values to determine how to approach further steps. If the data collection contractor does not meet the requirements of the data collection quality control plan or if MTC determines that the collected data does not meet the requirements established in the prequalification requirements, MTC can issue a stop work order and require corrective actions. This may include requiring the data collection contractor to re-qualify all of the survey teams being used to collect data under the P-TAP contract, and re-surveying all sections rated since the last checks that showed the contractor was meeting contract requirements. Other less onerous corrective actions may be imposed by MTC on the data collection contractor. Multiple violations could result in termination of the data collection contract. MTC or its contracted staff, the California State University at Chico (CSUC) will administer the Quality Acceptance Plan.

1) Administer Rater Certification Program

Because contractor qualification for the P-TAP does not ensure that all raters are capable of rating with the desired level of accuracy, all raters employed by the qualified contractors will need to complete rating of the same survey sites used in the P-TAP qualification tests, or new sites as directed by MTC.

The Rater Certification Program consists of a field pavement distress survey test and an online written test. CSUC shall facilitate pavement distress surveys and online written tests for raters up to twice a year. When needed, CSUC shall establish ground truths for pavement distress survey testing. Raters from non-qualified consulting firms and local agencies are also eligible and can be certified. CSUC ’s services shall include administering registration, field and online tests, issuance and renewal of certification, and database tracking.

2) Conduct Audits of Contractor's Quality Control Plan 

CSUC shall verify that the quality control plans (QCP) adopted by the data collection contractors are being completed prior to project sponsor acceptance of survey results. CSUC shall conduct on-project site audits. CSUC shall conduct audits of the QCP results to ensure that the data collection contractors are meeting the requirements established in their plans.

The requirements of the QCP are:

1. Re-survey of "control" sections by survey teams at least once every two weeks.
2. Re-survey of at least five (5) percent of the sections previously rated within one month of completing surveys. The same survey team may do the re-surveys.
3. Re-survey of at least five (5) percent of the rated sections by a supervisor.
4. Checks of collected data against prior survey data and checks of calculated PCI values against PCI values based on prior survey data projected to the survey date for the same section if no treatments have been applied since the prior survey. All of those outside plus or minus 15 PCI point differences should be checked by a supervisor or other person approved by MTC.

CSUC shall spot check and/or conduct full audits of the QCP from selected projects as directed by MTC.

3) Verify Data Collected by Contractors

CSUC shall conduct data verification actions including at least some of the following:

1. Checks of collected data against prior survey data and checks of calculated PCI values against PCI values based on prior survey data projected to the survey date for the same section if no treatments have been applied since the prior survey.
2. Survey of sections previously rated by the data collection contractor.

For more information about the MTC Data Quality Management Plan, please contact ​us.