By Chris Burroughs, Transportation Intermediaries Association (TIA) –
The much-anticipated report by NAS was released today… The National Academy of Sciences (NAS) was tasked by Congress in the FAST Act, to conduct a thorough comprehensive examination of the Federal Motor Carrier Safety Administration’s (FMCSA) Compliance, Safety and Accountability (CSA) initiative. This study was commissioned because of concerns from motor carriers, shippers and brokers about the validity of the system, and the impacts of the scores on third-parties.
In summary, the NAS found that the CSA and Safety Measurement System (SMS) is:
Structured in a reasonable way, and its method of identifying motor carriers for alert status is defensible. However, much of what is now done is ad hoc and based on subject-matter expertise that has not been sufficiently empirically validated. This argues for FMCSA adopting a more statistically principled approach that can include the expert opinion that is implicit in SMS in a natural way.
Additionally, the NAS based on the current information available to them, was unable to make a recommendation if a motor carrier’s SMS results and scores should be made publicly available. The NAS recommends that the Agency conduct a detailed review and analysis of the potential impacts to industry and the public of making those scores publicly available. Ultimately, this means that the scores will likely remain hidden from view until further analysis is conducted.
Specifically, the NAS makes the following recommendations to the FMCSA for improvements:
- FMCSA should implement an item response theory (IRT), which is a paradigm for the design, analysis, and scoring of tests, questionnaires, and similar instruments measuring abilities, attitudes, or other variables. It is a theory of testing based on the relationship between individuals’ performances on a test item and the test takers’ levels of performance on an overall measure of the ability that item was designed to measure. Several different statistical models are used to represent both item and test taker characteristics. This would:
– move towards leaning on data alone, and not expert opinions,
– enhance transparency of the evaluation system,
– support the direct estimation of variability of scores and rankings, and
– Allow the Agency to adapt to future changes.
- 2. FMCSA should continue to work with State Agencies and law enforcement to improve the quality of data in MCMIS. The two areas in need of immediate attention would be: carrier exposure and crash data and fault. The NAS calls for the development of a National Minimum Uniform Crash Criteria to be implemented nationwide.
- 3. FMCSA should investigate new ways of collecting data that will likely benefit the recommended methodology for safety assessment. This would include data on carrier characteristics – including information on driver turnover rates, type of cargo, method and level of compensation, and better information on exposure.
- FMCSA should structure a user-friendly version of the MCMIS data file used as input to SMS without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers. In addition, FMCSA should make user-friendly computer code used to compute SMS elements available to individuals in accordance with reproducibility and transparency guidelines.
- 5. FMCSA should undertake a study to better understand the statistical operating characteristics of the percentile ranks to support decisions regarding the usability of public scores.
- FMCSA should decide on the carriers that receive SMS alerts using both the SMS percentile ranks and the SMS absolute measures, and the percentile ranks should be computed both conditionally within safety event groups and over all motor carriers.
TIA participated and provided testimony during one of the NAS public meetings, where we advocated on behalf of the third-party logistics industry to keep the data from being displayed on public view, and outlined the liability concerns of having the scores publicly available.
If you have any questions, please contact TIA Advocacy at advocacy@tianet.org or 703.299.5705.