7 Things You Need to Know About Upcoming CSA Changes

The Federal Motor Carrier Safety Administration’s CSA program (Compliance, Safety, Accountability), while a good idea in theory, has been under attack by the industry for numerous shortcomings since before it was officially rolled out in 2010. Now it’s in the process of being significantly revamped on recommendations from the National Academy of Sciences. We have some highlights on what to expect:

1. CSA is changing to address shortcomings

CSA (which originally stood for Comprehensive Safety Analysis) was supposed to give FMCSA a better way to identify carriers that warranted a compliance review. That’s because there are vastly more carriers, especially small ones, than the agency can review.

The new program promised to use inspection results and other data to come up with scores that would signal which unsafe companies should be targeted for “interventions,” which could be a warning letter, compliance review, or other action. Eventually, the system was supposed to be used for issuing safety-fitness determinations for motor carriers.

But from the beginning, CSA was fraught with problems, including the fact that brokers, insurers, attorneys and others were using the publicly available scores in ways DOT never intended. Eventually trucking convinced Congress, in 2015, to order FMCSA to shut down public access to that data while it revamped the program. And Congress had a very specific way it wanted the agency to address it – by getting actual scientists involved from the National Academies of Science, Engineering and Medicine, a D.C. think tank retained by governments all over the world to tackle challenging problems like water shortages in Africa and life in outer space.

In the report from the NAS panel set up to study CSA, it noted that the system has been criticized for, among other things:

• Using highly variable assessments
• Not accounting for crashes where the motor carrier is not at fault
• Including carriers that have very different tasks in the same peer groups
• Using measures that are sensitive to effects from one or more individual states
• Using measures that are not predictive of a carrier’s future crash frequency
• Using measures that are not reflective of a carrier’s efforts to improve its safety performance over time.

2.  The New CSA Methodology Will Use Item Response Theory

The NAS panel’s report recommended that FMCSA develop a “more statistically principled approach” for the task, based on an item response theory (IRT) model. IRT is widely used in the education field (think SAT tests), and has been used for policy decisions in areas such as hospital rankings.

At the heart of CSA is the Safety Measurement System, used to identify commercial motor vehicle carriers at high risk for future crashes. But NAS found that some of the details of how those numbers are calculated weren’t really scientifically based, or as the report said, “ad hoc and not fully supported by empirical studies.” The scientists on the panel say IRT is a more scientific way to approach these safety scores.

3. FMCSA is Working on an IRT CSA Model

In July, FMCSA delivered a report to Congress on its “corrective action plan” for CSA outlining how it’s implementing the NAS recommendations. And it is indeed moving forward to develop and test an IRT model.

The FMCSA hasn’t released many details to the public about what that IRT model is going to look like. However, Steve Bryan, executive VP and GM, SambaSafety Transportation (you may know him better as the founder of Vigillo, which SambaSafety now owns), has been in every public meeting of the NAS panel as it developed its recommendations. In a recent presentation for the Motor Carrier Insurance Education Foundation, he described IRT as “very complex but time-tested science.” His company has licensed sophisticated IRT modeling software and developed its own version of what it believes FMCSA will develop, which Bryan says customers can use to get a feel for how they’re going to perform under the new CSA system.

FMCSA In July said it planned to run a small scale IRT model by September of 2018 and after evaluating the results, would run a full scale IRT model by April of 2019 and get it rolled out fully by September of 2019. While the agency doesn't have the best track record of meeting deadlines, Bryan did emphasize that this is something that’s going to happen, and fleets need to be prepared.

4. Safety Culture Will Replace Crash Risk

“We’re moving away from pretending we can predict future crashes and toward evaluating safety culture,” Bryan said.

One of the things the NAS panel was tasked with was looking at whether CSA scores actually predict future crash risk. “The existing CSA program is about predicting crash risk,” Bryan said. “None of us believe that ever worked. It does a terrible job. In some of the BASICs, some are not only not positively correlated, they’re negatively correlated, specifically the drug and alcohol BASIC — to where if you followed that logic, you should drink and smoke dope and get behind the wheel of a truck.”

The new IRT method, he explained, moves beyond the seven BASICs and creates a single safety culture score.

He said once FMCSA enacts all the planned changes to CSA resulting from the NAS review, there will no longer be violation weights, CSA points, BASIC measures, or Safety Event Groups. Instead, FMCSA will introduce a new, single CSA BASIC score that will be “a very different way of representing the safety culture of a motor carrier.

“From a methodology perspective, it actually works pretty well,” he said. “We’ve done some early research on the scores, and it is very well aligned that carriers with lower safety culture scores have more crashes. I think at the end of the day it will help find risky carriers, even though the IRT [model] isn’t designed to predict crash risk.”

5. Variety is Not Good for Your CSA Score

“A variety of violations will tank your scores,” Bryan said.

“Say we take the same test, and you do well in four out of five sections,” he explained. “I, on the other hand, am mediocre across the board. IRT hates me. It will score me very poorly. It doesn’t like the fact that I didn’t excel anywhere and I have a variety of poor responses. It sees that as a problem. I’m just dumb. I didn’t get anything very well.

“Think of that in terms of trucking. If you have a few violations in one area, IRT isn’t going to hit you very hard. But if you get all kinds of violations from a variety of sources, you’re just not minding the store very well.”

And that ties right in to the idea of a “safety culture” score.

6. Patterns Replace Points

Bryan said severity weights, which NAS targeted as unscientific, will be gone under the new program, along with violations being weighted by how recent they are. In addition, he says, those annoying percentile rankings will be gone. (That’s been a big complaint – you could be a very safe carrier under the existing system, but if your score is worse than others in your “peer group,” you’re in trouble.)

So how will IRT assign scores?

Bryan said it will look at the 66 “violation groups” that exist in CSA. Take lighting, for instance. Within that violation group, he explained, violations related to lighting are collected – such as headlights, turn signals, marker lights, etc. IRT, he said, looks for patterns across the industry and uses it to assign weights or priorities to these violation groups. Bryan said IRT can determine what things should carry more weight because they are “a gateway to lots of other bad things happening.”

IRT also takes into account  “exposure,” such as power units, driver count, inspection count, and vehicle miles traveled. “The science of IRT can extract out of there and discard exposures so you can essentially make everyone look alike.”

So instead of being compared to an arbitrary “peer group,” Bryan said, carriers will be compared to everybody, but IRT will “normalize everybody using a continuous pattern of exposure.”

7. Data is Still a Problem

The NAS report also identified a number of data quality issues. For instance, it said FMCSA needs to improve the collection of data on vehicle miles traveled by state and month, to help account for factors such as ice winter weather in the North. Another area it cites as lacking is information related to the contributing factors in a crash. It even said the agency should look at collecting data such as driver turnover rates, type of cargo, and driver compensation.

Bryan said of FMCSA’s plans that he’s a fan of the proposed IRT methodology, but the agency “largely ignored some of the data quality questions that still linger.”

When it comes to the data recommendations put forth by the NAS panel, FMCSA’s report basically said, “we’ll have to research that more.” For instance, while agreeing that better VMT data would help, it noted that it only collects that information from carriers every two years (via the MCS 150 form), and access to this data by state on a monthly basis is not currently feasible.

CSA has been in the top 10 of the American Transportation Research Institute’s annual top trucking issues survey every year since its introduction. The past couple of years, it has slipped to the middle of the pack as the industry waited to see how FMCSA would fix the troubled system. “I would guess in 2019,” Bryan said, “We’re going to see CSA slammed right back to the top.”

Originally posted on Trucking Info