Model Validation, Infrastructure Issues Hamper Machine Learning Implementation

LAS VEGAS — Lenders are becoming more adept at defining use-cases for machine learning, but infrastructure limitations and model validation are hindering implementation, panelists said at the American Financial Services Association’s Vehicle Finance Conference yesterday.

At Toyota Finance Services, for instance, machine learning has streamlined the process for updating scorecards. “Most of us have traditional scorecards we deploy,” said Scott Cooke, the captive’s chief risk officer. “In a very primitive sense, when we start with scorecards, we take information that we would traditionally run through a scorecard, go through a decision-tree of yes-no answers, and we would look at those scorecards every six months, one year, or two years, depending on the scorecard. We would refresh [the scorecard] with some research, build those scorecards, and put them back in production.”

Machine learning has shortened those cycles — a plus for TFS — and the system is “learning for itself” in real time, Cooke said. The challenge, though, is implementing those changes. “As an indirect lender, dealers don’t want us to change our underwriting criteria every day,” he said. “So, the capability to do that is nice, but that doesn’t mean the business model syncs in.”

Another drawback to machine learning relates to model validation. “How do we ensure that we’re putting in data that is accurate, complete, correct, and timely?” Cooke said. “Being able to take things that may give you a better answer, but you can’t explain or understand them — there’s no transparency. If you look at the research, there’s not a lot of performance data on any of these models, so we don’t know that they’re better.”

Outdated systems can also put a wrench in machine learning implementation. “While we have a lot more data and a lot more computational horsepower, many of us have data sitting in different places, and it’s not that easy to aggregate,” Cooke said. “We all have some legacy system issues that may prevent you from — at the moment that a customer calls in and wants to chat with a chatbot — aggregating the customer’s first three cars with their checking account to deliver the right answer; it sounds good in theory, but you still have to be able to pull that off.”

More broadly, companies mulling deployment of artificial intelligence are often confused about “the pain points” — for the business and for the customer — that they are trying to solve, said panelist Alecia Bridgewater, a consultant at Bridgewater Consulting. “That’s a big one to get clarity on,” she added.

Other obstacles to overcome in an AI deployment: an understanding of the kind of technology to use, and how the project will be funded and monetized.

For more content like this, attend the Auto Finance Performance & Compliance event, slated for May 9-10, at the Omni Dallas. For information, or to register, visit autofinanceperformance.com.