The self-driving car industry is currently confronting a critical juncture, particularly highlighted by Cruise’s recent recall of its entire fleet following a tragic accident involving a pedestrian. This incident prompted the California DMV to suspend Cruise’s operation of autonomous vehicles. Simultaneously, activists in San Francisco are actively protesting the use of their city as a testing ground for driverless cars.
In the midst of these challenges, Ghost Autonomy, a startup specializing in autonomous driving software, asserts that it possesses the solution to enhance the safety of self-driving technology. The company recently announced plans to explore the integration of multimodal large language models (LLMs) — AI models proficient in understanding both text and images — into self-driving systems. Ghost has partnered with OpenAI through the OpenAI Startup Fund, gaining early access to OpenAI systems, Azure resources from Microsoft, and a $5 million investment.
According to Ghost’s Co-founder and CEO John Hayes, LLMs provide a novel approach to understanding complex scenarios, adding reasoning to situations where existing models fall short. Ghost aims to leverage multimodal models for higher complexity scene interpretation, offering suggestions for road decisions based on images from car-mounted cameras.
However, skepticism surrounds Ghost’s approach. Critics argue that using LLMs for self-driving is a marketing tactic, likening it to the buzz surrounding blockchain in 2016. Os Keyes, a Ph.D. candidate at the University of Washington, questions the suitability of LLMs for autonomous driving, asserting they were not designed or trained for this purpose. Mike Cook, a senior lecturer at King’s College London, echoes these concerns, emphasizing the challenges of applying unpredictable and unstable technology to driving.
Despite skepticism, Ghost remains undeterred. The company believes that LLMs could enable autonomous driving systems to comprehend driving scenes holistically, utilizing broad-based world knowledge to navigate complex situations. Ghost is actively testing multimodal model-driven decision-making and collaborating with automakers to validate and integrate new models into its autonomy stack.
While Ghost acknowledges that current models are not yet ready for commercial use, it expresses confidence that application-specific companies like theirs, with extensive training data and a deep understanding of the domain, will significantly enhance the reliability and performance of general models over time. However, given the setbacks experienced by well-established players in the self-driving space, questions remain about the feasibility of Ghost’s ambitious claims and the unproven nature of the technology involved.
