Japanese venture capital group SoftBank claims to have developed an autonomous vehicle AI that improves safety and knowledge of complex driving scenarios.

Currently, driverless vehicles function alongside a remote human operator who must intervene when an autonomous vehicle approaches traffic lights or fails to perform a stop.

With this, issues arise as remote operators typically monitor multiple vehicles at once, putting pedestrians and passengers both at risk if dangers are not detected immediately.

Softbank is using multimodal AI to tackle this issue – a type of artificial intelligence that can understand and work with multiple kinds of information, like text, images, audio, and video, all at once.

Multimodal AI is applied to self-driving cars because it allows the vehicle to process and combine multiple types of information at the same time, which is essential for making safe driving decisions.

SoftBank claims that its multimodal AI has the ability to generate real-time information (collected by camera footage from the autonomous vehicle) on traffic conditions and issue instructions directly to the autonomous vehicle itself.

According to SoftBank, the foundational AI model has been trained on a broad range of Japanese traffic knowledge, including manuals and regulations, along with general driving scenarios and unpredictable risk situations.

The VC, which has also invested in UK unicorn Wayve, which develops AI-powered software for self driving vehicles, as well as semiconductor design firm Arm, launched a field trial of the solution at Keio University in Japan last month.

It tested different scenarios such as driving around a parked on-road vehicle onto a crosswalk, where the camera may not have spotted a pedestrian crossing due to the parked vehicle obstructing the view.

In this case, it found the multimodal AI generated the instruction: “There is a vehicle stopped ahead of the crosswalk. Please come to a complete stop, as a pedestrian may suddenly appear.”

Presently, remote operators will still intervene based on information analysed and verbalised by the multimodal AI – aiding the remote operators – but in future, the aim is to achieve fully unmanned operators by allowing the multimodal AI to directly issue instructions to the vehicles.

This AI is also being used in autonomous driving technology trials conducted by Monet Technologies, a mobility-as-a-service provider.

“SoftBank is advancing the development of one of the largest AI computing infrastructures in Japan, alongside a domestic large language model,” said Hirobobu Tamba, vice president and head of data platform strategy division at SoftBank.

“With the successful development of SoftBank’s proprietary traffic understanding multimodal AI and its trials in remote support solutions for autonomous driving, I am confident that the integration of communication technology and AI can offer promising solutions for societal challenges.”

SoftBank unveils new AI to help autonomous vehicles navigate traffic safely