This Is What Self-Driving Cars Actually See

This is something that I’ve been lucky enough to see on some prototypes, but it’s more than a little valuable for the public to get a sense of what a self-driving car “sees” when driving down the road. In this case, we have one of Waymo’s test cars predicting the path of a cyclist before a crash might even happen.

Waymo sent us two new videos of what information its tech is getting back from its surroundings. In the first video, we see a Waymo self-driving Chrysler Pacifica minivan recognise a group of kids crossing the street:

In the second we see a Waymo-ized Pacifica anticipate a cyclist moving into its lane to get around a trailer parked in the bike lane:

Look at how the car perceives the cyclist’s plotted path, recognising any point of collision looking like a bridge, an obstacle just like a car passing by or the trailer itself.

We’ve written about just how self-driving cars see before, usually through some combination of cameras, radar, lidar, and ultrasonic sensors. But just observing certain objects doesn’t really get a car to steer itself around streets or obstacles or even people. It has to know what those objects are, which is why the car’s ability to identify a cyclist versus a child is so important. Fundamentally, cyclists and small children tend to behave differently, and knowing their general behaviours can help the car predict what’s going to happen.

At the same time, past behaviour isn’t necessarily an indicator of future performance, which is what makes creating self-driving cars so hard. Let’s say a child is walking down the footpath, for instance, all fine and normal. But then it sees an ice cream truck across the way, and darts into the street, as children sometimes do. That sort of behaviour can make things tough, and why getting it right is so important.

The vids themselves are super interesting, maybe not in that the car doesn’t run into a line of children or hit a cyclist (that’s something we’re going to want to take that as a given with a test car operating on the road), but more in that it’s wild to see how precisely the system does it.

This story originally appeared on Jalopnik