Ford patents lip reading & emotion detection inside the car

This article discusses a new Ford patent application concerning in-vehicle lip reading and facial expression detection, placing it within the context of previous Ford patents and legal cases related to privacy.

Background: Prior Ford Privacy Concerns

Before delving into the latest patent, it's crucial to understand Ford's history regarding in-car privacy and data collection.

Storing Text Messages and User Data

A case from 2023 (as referenced three years prior to the video's recording) highlighted a ruling that it is legal for automakers to download and store text messages under a Washington privacy law 0:16.
When a user pairs their phone to their car and grants permission for the car to read text messages aloud for convenience, the car stores these messages. The significant issue arises because there is no readily available way for users to delete this stored data from the vehicle 0:58. This means if a car is sold, personal text messages and phone data could remain accessible to someone with the right software 1:15.

The judge in this case ruled that the privacy law in question only applies if the individual has suffered "damages or reputational harm" 1:24. This implies that if personal information, such as listening to Rick Astley by choice 1:34, is extracted from the car but not used to cause harm, no privacy violation has technically occurred 1:58. More details on this case are available on the consumer rights wiki.

In-Car Advertising Patent

Another notable Ford patent involves an "in-vehicle advertisement presentation system" 2:19. This system is designed to monitor conversations among occupants to determine when and how many advertisements to serve, and whether to deliver them as audio, video, or an audiovisual format based on driving conditions 2:27. This raises concerns about privacy, especially given that cars have become more expensive and, in some ways, less reliable 2:37, now potentially adding constant advertising to the user experience.

Ford's New Patent: Lip Reading and Emotion Detection

The latest Ford patent application describes an "in-vehicle lip reading and facial expression detection system" 2:56. The full text of this 16-page patent is available via the consumer rights.wiki article.

Key aspects of this patent include:

• Acoustic Signal Detection: The vehicle may use acoustic signals to detect facial movements by emitting "nearly inaudible sound waves" and analyzing the echoes from the user's lips and mouth 3:15. The patent does not describe any consent process for this "active sonar" on a user's face 3:28.
• Lip Reading and Speech Synthesis: Cameras can read lips and synthesize speech even if the car's microphone is muted or there is significant ambient noise 3:32. The system can generate and synthesize speech based on lip movements and/or gesture detection, converting it into text or audio data for transmission to another party in a conversation 3:50. This means a user's "voice" can be transmitted as data without ever being acoustically recorded 4:05.
• Facial Expression and Gesture Detection: The car is designed to monitor a user's face to determine emotional states like confusion or frustration 4:12. It can detect gestures such as head nodding, head shaking, or eye-rolling in response to verbal communications 4:20. These facial gestures and expressions can serve as "trigger condition[s] to calculate the actual ambient noise levels in the vehicle" 4:33, indicating that emotion inference is a core part of the system 4:39. This is compared to Spotify's patent for inferring emotional states to serve advertisements 4:44.
• Cloud Connectivity and Data Sharing: The control server for this system is cloud-based and explicitly linked to various entities, including a "telematic service delivery network," "tow assistance firm," "vehicle maintenance and repair firm," an "insurance firm," and a "transportation firm" 5:03. This direct connection to insurance firms is particularly concerning given existing practices where car manufacturers sell driving data to brokers, who then sell it to insurance companies to raise rates 5:34. For example, hard braking to avoid a drunk driver could be recorded and used to increase insurance premiums, without accounting for the context 5:42. This practice is not a conspiracy theory and has been widely reported, including by the New York Times 6:05.
• Persistent Biometric Database: The patent describes storing "association information between each of a plurality of gestures and a corresponding verbal command associated with the vehicle" in a "per vehicle database" 6:18. This means a persistent biometric database of a user's gestures and facial expressions, tied to commands, would reside in the car 6:38.
• Lack of User Control: Crucially, the patent does not describe deletion mechanisms, data retention limits, or owner access to this biometric data 6:43. Previous court rulings suggest that without demonstrated damages, the retention of such data does not constitute a privacy violation 6:48, allowing automakers and law enforcement agencies potential future access 7:01.

Broader Implications and Concerns

While features like voice commands are useful in modern cars, especially with the decline of physical knobs 7:11, the broader context of automakers actively fighting against user data deletion, patenting in-car advertising based on conversations, and selling data to insurance companies raises significant privacy concerns 7:31.

It is important to recognize that these privacy issues are not exclusive to new electric vehicle manufacturers like Tesla. This trend of spying, data collection, and advertising patents is becoming commonplace across all automotive manufacturers, affecting both new and traditional gas-powered vehicles 8:43. Focusing solely on one brand as a "universal piñata" 8:29 can distract from the industry-wide nature of these practices.

The argument that "just because they patent it doesn't mean they're going to use it" is dismissed as naive when considering the comprehensive nature of such patents and the ongoing history of data exploitation by car manufacturers 9:31.