BMW gesture control explained

In many loungerooms and rumpus rooms around the nation kids are bowling, batting, kicking and punching objects that don't actually exist, other than in a cyberspace rendition of three-dimensional space between themselves and a device such as Xbox Kinect.

BMW has taken this technology, which allows game players to interact with software in games consoles, and rejigged it for drivers and passengers to control the infotainment systems in the new 7 Series. With BMW's system, which the company calls 'Gesture Control', the user has a choice of five hand gestures that resemble the movements of an orchestra conductor.

A camera located in the headlining of the car monitors the space between driver and front passenger, around the centre fascia section of the dashboard. Stephan Peters, BMW's Project Manager Human Machine Interface for the 7 Series, refers to this virtual monitored space as 'the strike zone'. If either the driver or passenger use one of the five gestures within the strike zone, in the correct context, output from the camera is interpreted by a processing unit and the desired change is made. That change could be reducing the volume of the audio system or rejecting a phone call, as a couple of examples.

The five hand gestures are: a pointing gesture, anti-clockwise rotation of the index finger, clockwise rotation of the index finger, a two-fingered pointing gesture and a swiping gesture.

Used to select a menu item (frequently presented in the car's touch-screen display as an icon), the pointing gesture can also be used to swivel a computer-rendered image of the car around its base in the surround view depiction in the touch screen. This provides the driver with an opportunity to check for any otherwise unseen obstacles before moving the car forwards or backwards from its parked position.

The two rotational gestures can be used to lower or raise audio system volume. By moving the index finger anti-clockwise, the audio volume is reduced, and by moving the finger clockwise, the volume is increased. This also applies to satnav voice prompts and phone calls.

The two-fingered gesture (index and middle fingers spread apart and thrust forward) actuates a selection from the car's favourites menu, which is based primarily on the touch screen readout showing at the time. If it's the audio system data on screen, the favourites gesture may switch the radio to a previously selected FM station or a much-loved song. If it's the satellite navigation function on screen, the gesture may activate the car's guidance system for the pre-programmed home destination.

The swiping gesture is used to reject an incoming phone call (the pointing gesture is used to accept the phone call) or cancel some other function. In the embedded video, the swiping gesture is used to cancel route guidance.

While gesture control is sure to attract criticism in some quarters, it is different from the infamous iDrive in earlier iterations, in that gesture control is not the exclusive means of controlling the car's on-board infotainment systems. As Stephan Peters explained to motoring.com.au during the international launch of the 7 Series last week, owners are not required to use gesture control if they find it too complex.

In fact, BMW has learned well from its previous mistakes with iDrive. Every BMW now comes with redundant control systems to provide the necessary flexibility for all users. Gesture control is a redundant system, and in the case of entering a destination in the satnav system, gesture control is actually one of three redundant systems.

Users can set a car's destination by voice control, the conventional iDrive single-point controller or handwriting before they have to resort to gesture control. For the handwriting facility, users need merely trail a finger over the top of the iDrive controller. The resulting letter will be reproduced in the touch screen.

Peters was asked whether gesture control would supersede existing redundant systems, like handwriting, for instance, since handwriting is notoriously impractical from the driver's seat of a right-hand drive car – most drivers being right-handed. Handwriting works better in left-hand drive markets and in countries where use of the Roman alphabet is not widespread. Particularly in those Asian countries where one pictograph represents an entire word, handwriting in cars has proved to be very popular.

"[Redundancy] is not going to be really reduced, because what we figured out... we have so many different customers – we have [in] Asia and China – they love the touch [screen function]," BMW's ergonomics expert replied.

"Looking all over the world, then we have countries in Europe where the average age of a 7 Series [buyer] is by far above 50... even above 55. They love the classical controller.

"We have the requirement to stay with the controller, [and] to have new features like the touch interface and the voice...

"Therefore, I don't see that the customers would accept any step back to reduce [any] of these functions; [it's] more that we offer all of these functions, and the driver can select the function that the person likes the most."

Peters admits that BMW is open to dispensing with control systems that might outlive their usefulness in future.

"It's really about the need of the customer; if the customer at a certain point wouldn't need [something] anymore, and he's going to tell us, then we will react on this. It's not that we would come up with the idea; we have now a perfect balance of using hardware buttons and using touch interface controls to reduce [redundancy] if there is no need [for a specific function]."

Gesture control provides a manufacturer like BMW – selling cars in left and right-hand drive markets – an opportunity to reduce the complexity of its switchgear for human/machine interface. The company has to switch the location of the iDrive single-point controller from the right side of the centre console in left-hand drive markets to the other side of the transmission shifter in right-hand drive markets. If gesture control can eventually replace the single-point controller altogether, that's one more way of lowering the engineering cost up front. Peters says that gesture control works equally well for the front-seat passenger as the driver – irrespective of the location of the steering wheel in the car. Even the swiping gesture to reject an incoming phone call works with the front passenger using the opposite hand and swiping in the opposite direction.

"Due to the fact that the camera's in the middle, it's working for both – passenger and driver's side, and switched around with the right-hand drive car it's working for driver and passenger side."

Was right-hand drive a catalyst for the initiative to develop gesture control?

"Gesture Control was not driven by ergonomic effects for right or left-hand drive," Peters responded.

"While developing the new iDrive system, we really went into customer clinics... we asked these 7 Series customers – and even S-Class and A8 customers – we asked them: 'What do you expect for the new iDrive?'

"And they said, obviously, they wanted to have 'touch'. Then we asked them further: 'Is there anything else in mind, which we could use for the iDrive?' And some of them came up with the idea: 'We don't know how to use it, but we have game consoles or television sets at home that use gesture... this is cool, I like it'.

"And they didn't come to us to say: 'You need to do this with gesture because we think it's kind of useful'; they just said: 'If there is anything else [other] than 'touch' or 'controller', it could be 'gesture'.

"So when we started at this point thinking about this... we started a whole process to work together with our concept engineers... regarding the technical solutions and implementation – what we could do with gesture. And we started to investigate even the gesture, which gestures can we use all way around the world and which [ones] not. There are a couple of gestures which we will not use at all, because they have some bad meanings.

"I heard about one gesture in New Jersey and New York that is kind of bad...

"By that time really, we started thinking about this process of gesture, and it was not a matter of right-hand or left-hand drive; it was more or less to find something useful for a car and – because when we talk to each other we use hands for gesture – in terms of a car, having a shortcut done by a gesture is a nice method to do something quick.

"Where we have other things, such as destination input, which takes longer, we use all the other methods we have in a car."

The so-called 'strike zone' – that three-dimensional area of space located aft of the centre fascia and above the transmission selector – is large, but unoccupied by physical objects. It offers the potential for massive de-cluttering of the HMI elements of physical switchgear and large touch screens in the centre fascia and centre console.

"But this is talking about future, and what we have right now is the first [generation] of gestures – and we bring them out worldwide," Peters said.

"This is the first step for gesture; let's see what the future's going to bring for us regarding gesture. But for right now we have the set of five gestures we are using, and we truly believe it will be accepted by the customers, because some of these gestures are so amazing – like rejecting a phone call... it's done."

Gesture control ultimately means migration away from touch screens – with their aggregation of grimy finger prints on the screen. Peters doesn't agree that touch screens are unpopular with users due to those aesthetic reasons.

"If there are finger prints on [the screen], you are going to clean it... and as I said, you have the ability to only use the controller like you used to do if you're the type of guy who says: 'I don't like to see this, I don't want to touch it'. You can go ahead and use the controller, and all these people who love to touch it, they're completely fine with it."

Down the track, gesture control could be employed to run vehicle operations, one might think. Peters doesn't believe that will happen.

"The gesture [control] in use in automotive is kind of a shortcut; and what we do not see is using gesture as a language, because that is not intuitive. When we talk to each other, we use language – and we use the gesture to underline – so you need to learn gesture language, which is quite hard.

"Therefore it's not practical for the automotive industry."

What the BMW spokesman left unsaid is that iDrive has never progressed beyond control of infotainment systems, and with all the world's car companies working furiously on autonomous motoring, it's possible the self-driving car will be with us long before companies like BMW ever get around to establishing a comprehensive 'sign language' set that could control all the car's functions.

The only obvious shortcoming in gesture control is the human factor in the human machine interface. Owners of gesture control-equipped BMWs should probably avoid driving anywhere with their inebriated, fun-loving mates in the front passenger seat...