Physics matters: Haptic PIVOT, an on-demand controller, simulates physical forces such as momentum and gravity

When you reach out an empty hand to pick an apple from a tree, you’re met with a variety of sensations—the firmness of the apple as you grip it, the resistance from the branch as you tug the apple free, the weight of the apple in your palm once you’ve plucked it, and the smooth, round surface under your fingertips.

In recent years, steady progress in haptic controllers from Microsoft Research has moved us toward a virtual reality (VR) experience in which those feelings will be on par with the awe-inspiring and realistic visual renderings being produced today by head-mounted displays. With previous devices such as NormalTouch, we can simulate a virtual object’s surface inclination and texture on the tip of an individual’s index finger. CLAW enables a person to feel she’s grabbed an object between her fingers to explore its compliance and elasticity, and TORC allows a new level of dexterity, parallel to real life. Using these prototypes, an individual can feel the skin of a virtual apple, squeeze the virtual fruit, and move it around in her hand. However, to facilitate a complete interaction with that apple in its virtual surroundings, we also have to take into account the dynamics of the objects in the space. Now, with Haptic PIVOT, we bring the physics of forces to VR controllers. Worn on the wrist, PIVOT is a portable device with a haptic handle that moves in and out of the hand on demand.

If Sir Isaac Newton were to have found the inspiration for his laws of motion and gravity from a virtual apple falling from a virtual tree, he would have needed a controller like PIVOT. By grounding PIVOT to the wrist, we’re able to render the momentum and drag of thrown and caught objects, which are governed by Newton’s laws, including simulating speeds of objects upon reaching the hand: The robotized haptic handle deploys when needed, approaching and finally reaching the hand, creating the feeling of first contact—going from a bare hand to one holding an object—thus mimicking our natural interaction with physical objects in a way that traditional handheld controllers can’t. We studied the performance and limits of PIVOT and co-authored “Haptic PIVOT: On-Demand Handhelds in VR” (opens in new tab) with fellow Microsoft researchers Mike Sinclair (opens in new tab) and Christian Holz, who is now with ETH Zurich, and Róbert Kovács, Alexa Fay Siu, and Sebastian Marwecki, who were interns at the time of the work. This week, we’re presenting Haptic PIVOT at the 2020 ACM Symposium on User Interface Software and Technology (UIST) (opens in new tab).

From the physical to the virtual—on demand

At the core of PIVOT’s design is its hinge mechanism and haptic handle. The haptic handle is interchangeable and can be swapped out for existing controllers. However, for our work with PIVOT, we outfitted a prototype handle with capacitive touch sensors that detect contact and release of objects; a voice coil actuator for providing vibrotactile feedback; and a trigger switch for control input. The haptic handle operates via a modified servo motor (driving the hinge) and can be summoned into individuals’ hands on demand, keeping their hands free when not in use. This capability makes PIVOT ideal for augmented reality or blended scenarios. An individual can be typing on a keyboard, using a mouse, or working with other physical objects in her environment. Whenever needed, a quick flick of the wrist can initiate PIVOT to rotate the handle into the person’s palm so she can interact with virtual objects. The handle can be retracted with another flick of the wrist. Both summoning actions are detected by an internal accelerometer.

The haptic handle’s motor stops running once the handle has been grabbed, as sensed by the capacitive sensors, and thanks to a passive radioulnar hinge, individuals can move their wrists freely from side to side (up to 60 degrees) and up and down while continuing to hold the handle. To prevent the haptic handle from hitting the thumb as it moves between its resting and activated positions, the motorized hinge is slanted toward the hand, as opposed to perpendicular to it, and a 190-degree range was set to prevent the handle from getting in the way when not in use.

May the forces be with you

The true power of PIVOT shines when interacting with virtual objects. Take picking the apple from the tree as an example. A combination of mechanics, electronics, firmware, and software works together from the moment the apple enters reaching range to the moment it’s resting in the palm of the individual’s hand.

Computer-vision tracking of the hand via a head-mounted display such as Microsoft HoloLens or a consumer VR tracker worn on the back of the hand allows for absolute position tracking so our control system can detect when an individual begins reaching for the target—in this case, the apple. When the apple is within a 30-centimeter radius of collision, PIVOT moves the haptic handle into a preparation position. As the individual’s hand closes in on the apple, within 10 centimeters of it, the handle moves proportionally closer and then finally lands in her palm at the same time she wraps her fingers around the virtual fruit. The handle moves as fast as the individual, providing a very realistic simulation of impact. The four capacitive touch areas along the surface of the handle register that contact with the handle has been made, and a signal is sent via serial communication interface to the virtual hand, which closes around the apple just as the hand grasps the handle. With polling that takes less than 1 millisecond, the interface offers a latency that supports the immediacy needed to delivery haptic response times that align with user expectations.

As the individual pulls the apple from the tree, she encounters the expected resistance from the branch on which it’s attached as PIVOT uses its motor to pull the haptic handle away from the hand. She experiences the resistance until the apple is detached—an action accompanied by a “thud” sensation generated by the voice coil actuator—at which point, she then feels the impact and weight of the apple in her palm. Instead of pulling the handle away, PIVOT presses it into the palm, creating a sense of momentum and weight. PIVOT can render such forces on the palm and fingers because it’s grounded on the wrist and not in the palm. With a simple rotation of the hand and release of the handle, the individual can drop a bad apple to the ground or a good one into a basket with others. Additionally, when worn on both wrists, PIVOT can facilitate two-handed interactions, such as picking up that basket of apples by the handles. The devices render the feeling of holding by synchronizing the haptic feedback in each device.

Play ball!

In making the design decision to ground PIVOT to the wrist, one of the first things we considered was baseball. From pitching to a batter to throwing a runner out at second, there’s a lot happening in the arm, and the same could be said of other sports. Introducing the wrist form factor, or design, into our offerings presented an opportunity to provide a wider range of actions without interfering with the physical environment around the player.

With PIVOT, individuals can catch and throw virtual objects. The reaction time of catching a flying virtual object is significantly shorter than grabbing a stationary virtual object (we can simulate the catch of a 55.9-mph throw through visuo-motor illusions!). Like with simulating the grasp of a nonmoving object, the simulation of catching objects requires that PIVOT and the visual input are aligned correctly to accurately render when object meets hand. For high speeds, a larger collision radius can be implemented to increase the responsiveness of the device.

As is the case with dropping an apple into a basket, throwing relies on PIVOT sensing the motion of the hand and the release of the haptic handle, which coincides with the release of the virtual object. Upon release, the handle is driven out of the palm by the motor at the physically correct angular speed, up to 0.55 milliseconds/degree. In other words, the handle can go from grasp to fully retracted (at approximately 190 degrees) in 340 milliseconds, the time it takes to blink an eye. Throwing, catching, and passing objects among people not only enables simulation for sports games, but can also extend to collaboration in the virtual workplace, where factory workers or industrial designers can feel the forces of virtual designs or products in a completely new way, even before manufacturing them.

The ultimate frontier

Touch is the ultimate frontier in rendering. Once you’ve achieved incredibly realistic visual renderings of objects in virtual and augmented reality, next you want to simulate natural interactions with these virtual objects. That’s when haptics takes center stage.

Today, VR’s visual renderings are immersive, sophisticated, and appealing—so much so that when you put on a VR headset and are transported into a virtual world in which an apple hangs from a tree branch, you can’t help but grab it. But when you reach for that apple and don’t feel its smoothness and firmness, the pullback of the branch when you try to pluck it, or its weight in the palm of your hand, the illusion is shattered. With haptic controllers like PIVOT, Microsoft researchers are working to solve the challenge.

Opens in a new tab