Spell: How to do hand tracking on HoloLens

I am finishing reading Off to Be the Wizard book by Scott Meyer about a guy who dabbles in programming, finds a file that allows him to modify reality and travels to Medieval England to become a wizard. It is a fun and easy read, but the especially interesting part about the book is how it describes creation of magic through code. Reading through the book could not make me stop thinking that they are describing my experiences creating apps for HoloLens. You may not be modifying the real world per say in HoloLens, but you are seeing your hands do magic in the real world. Powerful stuff.

How can you create magic with HoloLens code?

HoloLens sees your hand as long as it is in front of the display (where the cameras are). How can you track your hand with code?

First, make sure you import HoloToolKit Unity into your project. Microsoft has developed a bunch of handy APIs that you can access in your HoloLens app code. To track the location of the hand, we start by registering a couple of events to track whether the hand is visible or not. Snippet below subscribes to the events our code will be watching for.

private void Start()
{
    // Register for hand and finger events to know where your hand
    // is being tracked and what state it is in.
    InteractionManager.SourceDetected += InteractionManager_SourceDetected;
    InteractionManager.SourceLost += InteractionManager_SourceLost;
    InteractionManager.SourceUpdated += InteractionManager_SourceUpdated;
}

 

The source lost and source released events tell us if the hand has become visible to the camera or is out of view. The functions executed at the event occurrence print to the debug console that the event has been seen.

private void InteractionManager_SourceDetected(InteractionSourceState hand)
{
    Debug.Log("Source detected!");
}

private void InteractionManager_SourceLost(InteractionSourceState hand)
{
	Debug.Log("Source lost!");
}

Source updated event is more interesting. This event gets triggered as your hand moves through space, and can be used to attach a cursor or an object to your hand (such as a magic wand or a ball of sparkles. Note that in the source updated event we get the current position of the hand and save it to a variable named position of type Vector3.

private void RemoveTrackedHand(InteractionSourceState hand)
{
	Vector3 position;

	if (state.properties.location.TryGetPosition(out position))
	{
		Debug.Log("Hand position: " + position);
	}
}

You can make a Unity asset such as sparkles here visible any time you detect a hand and set the asset location to hand location. You are now on your way to becoming a true CodeBrave magician!