Prototyping with HoloLens Emulator: Class Notes Edition

Have you been waiting for that letter from Hogwarts to invite you to become a wizard? Then you are in the right place! In this blog post I will guide you through detailed steps of becoming a magician and learning how to develop your first prototypes for HoloLens. HoloLens is a holographic computer through which you can see the real world and modify it with the power of code.

You will create a prototype for a HoloLens application that can be used in music concerts and festivals. In this prototype, the music fans will be able to discover musical instruments in the room where bands will be playing and interact with the musical instruments to hear music samples.

 

You will learn how to:

– Setup a Windows Holographic project in Unity

– Use spatial mapping to place a hologram of a guitar in the room

– Map a tap gesture to the guitar hologram to interact with it and open a UI menu

– Map tap gesture to interact with UI menu and play spatial sound

– Deploy the prototype onto HoloLensemulator 

Part 0. Installation Chapter

What do you need to install to start developing prototypes for HoloLens?

• Unity 5.5
• Visual Studio 2015 Update 3
  • Choose a custom install, ensure that Tools (1.4) and Windows 10 SDK (10.0.10586) is enabled under Universal Windows App Development Tools node.
• HoloLens Emulator

For more details on how to install the tools, please consult Microsoft HoloLens page.

Part 1. Hello, Holocube!

Once your tools are installed, we are ready to create a Unity project for the HoloLens application!

1. Launch Unity.
2. Create a new project.
   1. Click on New to create a new Unity project.
   2. Select 3D and name the project IntroToHoloLens.
   3. Click on Create project button.
3. Add HoloToolkit to the project.
   1. Download HoloToolkit-Unity Asset Package (I have unity 5.5.2f1 installed, so I downloaded HoloToolkit-Unity for Unity 5.5.2f1+).
   2. Import HoloToolkit package into your project: Assets->Import Package -> Custom Package.
   3. Select the HoloToolkit asset package you just downloaded and click on Import.
   4. You should now have a HoloToolkit menu item in Unity.
   5. Set up Project settings for your HoloLens project (it may prompt you to restart Unity, do so): HoloToolkit -> Configure -> Apply HoloLens Project Settings.
   6. Set up scene settings for your HoloLens project: HoloToolkit -> Configure -> Apply HoloLens Scene Settings.
4. Setup your first scene.
   1. Create a new Scene: File -> New Scene.
   2. Remove the default Main Camera and Directional Light objects.
   3. Add the HoloLensCamera.prefab (found under HoloToolkit/Input/Prefabs).
   4. Add a Cube object to the scene: Create->3D Object -> Cube.
   5. Set the position of the Cube to (0, 0, 5) and rotation to (45, 45, 45).
5. Build.
   1. Open Build settings: Build -> Build Settings…
   2. Click on Add Open Scenes.
   3. Click on Windows Store.
   4. Click on Switch Platform.
   5. Enable Unity C# projects.
   6. Click on Build.
   7. Create a new folder, name it App and click on Select Folder.
6. Deploy.
   1. Once the build completes, you should see a new explorer window pop up. Go to App folder you created in the previous step, and double click on IntroToHoloLens.sln. We can now deploy the application to HoloLens emulator!
   2. In Visual Studio, switch Debug to Release, ARM to x86 and Local Machine to HoloLens Emulator.
   3. To start the emulator, click on Debug- -> Start Without Debugging
7. Enjoy.
   1. When the emulator launches and your projects gets deployed, you will see your first hologram. Hello, Cube!

Part 2. Your Input matters – even in the emulator!

1. Add the Cursor.prefab (found under HoloToolkit/Input/Prefabs/Cursor).
2. Create an empty object in your scene. Rename it ‘Managers’.
3. Add the InputManager.prefab (found under HoloToolkit/Input/Prefabs) as a child to your new ‘Managers’ Object.
4. Let’s add the functionality to move the Cube using the power of HoloLens Tap gesture!
   1. Click on the Cube in the Heirarchy list, and look in the Inspector panel for a conspicuously named button named ‘Add Component’.
   2. Type ‘GuitarInputManager’ to create a new C# script that will handle interaction with the Cube.
   3. Open the script in Visual Studio and replace it with the following code:

5. using HoloToolkit.Unity.InputModule;
   using UnityEngine;
   
   public class GuitarInputManager : MonoBehaviour, IInputClickHandler
   {
    public virtual void OnInputClicked(InputClickedEventData eventData)
    {
    Debug.Log("Input clicked");
    }
   }
   

6. Since we are deploying to an emulator, and not a real device, we will not be able to see the environment surrounding us. However, we can emulate the room using the emulator’s pre-uploaded room scans and visualize them via mesh drawing.
   1. To enable spatial mapping in your scene, add the SpatialMapping.prefab (found under HoloToolkit/SpatialMapping/Prefabs) to your ‘Managers’ object.
   2. Enable Spatial PerceptionCapabilities: Edit/Project Settings/Player -> Inspector -> Publishing Settings/Capabilities.
7. Finally, let’s build and deploy the application to emulator following steps 6-7 in Part I! In the emulator you will see the cube surrounded by surreal looking triangles, the spatially mapped room. You can click on the cube using a right mouse button to simulate the air tap gesture, and in the Debug window of Visual Studio you should see the “Input Clicked” text printed out.

Part 2. Where is my guitar asset?

At this point you are probably wondering. “Wait, you promised there would be a guitar and music playing and we will have a wizard party with martinis.” And you are right, I did promise some of those things. Let’s implement them!

1. Find the guitar asset in the Unity Asset Store and import it into the project (uncheck the Scenes folder when importing, since we already have a cool scene in our project). Note that to download an asset from the Unity Asset Store into your project, you have to have a Unity account.
2. It’s time to replace the boring cube with a cool guitar.
   1. Delete the cube object, and drag the Guitar.prefab (found under Assets\Prefabs) into the Heirarchy panel.
   2. Set guitar Position to (0, 0, 5) and Scale to (0.5, 0.5, 0.5). Leave the rest to default.
3. With Guitar selected, add GuitarInputManager component to Guitar.
4. Add Mesh Collider component to Guitar.
5. You can test deploy the app once again to make sure that the guitar shows up correctly, is tappable, and that the ‘Input clicked’ message is displayed in the debug window.
6. The Input Clicked message is not the most exciting or realistic feature a guitar can have, so let’s add something cooler and more useful. For this tutorial, when the Guitar object is tapped, information about a band will be shown and giving the user an option to play a music sample for the band. Off we go! Let’s add the band info panel.
   1. In the Hierarchy panel, click on Create->UI->Text. This will create a Canvas object with a Text object as its child as well as the EventSystem object to handle UI events.
   2. Change the Canvas object to map to World Space, and have 10 Dynamic Pixels Per Unit to make sure the text is displayed with higher resolution.
   3. Change Canvas Position, Width, Height and Scale properties to match that of the image below.
   4. Change Text Position, Width, Height, Scale, Text and Font Size properties to match that of the image below.
   5. The last UI component we need to add is a button that will allow us to play the music sample for the band. Right click on the Canvas object, and select UI -> Button to add the button.
   6. Change the button Position, Width, Height, Scale and Color settings to match that of the image below.
   7.  If you expand the button object and click on the Text object, set the button test to a little cute Play triangle character ▶.

 

1. Build and deploy to see a Guitar and band info billboard appear in your virtual room! Clicking the buttons will not do anything right now. We will add that functionality in the next steps.

Part 3. Time to have a wizard guitar party

Now that we have a guitar and a UI menu for playing music samples, it is time to wire up the code to display the menu when the guitar is tapped and play a music sample when the Play button is pressed.

1. Modify the code for Guitar Manager to add Billboard game object, AudioSource object and IsMusicPlaying flag. Add a PlaySample function to play the music when the play button is clicked. The final version of GuitarInputManager should look like this:

2. using HoloToolkit.Unity.InputModule;
   using UnityEngine;
   
   public class GuitarInputManager : MonoBehaviour, IInputClickHandler
   {
    public GameObject billboard;
    public AudioSource BandSample;
   
    private bool IsMusicPlaying;
   
    public virtual void OnInputClicked(InputClickedEventData eventData)
    {
    Debug.Log("Input clicked");
    billboard.SetActive(true);
    }
   
    public void PlaySample()
    {
    IsMusicPlaying = !IsMusicPlaying;
    if (IsMusicPlaying)
    {
    BandSample.Play();
    }
    else
    {
    BandSample.Pause();
    }
    }
   }
   

3. Add a AudioSource component to Guitar and set AudioClip to Across the Universe midi music file, which can be downloaded here. Make sure to uncheck “Play on Awake”.
4. Drag the Audio Source created in step 3 onto the GuitarInputManager’s BandSample object. Drag the Canvas object created in Part 2 onto the Billboard. 
5. Disable the Canvas object in the Inspector.
6. Add spatial sound capability so that when we turn away from the guitar, the sound comes from the correct location, and being far away from the guitar makes the music sound quieter, just like in the real world.
   1. Set the MS HRTF Spatializer in your audio settings:
      Edit -> Project Settings -> Audio -> Spatializer.
   2. In Guitar Inspector settings, check the Spatialize to enable spatial sound and drag SpatialBlend slider all the way to the right for 3D sound.
7. Finally, select the Button object and look at the inspector Inspector panel. Let’s map the OnClick event at the bottom of the Inspector panel to the PlaySample function we created in Part 2. Select the Guitar object, and in the function drop down choose GuitarInputManager->PlaySample.
8. We are all set for the final build, deploy and enjoy! When you launch the application, the menu will be hidden. Try tapping on the guitar, and the menu will appear. Tapping on the play button will start the iconic Across the Universe trek by The Beatles.

Congratulations! You have taken the first step to becoming a HoloLens wizard and filling this world with holograms.