Microsoft HoloLens Developer’s Guide电子书

Title Page

Copyright

Microsoft HoloLens Developer's Guide

Credits

About the Author

Acknowledgments

About the Reviewer

www.PacktPub.com

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Lets Begin!

Welcome to the HoloLens!

The device

A wearable computer

The sensors

Cameras

Microphones

Holographic lenses

Spatial speakers

Processors

The device in a nutshell

Displaying 3D images

Wearing the device

Turning the device on for the first time

Learning the gestures

Calibrating the device

The device portal

Overview of the portal

The different menus in the portal

The Views menu in the portal

Writing your first holographic application

The tools we need to write software

Your very first application

Deploying your first app to the emulator

How to use the emulator

Deploying the app to the device

Deploying through Wi-Fi

Deploying through USB

Let's have a look at the code

The structure of the project

Inside the AppView class

Using Unity3D to create great holographic apps

Our first HoloApp in Unity3D

Fixing the camera

Changing the position

Changing the background color

Adding holographic objects

Create a sample script

Building the project

Continuing in Visual Studio after Unity

The structure of a Unity HoloLens UWP app solution

Build configurations with Unity

Building and running your HoloLens Unity app

Summary

Designing Your App

Unique features of the device

Thinking in three dimensions

Making sure objects are in a logical place

Making sure objects can be reached

Making sure objects can be found

Pointers

Sounds

Dealing with limited input methods

Gestures

Audio

People do not expect sudden changes

The social aspect in collaborative situations

Give your user superpowers!

Forming a design team

Big projects need big teams

Small projects need small teams

Using your experience

Where holographic apps are "just another app"

Where do holographic apps differ?

The workflow to identify your features

Writing down your mission statement

Brainstorming all ideas

Removing ideas that don't support the mission statement

Separating must-haves from nice-to-haves

Prioritizing must-haves

Implementing your app

Introducing the RockOn app

What do we want to achieve?

How do we get there?

Decision time - tooling

Developing with DirectX

Developing with Unity3D

Creating a new project

Getting started with Unity3D

The role of Visual Studio in developing for the HoloLens

Adding assets to the scene

Building the project

Deploying the project

Using the emulator

Using the device

Making some small adjustments

Summary

Gestures and Events

Working with gestures

Some UX considerations

A brief history of user interfaces

Command-line interface

Graphical user interface globally

Natural user interface

Feedback

Types of feedback in the HoloLens world

Getting to grips with gaze

Gaze in DirectX

The bare HoloCursor class

A word about shaders

The processing pipeline

Programming shaders

Adding shaders to our project

Additional data

Defining images in three dimensions.

Our square cursor

Rendering and updating

The Render() method

Updating after changes

Using our new class

Positioning

Rotating

Next steps

Gaze in Unity

The cursor

Creating the assets

Editing the script in Visual Studio

Using the script in Unity

Implementing the cursor code

Testing for touch

Events

Gestures in DirectX

The InteractionManager class

The SpatialGestureRecognizer

Gestures in Unity

Creating a prefab

Creating the creation code

Summary

Make Some Noise - Sounds

How our bodies work when it comes to sound

Head Related Transfer Function

Left and right

Front and back

Up and down

Head Related Transfer Functions

Moving our heads

Echo

Spatial sound in computer programs

Sound design

The design principles for spatial sound

The benefits of spatial sound

Grounding

User attention

Immersion

Interaction design

Best practices in sound design

Applying these principles

Licensing

Background music

DirectX

Getting the right NuGet packages

Loading and preparing the sound we want to play

Playing the sound

Actually play the sounds

Unity

Spatial sound

DirectX

The SpatialSound library

Using the library

Unity

Setting the audio settings in Unity3D

Adding spatial sound to GameObjects

Play the sound at the right time.

Some final thoughts on spatial sound

General concepts when it comes to sounds

What is next?

Summary

Be Heard - Voice Interactions

Let the HoloLens speak

General principles behind voice generation

Voice generation in DirectX

Creating the base of the VoicePlayer class

Initializing the sound system for streaming

Implementing IDispose the right way

Playing the sound

Synthesizing voice

Setting things up

Calling Play to hear the voice

Making the voice spatial

Voice generation in Unity3D

Creating the Unity app

Adding the helper script

Creating the scripts for the asset

Our SpatialVoicePlayer script

What is it about these #if statements?

Asynchronous stream generating

Creating the reusable asset

Adding voice to RockOn

Making things more lifelike

Introducing the SSML stream

Overview of the most important SSML tags

Listen up - voice commands

Default commands

Select

Hey, Cortana

App commands

Hologram manipulations

Dictation

Troubleshooting speech recognition

Building a VoiceRecognizer in DirectX

The VoiceRecognizer class

Building the VoiceRecognizer in Unity3D

What is next?

Summary

Take a Look around You - Spatial Mapping

VR, AR, and MR

Virtual reality

Augmented reality

Mixed reality

Usages of spatial mapping

Occlusion

Navigation

Placement

Visualization

Physics

Masquerading imperfection

Things to be aware of in spatial mapping

Errors in the spatial mapping

Holes

Fake surfaces

Misaligned surfaces

Causes for errors

The user's movements

The physical qualities of the surfaces

Lighting

Moving objects

App-specific requirements

Optimizing meshes

Spatial understanding in DirectX

Initial app

Spatial capabilities

Initial classes

Checking capabilities

Setting up device-dependent resources

Creating the SpatialSurfaceObserver

Getting the surface data

Getting the actual mesh data

Pruning the list

The SurfaceMesh class

Updating and rendering

Mesh updating and rendering

Cleaning up

Putting it all to work

Spatial understanding in Unity3D

Writing scripts for spatial mapping in Unity3D

Adding surface mapping

Capabilities

Spatial mapping in code

Handling changes in the surfaces

New and updated surfaces

Wireframe

The easy way

Occlusion

Summary

Lets Talk! Communications between Devices

The position of holograms

Local coordinate systems

Spatial mapping and coordinate systems

Spatial anchors

Using the spatial anchors in DirectX

Using SpatialAnchors in Unity3D

Changing GestureHandler.cs

Problems with anchors

Stationary frame of reference

Using the stationary frame of reference in DirectX

Using the stationary frame of reference in Unity3D

Spatial locator-attached frame of reference

Using the locator attached frame in DirectX

What this does?

Using the locator-attached frame in Unity3D

Sharing

Setting up networking

Sharing scenarios

Using a hub

Setting up the data

Setting up the TCP/IP layer

Creating the thread for handling data traffic

Handling connections

Sending data

Creating the mixed-reality client app

Setting up the connection

Consuming the connection

Connecting from Unity3D

Peer to peer

Introducing Bluetooth advertisements

Setting up the advertisement

Watching for other devices

Handling found devices

Connecting to the other HoloLens

Sharing world coordinates

Sharing spatial anchors in DirectX

Should you send one or multiple anchors?

Sharing world anchors in Unity3D

Sending the data from Unity3D

Receiving the data in Unity3D

Summary

Speed Up Your Development - Advanced Unity3D

Framework versions

Mono runtime

.NET in HoloLens

Script compiling in Unity3D

Compilation errors and how to fix them

Three possible solutions

Option 1 - do not go back and forth

Option 2 - sprinkle the code with #if/#endif statements

Option 3 - placeholder assemblies

Why does this work?

Performance in Unity3D

Remoting in Unity3D

Profiling in Unity3D

Unity3D performance tricks

QualitySettings

Polygon/vertex reduction

Static objects

Level of Detail

Lighting optimizations

Texture optimizations

Scripts optimizations

Threading

Garbage collections

Structs versus classes

Shader optimizations

Introducing the HoloToolkit

Where to get it

Importing the toolkit

Using the toolkit

Menu items

Assets

Shaders

Sharing service

And much more

Summary