Kinect for Windows SDK Programming Guide电子书

Kinect for Windows SDK Programming Guide

Table of Contents

Kinect for Windows SDK Programming Guide

Credits

About the Author

Acknowledgement

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers and more

Why Subscribe?

Free Access for Packt account holders

Instant Updates on New Packt Books

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Understanding the Kinect Device

Components of Kinect for Windows

Inside the Kinect sensor

The color camera

IR emitter and IR depth sensor

How depth data processing works

Tilt motor

Microphone array

LED

Kinect for Windows versus Kinect for Xbox

Where can you use Kinect

Summary

2. Getting Started

System requirements for the Kinect for Windows SDK

Supported operating systems

System configuration

The Kinect sensor

The Kinect for Windows sensor

The Kinect for Xbox sensor

Development tools and software

Evaluation of the Kinect for Windows SDK

Downloading the SDK and the Developer Toolkit

Installing Kinect for Windows SDK

Installing the Developer Toolkit

Components installed by the SDK and the Developer Toolkit

Kinect management service

Connecting the sensor with the system

Verifying the installed drivers

Not able to view all the device components

Detecting the loaded drivers in Device Manager

Testing your device

Testing Kinect sensors

Testing the Kinect microphone array

Looking inside the Kinect SDK

Features of the Kinect for Windows SDK

Capturing the color image data stream

Processing the depth image data stream

Near Mode

Capturing the infrared stream

Tracking human skeleton and joint movements

Capturing the audio stream

Speech recognition

Human gesture recognition

Tilting the Kinect sensor

Getting data from the accelerometer of the sensor

Controlling the infrared emitter

The Kinect for Windows Developer Toolkit

The Face Tracking SDK

Kinect Studio

Making your development setup ready

The Coding4Fun Kinect Toolkit

Summary

3. Starting to Build Kinect Applications

How applications interact with the Kinect sensor

Understanding the classification of SDK APIs

Kinect Info Box – your first Kinect application

Creating a new Visual Studio project

Adding the Kinect libraries

Getting the Kinect sensor

The Kinect sensor

Defining the Kinect sensor

The collection of sensors

Starting up Kinect

Inside the sensor.Start() method

Enabling the data streams

Identifying the Kinect sensor

Initializing the sensor using device connection ID

Stopping the Kinect sensor

The Stop() method does the clean-up operation

Displaying information in the Kinect Info Box

Designing the Info Box UI

Binding the data

A quick look at INotifyPropertyChanged

Using INotifyPropertyChanged for data binding

Setting the DataContext

Setting up the information

That's all!

Dealing with the Kinect status

Monitoring the change in sensor status

Properties of the StatusChangedEventArgs class

Resuming your application automatically

Building KinectStatusNotifier

Setting up an application

How it works

Using KinectStatusNotifier

Test it out

Summary

4. Getting the Most out of Kinect Camera

Understanding the Kinect image stream

Types of color images

Different ways of retrieving the color stream from Kinect

Event model

Polling model

KinectCam – a Kinect camera application

Setting up the project

Designing the application – XAML and data binding

Capturing color image from the Kinect camera

Enabling the color stream channel

Enabling a channel with the image format

Choosing the image format

Disabling the color stream channel

Attaching the event handler

Processing the incoming image frames

Rendering image frames on the UI

Running the KinectCam

Looking inside color image stream helpers

The ColorImageStream class

The ColorImageFrame class

Capturing frames on demand

Extending the KinectCam

Getting the frame number

Changing image format on the fly

Bind available image formats

Changing the color image format

Calculating frame rate

How to calculate frame rate

Capturing and saving images

Saving images periodically

Trying to save image frames directly

Changing the sensor elevation angle

Maximum and minimum elevation angles

Adjusting the Kinect sensor angle

Playing around the color pixels

Applying RGB effects

Making grayscale effects

Inverting the color

Applying more effects to the camera

Applying the backlight compensation mode

Applying slow motion effects

Kinect Camera Effects – application

Seeing in low light

Making your application perform better

Using the Coding4Fun toolkit

Installing the Coding4Fun Kinect toolkit

Using assembly

Using the NuGet package

Using Coding4Fun Kinect libraries in your application

Summary

5. The Depth Data – Making Things Happen

Understanding the depth data stream

Depth data – behind the scenes

Stereo triangulation

Capturing and processing depth data

Enabling the depth stream channel

Attaching the event handler

Processing the depth frames

Depth data at first look

Looking inside depth image stream helpers

Depth data and distance

How the distance is calculated

Getting the distance from a particular pixel

Accessing the range of distance

Colorize depth data processing

Working with depth range

Special depth range values

Depth data distribution

Player index with depth data

How player index works

Identifying players

Getting the depth and player index automatically

A 3D view of depth data

The basics of the coordinate system

Basic elements of 3D graphics

Setting up the project

Give it a 3D effect

Creating the ViewPort

Using the camera

Controlling the camera position

Creating the 3D Model

Building the mesh object

Setting up the initial data points

Getting the depth data from Kinect

Have a look at 3D depth

Summary

6. Human Skeleton Tracking

How skeleton tracking works

Steps to remember

Skeleton tracking with the Kinect SDK

Start tracking skeleton joints

Tracking the right hand

Setting up the project

Creating a joint placeholder

Get Kinect running and instantiate skeleton tracking

Enabling and disabling the skeleton stream

Processing the skeleton frames

Mapping the skeleton joints with UI elements

Running the application

Adding more fun

Flow – capturing skeleton data

An intrusion detector camera application

Adding night vision

Looking inside skeleton stream helpers

The skeleton frame

The skeleton stream

Skeleton-tracking mode

Default skeleton tracking

Seated skeleton tracking

Using seated-skeleton tracking

Points to be considered with seated-skeleton tracking

Skeleton tracking in near mode

The Skeleton

Skeleton-tracking state

Counting the number of tracked skeletons

Choosing which skeleton to track

Skeleton-tracking ID

Monitoring changes in the skeleton

Limiting tracking for the intrusion-detector camera

The building blocks – Joints and JointCollection

Joint-tracking state

Steps to be followed for joint tracking

Create your own joints data point

Bones – connecting joints

Bone sequence

Bone sequence for a default skeleton

Bone sequence for a seated skeleton

Drawing bones between joints

Adjusting the Kinect sensor automatically and giving live feedback to users

Skeleton smoothing – soften the skeleton's movement

What causes skeleton jitters

Making skeleton movement softer

Smoothing parameters

How to check if skeleton smoothing is enabled

Exponential smoothing

Skeleton space transformation

The Advanced Skeleton Viewer application

Debugging the applications

Using conditional breakpoints

Using Kinect Studio

Getting data frames together

Summary

7. Using Kinect's Microphone Array

Verifying the Kinect audio configuration

Troubleshooting: Kinect USB Audio not recognizing

Using the Kinect microphone array with your computer

The Kinect SDK architecture for Audio

Kinect microphone array

The major focus area of Kinect audio

Why microphone array

Audio signal processing in Kinect

Taking control over the microphone array

Kinect audio stream

Starting and stopping the Kinect audio stream

Starting audio streaming after a time interval

Kinect sound recorder – capturing Kinect audio data

Setting up the project

Designing the application – XAML and data binding

Recording the Kinect audio

Starting the recording

Playing the recorded audio

Running the Kinect Sound Recorder

Processing the audio data

Echo cancellation

Noise suppression

Automatic gain control

Audio data processing with the Kinect sound recorder

Sound source localization

Sound source angle

Confidence level

Beamforming

Beam angle mode

Extending the Kinect Sound Recorder with sound source localization

Summary

8. Speech Recognition

How speech recognition works

Using Kinect with your Windows PC speech recognition

Beginning with Microsoft Speech API (SAPI)

Steps for building speech-enabled applications

Basic speech-recognition approach

Building grammar

Using Choice and GrammarBuilder

Appending new grammars

Building grammar using XML

Creating grammar from GrammarBuilder

Loading grammar into a recognizer

Unloading grammars

Draw What I Want – a speech-enabled application

Setting up the project

Designing the application – XAML and data binding

Data binding

Instantiating speech recognizer

Working with the speech recognition engine

Configuring Kinect audio

Creating grammar

Start the speech recognizer

Drawing an object when speech is recognized

Testing your application

Summary

9. Building Gesture-controlled Applications

What is a gesture

Approaches for gesture recognition

Basic gesture recognition

Gesture-detection technique

Representing skeleton joints

Calculating the distance between two joints

Building a clapping-hands application

Setting up the project

Implementing the gesture recognizer

Defining the types of gestures

Defining the types of recognition results

Creating the event argument for the gesture

Wrapping up everything with the gesture recognition engine

Plugging gestures into the application

Testing your application

A virtual rope workout application

Hands-raised-above-head gesture recognition

Steps to recognize basic gestures

Algorithmic gesture recognition

Which gestures can be considered as algorithmic

Understanding the algorithmic gesture detection approach

Implementing an algorithmic gesture

Adding gesture types

Extending the Event argument

Adding a GestureHelper class

Defining the GestureBase class

Implementing the SwipeToLeftGesture class

Adding the ZoomIn, ZoomOut, and SwipeToRight gesture classes

Implementing the GestureRecognitionEngine class

Using the GestureRecognitionEngine class

A demo application

Making it more flexible

Weighted network gesture recognition

What is a neural network

Gesture recognition with neural networks

Jump tracking with a neural network – an example

Template-based gesture recognition

Building gesture-enabled controls

Making a hand cursor

Getting the hand-cursor point

Identifying the objects

Enabling action for the objects

The Basic Interaction – a WPF application

Key things to remember

Summary

10. Developing Applications Using Multiple Kinects

Setting up the environment for multiple Kinects

Plugging the first Kinect sensor

Plugging the second Kinect sensor

Kinect sensors require an individual USB Controller

Multiple Kinects – how to reduce interference

Detecting multiple Kinects

Getting access to the individual sensor

Different ways to get a Kinect sensor's reference

Developing an application with multiple Kinects

Setting up the project

Designing the UI

Creating the KinectInfoCollection

Getting information from Kinects

Running the application

Controlling multiple sensor status changes

Extending Multiple Kinect Viewer with status change

Registering and handling the status change

Running the application

Identifying the devices automatically

Integrating with KinectStatusNotifier

Capturing data using multiple Kinects

Handling a failover scenario using Kinects

Challenges faced in developing applications using multiple Kinects

Applications where multiple Kinects can be used

Summary

11. Putting Things Together

Taking Kinect to the Cloud

Required components

Windows Azure

The Windows Azure SDK

The Kinect for Windows SDK

Designing the solution

Real-time implementations

Remotely using the Kinect with Windows Phone

Required components

The Windows Azure Service Bus

The Windows Phone SDK

Designing the solution

Real-time implementations

Using Kinect with a Netduino microcontroller

Required components

Microsoft .Net Micro Framework

Netduino

The Netduino SDK

Blinking of the on-board LED

Changing the Deployment Transport

Running the application

Connecting Kinect to a Netduino

Using an Internet connection

Listening to the request

Sending a request from a Kinect application

Taking it further

Augmented reality applications

Working with face tracking

Working with XNA and a 3D avatar

Summary

Index