Overview Part 1: Using the Kinect 2 for Oculus Rift position tracking

This is the first part of my two-part explanation on what will be done in this project (you can find the second part here). This part tries to give more information on how the Kinect 2 can be used in combination with the Oculus Rift.

The Oculus Rift uses different sensors to estimate how the head moves by default. It has a built-in gyroscope, accelerometer and magnetometer with a refresh rate of 1000 Hz. Externally it can use a special Near-IR-camera for position tracking at a speed of 60 Hz. This external camera only has a very limited range and is meant to be used in a rather stationary position (in front of your desk, for instance) and normally facing one general direction (of course people are still able to turn around, if they want to).

The internal display uses 75 Hz (or fps – frames per second) as refresh rate. This is important, as one of the early problems in VR was a lack of fluid motion, when moving your head. The current generation of the Oculus Rift (DK2 – development kit 2) running at 75 Hz (and using a predictive algorithms) makes sure, that head-turning motions are smooth and that it reacts instantly.

So while it is possible to look around with the Oculus Rift, it is actually not possible to move around by other means than using traditional inputs (keyboard/mouse/gamepad).

The current Kinect version 2 by Microsoft on the other hand is a device that can not only deliver a FullHD (1920 x 1080 pixels) color image feed (like a video camera or webcam), but also streams with depth and infrared informations (both at 512 x 424 pixels). All three of these image streams run at 30 Hz (= 30 fps).

It is easy to recognize – from the previous descriptions – the importance of fluid motion and that 30 Hz will not be enough for precise head tracking. The good news is, that this is not necessary. As the Oculus Rift already has built-in motion-tracking sensors, we only need to supplement them on a regular basis with absolute position updates. This way, if someone moves one meter to the side, the change will become visible immediately from the built-in sensors and the absolute position of the person will be determined by the external Kinect camera.

As the Kinect was build with people playing in front of a TV in mind – and not people who turn around 360° – it is a rather good device for skeleton tracking, but not for a person that moves around in any direction. That is why it will become necessary to use multiple Kinect cameras to cover a wider angle and range than one camera could.

Using multiple Kinect 2 cameras is currently a bit of a difficult task. The main reason being, that the current version of the SDK does not allow for more than one Kinect per computer. The only way to use more than one, is to hook them up to different computers and send the data via a network connection.

The Kinect 2 needs a USB 3 connection and is said to send around 2 or 2.3 Gbit/s plus overhead. While a good PCI express card that uses enough lanes would have enough bandwidth to communicate with multiple Kinect, it is not possible with the current SDK (a Gen2 PCIx lane can transfer around 4 Gbit/s, so a USB 3 PCIx card that uses 4 lanes (x4) could easily satisfy the needs for four Kinect 2 devices).

In order to be able to use multiple Kinect 2, I will use four smaller computers that are hooked up to one main computer. The smaller computers will still have to be somewhat powerful, as they have to process all of the Kinect 2 data in a short time. I also want to use them for pre-processing the data, so that the main computer doesn’t have to do too much work.

They will be connected to the main computer via a 1 Gbit/s Ethernet connection. Each of the four smaller computers will have its own Ethernet cable running to a separate network socket on the main computer. As I wrote before, the Kinect 2 sends data at a speed of over 2 Gbit/s. Therefore the data either needs to be compressed or reduced, or both.

The main computer can now get the precise location of the users head, in order to update the real (absolute) position of the Oculus Rift. Since the Kinect also has a stream for IR data, it might also be possible to use the IR diodes of the Oculus Rift in order to track it more precisely.

Here are some ideas about the challenges of this project:

  • 30 Hz from the Kinect could be too slow (minor concern)
  • The latency from the Kinect, added with the latency from a network connection and two computers the data has to cross and the latency from the Oculus Rift itself could make the system react sluggish (major concern)
  • If Microsoft decides to add the ability to connect multiple Kinect cameras to one computer, it would make the whole network layer obsolete (cannot be avoided at the moment and the project could only benefit from the reduced latency)
  • The resolution from the depth (and IR) sensor is not great enough to accurately determine the heads position (not true, the Kinect can track the head position quite well)

And here is the link to part 2 of the series:

Overview Part 2: Using multiple Kinect 2 to create “holographic” films (animated 3D-models)

4 Comments:

  1. Pingback: Project Overview – Holodeck Now!

  2. Pingback: Overview Part 2: Using multiple Kinect 2 to create “holographic” films (animated 3D-models) – Holodeck Now!

  3. hi nico this sound very interesting, but i need your recomendation. I need a PCI CARD that support oculus CV1 and kinect V2 at the same time…can you tell me what pci card need for working with this devices?

  4. Are you talking about USB 3 for PCI (and not PCI-Express)? Then you probably won’t find any card that will work. As far as I remember PCI has a maximum bandwith of 133 MB/s for a 32-bit bus (so roughly 1Gbit/s) and I think the Kinect v2 alone needed around 1,5 or 2Gbit/s for the raw data. Also: those old cards are pretty hard to find, I think. With even a tiny PCI-E gen 2 x1 card, you get 500MB/s, so if that is the problem, it’s probably time to upgrade … (also, basically all of them will have USB3 onboard).

Leave a Reply

Your email address will not be published. Required fields are marked *