360° Stereoscopic VR Videos with Unity 3D


I
n 2016, 21st-century virtual reality really arrived. From cheap mobile experiences to exuberant desktop machines, if you wanted to dive into a virtual world, there was a way. These are some of the headsets which opened up possibilities. whatsapp-image-2017-01-04-at-1-12-06-am

The Oculus DK2, Samsung Gear VR and Google cardboard (in the picture above) and later, HTC Vive, PlayStation VR(Project Morpheus), Oculus CV1 and many other VR headsets also got arrived in the consumer market.

Google Cardboardgoogle-cardboard.jpg
The Cardboard is a platform that was developed by Google for use with a android smartphone. To work, you just start your video (or whatever VR thing it is you want to view), pop your phone into the Cardboard, hold the device up to your face, and you are in the immersive VR mode.

After initially supporting only Android, Google announced iOS support for the Unity plugin in May 2015 at the Google I/O 2015 conference. In March 2016, Google released VR View, an expansion of the Cardboard SDK allowing developers to embed 360-degree VR content on a web page or in a mobile app, across desktop, Android, and iOS.

Google provides three SDK’s for developing Cardboard applications: one for the Android  using Java, one for the game engine Unity using C#, and one for the iOS.

For latest updates on Google SDK :Google SDK Information

Samsung Gear VR
samsunggearvr.cms.jpg

The Gear VR is a mobile virtual reality headset developed by Samsung Electronics, in collaboration with Oculus, and manufactured by Samsung. The headset was released on 27 November 2015. Compatible devices are  S7, S7 edge, Note5, S6 edge+, S6, S6 edge. The development is only possible for Android platform using unity 3d and epic games unreal engine. Gear VR headset is to dive into a world of 360° experiences.

Gear VR is the most usable virtual reality headset I’ve ever tried, and the first to market after several years of hype. It’s the closest anyone’s come to making virtual reality into a palatable consumer experience.

You can order your Gear VR here: Buy Samsung Gear VR

whatsapp-image-2017-01-04-at-1-06-15-am

360° VR Video Publishing Platforms :
Over this past year we’ve seen the emergence of many 360 video publishing platforms where 360 video content is being hosted. popular ones are as follows:

  • Netflix
  • Facebook 360 Video
  • YouTube 360
  • Twitch
  • Littlstar

360 VR Video Player on PC or Mac for development or testing? You must download this free player GoPro VR Playergopro-ss

Before we start the development, its important to understand Stereoscopic and Monoscopic 360° Video for VR

Stereo vs Mono 360° Video for VR

What is the Difference?

A standard 360 video is just a flat equirectangular video displayed on a sphere. Think of it like the face of a world map on a globe, but with VR your head is on the inside of the globe looking at the inner surface. As you move, the head tracking on your device moves with you, giving you that feeling like you are inside the scene.

Stereoscopic 3D can add another level of immersion by adding depth data between the foreground and background. Your favorite 3D blockbuster films are typically shot with 2 lenses side by side, to give you a feeling of a different vantage point per eye. Like any production, this can look strange if poorly implemented, or absolutely amazing if done right.

Having said that, with stereoscopic 3D in VR, that depth information has to be overlaid and mapped to sphere. Because of parallax between cameras, this can be especially challenging. Any minor flaws or “stitch seams” in the footage are magnified in 3D, and sometimes anomalies occur in different places per eye – which makes it uncomfortable to watch.

 A lot of viewers gets confused about the difference of true 3D stereoscopic and monoscopic footage. Often times they’re fooled into thinking they’re looking at a 3D video. In a way, it is 3D because it’s projected on to a sphere, but it has no depth information between background and foreground. But after hundreds of tests and demos, I don’t think the general public even cares.

_______________________________________________________________

Now, Lets get started and dive in the development of 360 VR Videos.
In this post we will look at the following:

  •  monoscopic 360° VR Video Integration for Cardboard & Gear VR.
  • stereoscopic(Left-Right) 360° VR Video Integration for Cardboard & Gear VR.
  • stereoscopic(Top-Bottom) 360° VR Video Integration for Cardboard & Gear VR.
  • Flat 2D video on smartphone (full screen).

________________________________________________________________

***Remember, the checklist needed for us in order to develop google cardboard or Gear VR applications:

  1. Google cardboard SDK
  2. Oculus Utilities SDK
  3. Video Player.
  4. high-quality mesh sphere.
  5. Movie Shader
  6. 360 Video (mono or stereo)
  7. Radial Slider for Unity UI (download progress – Optional)

________________________________________________________________

Google VR SDK Download : Google SDK for UnityGoogle VR SDK Download SS.png

Or you  can also browse their git fork and samples here: Google Unity SDK Github Repository
gvr-github-ss

Import the sdk package in Unity:Google SDK Import SS.png

You may find Errors while importing: Please resolve as follows:Google SDK Errors.jpg

Also, Make sure to import the Google Backwards Compatibility package in the Assets folder:
GVR Backwards Compatibility Message SS.png

__________________________________________________________________

Please Note: Follow this section only if you’re developing for gear VR or Oculus platform. (Skip this section if you’re developing for only google cardboard)

Oculus SDK Download : Oculus Downloads or you can straightway download their oculus utilities package for unity : Download Oculus Utilities for unity 5Oculus-SDK-Utlities SS.png

Import the Oculus utilities package in the project if you’re building it for Gear VR or Oculus Rift platform.
:OVR-SDK-Import SS.png

_________________________________________________________________

Video Player in Unity
With Unity 5.6 beta version, good news is that a high quality video player will be integrated in the engine itself which will support mobile and desktop both the platforms. However, the Unity 5.6 version it is still in the development phase and a stable version will be released in March 2017. For more information : Unity RoadMap (Currently there is a Media Player Class which only supports PC and Mac).

Video Player available on unity Asset store:
Easy Movie Texture by JaeYunLee for $65easy-movie-texture-ss

AV Pro Video Player by Renderheads for $450av-pro-ss

If you are targeting multiple platforms and highest quality possible then , AV Pro Video Player is a must. You can also download their free trial version here : AV Pro Trial Versionav-pro-trial-version-ss

_________________________________________________________________

High-Quality 3D Mesh Sphere
You must use a high quality 3d mesh sphere to play the 360 videos on it. The default unity sphere has less polygons and will cut out certain portion of the video from the bottom of the model: hence a high quality model is definitely required.Do Remember to rotate the transform values to -90 in X direction of the model in the unity scene. Download link: High Quality Sphere 3d modelHigh Poly Spehere SS.jpg

__________________________________________________________________

Movie Shader
You can use an optimized custom shader written by me to wrap the sphere mesh normal’s inwards. With this shader you must use a black (4096*4096) Texture. You can download it here: Movie Shader Link
MovieShader SS.jpg

____________________________________________________________

360 Monoscopic Video
You can download a sample 360 Monoscopic video of 30MB and 4k resolution from YouTube link : Sample 360 Monoscopic Youtube Video or if you’re unable to download you can download from here which is saved on my drive:360 Monoscopic Interior Demo Video(this video is a copyright of Applicata Studio)360 Mono SS.jpg

360 Stereoscopic Video (Top-Bottom)
You may download a sample 360 stereoscopic video for testing from You Tube . here is what i have used: https://www.youtube.com/watch?v=tfoNzYTFXi0. (the copy right of the video solely belongs to the link owners).360 Stereo TB SS.jpg

_____________________________________________________________

Radial Slider Bar for Unity UI (Optional)
If you stream from any video link, you must show a menu radial slider bar which shows the progress of the download, as it will take some time to play: Here is the link of the Radial Slider package developed by “mgear” unity user:Radial Slider by mgear with values and lerp feature

______________________________________________________________

Once, we have downloaded and imported the requisite assets, lets start with the development in Unity.

360 Monoscopic VR Video Development for Google Cardboard
1> Create an empty scene and delete the existing Main Camera and Directional Light.
2> Drag and Drop the high poly sphere in the scene. Keep the Transform as below:
Transform values of sphere.jpg
3> Create a new material and apply the Unlit/Movie Shader to it. In the texture you can use dummy black texture already provided.Mono Material Info.jpg4> Drag and Drop GvrViewerMain and GvrControllerMain prefabs from “Assets\GoogleVR\Prefabs” directory in the project.
5>
Go to ‘Assets\GoogleVR\Demos\Scenes\GVRDemo’ and copy the Player object from the scene and paste that object in your scene that your working. I named it “360 Mono”.
6>
Create a new Object and name it Video Manager and attach the “MediaPlayerCtrl” script to that object. In Str File Name variable of MediaPlayerCtrl script, you must enter the name of your video which you have downloaded and must be kept in Streaming Assets Folder. (Make sure to add “.mp4” or the extension of your video may be “.mov”) at the end of the file. Look at the below reference:MediaPlayerCntrlReference.jpg

7> Check your Google Cardboard Hierarchy Scene View:

GVR Scene Overview.jpg
Unity Scene Objects Reference

8> Once your scene has everything stated above, click Play and you will be able to get split screen and the video playing screenshots below:

360 Mono GVR SS.jpg
Google Cardboard 360 Mono Scene in Play Mode
360 Mono GVR 2 SS.jpg
Google Cardboard 360 Mono Scene View in Play mode

9> (Optional) Now, if you wish to stream the video through your private server or google drive link or Dropbox link. You can stream & play with the link. :”https://dl.dropboxusercontent.com/s/0nevaxaxllcdhvp/Interior%20360%20Mono%20Demo.mp4″.
However, if you upload your own video, just be sure to replace in Scene link in the MediaPlayerCtrl script. Change : “https://dl.dropboxusercontent.com” with “https://dl.dropboxusercontent.com” and remove the end part: “?dl=0”.

GVR Streaming 360 video Mono 2 SS.jpg
Live streaming using a link

Note: For monoscopic 360 video there is no need for different LR or TB Materials. However for 3D Stereoscopic as explained we will need to create Left and Right or Top and Bottom materials with their offsets and tiling values as below.

Note: With Google Cardboard apk build please ensure that Virtual Reality is not enabled in the build settings.

This ends with the monoscopic 360 VR Video development.
__________________________________________________________________

360 Stereoscopic VR Video Development for Google Cardboard
With stereoscopic video, you need to store two images for each frame (one for the left eye and one for the right eye) Though modifying the code to support side-by-side (SBS) stereo is straightforward once you understand how it works.


1> Because we have two eyes, we need to have two identical spheres in the scene for (for playing L/R or T/B videos) instead of just one. The left eye image will be on one sphere, the right eye image will be on the other.

2> Unity materials have “tiling” and “offset” fields. The “tiling” field says how many times the texture should be repeated in the corresponding direction (X or Y) across the surface of the object.
Create two Materials for L/R or T/B Video to play. If  you have one video with T/B layout, The Reference, as follows:

TB_material.jpg

lr_material

3> You must create two layers for L/R or T/B respectively for Camera culling. For both type of LR and TD videos i have created 4 layers.

unity-layers-ss
Tags and Layers Unity

 

 

 

 

 

 

 

 

 

4> Create 2 Cameras and set their Camera rendering culling mask for Left Eye View and Right Eye View for Top and Bottom tags as shown. If  you have Left and Right videos then tags and apply it accordingly.

Top_Bottom-CameraCulling.jpg

5> After creating Left and Right Materials. Apply the materials to their Left and Right 3d sphere objects. This material will ensure that your rendering is on that object as per the tiling and offsets values set in the material.

Oculus Left-Right Object.jpg

6> After applying Left and Right Materials. Apply the Tags of Layers to to their Left and Right 3d sphere objects. This layers will ensure that your rendering is on that object from camera in the culling mask.

LayerSet.jpg

7> MediaCtrlPlayer should be provided with 2 target objects to play the respective videos as shown in the inspector.MEDIA PLAYER OBJECT SS.jpg

8> Play in the scene and you should be able to get a 3d depth and stereoscopic View.LEFT RIGHT SS.pngA preview of 360 Stereoscopic VR Video in Play mode

Note: With Google Cardboard apk build please ensure that Virtual Reality is not enabled in the build settings.

This ends with the stereoscopic 360 VR Video development.
_________________________________________________________________

360 Monoscopic and Stereoscopic Video Development for Gear VR:

For Gear VR development the scene objects remains the same, except the Camera prefab.

360 Monoscopic for Gear VR
1> Delete GvrViewerMain and GvrControllerMain prefabs and drag and drop “OVRCameraRig” from “Assets\OVR\Prefabs” directory in the project.Rest everything remains the same. Look at the reference below:

If you’re using Gear VR Oculus Utilities SDK , then here is how your scene should ideally look like:
oculus-360-mono-ss

360 Stereoscopic for Gear VR
1> Create two spheres for the stereoscopic video and remember to scale the sphere objects as per your camera FOV. Reference below:

OCULUS TOP BOTTOM SS.png
360 Sphere mesh Object Scaling

2> Just duplicate the MainCamera in OVRCameraRig prefab and mark it left and right eye camera. Also do change their culling mask and target eye options as shown below.

LR-EYE_Camera.jpg
Stereoscopic Camera Layouts for Oculus Gear VR

3> MediaCtrlPlayer should be provided with 2 target objects to play the respective videos as shown in the inspector.MEDIA PLAYER OBJECT SS.jpg

4> Play in the scene and you should be able to get a 3d depth and stereoscopic View for Gear VR.OCULUS TOP BOTTOM SS.pngStereoscopic 360 VR Video playing in the scene

 Note: Make Sure to enable the Virtual Reality in Build Setting:
OCULUS BUILD SETTING SS.png

This ends the development of 360 Mono and Stereo for Gear VR using Unity 3D.
__________________________________________________________________

Tips
1> Keep the quality of the content and video 4k  resolution otherwise you then start noticing the pixels of the display, the pressure of the headset on your face, the growing motion sickness and the chance to lose yourself in virtual reality disappears.
2> If Android and iOS both the platforms, you wish to develop, then you must start with android, first develop and test it completely. Once completed the development then just simply port it for iOS.
3>  Your google cardboard needs to configured with your phone. hence you will need to scan the QR code with your phone. If you do not configure your 360 video will have a cutout for left and right eye while playing in the cardboard. for more information: Cardboard QR codes
4> Gear VR requires the app open in the headset, hence you will need to insert it every time and play. if you want to escape that and save your development time, you can go to developer settings and disable Gear VR service. now it will allow you to open the apps even without the headset.

Thank you for reading out this post. Hope this helps to those who are looking forward for virtual Reality development using unity3d.

Next week, i will be sharing a expertise notes and in depth information for 3D modelling team. The coming post will be crucial for developers and design team and serve as a in depth information to design and develop 3d assets. stay tuned 🙂

P.S For any queries or clarification feel free to post a comment. I will be happy to answer.

P.P.S This post is dedicated to a fellow developer and a good friend, Nidhi Ajwani. Happy Coding! 🙂

Advertisements

7 thoughts on “360° Stereoscopic VR Videos with Unity 3D

  1. orz

    Unity has a media player now, so I can make monoscopic video without buying any asset, however, stereoscopic videos don’t work, as I don’t have the MediaPlayerCtrl script that you have. How do I do it with unity’s media player?

    Like

  2. Conner Jeong

    Dear Summit, I am implementing stereoscopic 360 photo viewer using Unity 3D/GearVR. your article is very helpful for me. I have questions. You explained stereoscopic 360 video. But I’d like to watch stereoscopic 360 PHOTO. My some trial were failed with wrong rendering. Image on GearVR was distorted. I’m using side-by-side sample image from internet. Can you give me any hint?

    Like

    1. its very easy.. just put two spheres.. actually the ovrcamera rig has changed a bit and thr latest versiin has stereoscopic camera check or enabe option.. to answer ur question u will need to 2 spheres.. onr matk it as left and other as right in layers.. go tonur camera rig and out the culling mask to left for left camera and right to right camera.. thats about it u will get it.. if u want u can shoot an email to summitgamesentertainment@gmail.com.. i can take team viewer and do it in 3 minutes for u

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s