Ideas - Game Development and Vision: August 2017

Unity tutorial on using a 3D stereoscopic display with shaders

This is a small tutorial on how to use the Dimenco monitor with Unity3D. At work we managed to get one of these monitors and we were interested in using it with Unity applications, and this is how I did it. As I have not found any information on how to use the 3D display in Unity I think the community will appreciate a little bit of light into the issue.

Introduction

Let's start with a brief introduction on Dimenco displays: Dimenco monitors are 3D monitors without glasses (3D displays). This implies that at certain positions and angles the image in the monitor will be seen in 3D. The pros is that you do not need glasses, the cons is that you need to be in specific positions and cannot move much (and usually far away from the monitor).

How does a Dimenco monitor, or most of glass-free 3D technology work? The image is divided into 2 parts: Color information and depth information. The color side has a header that tells Dimenco this is a 3D image. Both sides, color and depth, will have in between dark lines. The monitor itself will do all the merging to display the image in 3D. This tutorial will explain step by step how to achieve this. is how the image is setup for the Dimenco 3D display

Stereoscopic view of the image

Monitor setup:

This is the easy part. Have the monitor on and connected to the PC at 4K resolution (3840x2160). Simple :)

Unity setup:

The basic idea is to have a view of the scene divided into 2 parts: color image and depth image. The key here is that the output resolution of the scene has to be normal HD resolution (1920x1080), not in 4K.

Part 1 - Unity

Visualization of two images side by side:

Create a Canvas with 2 child objects. The canvas needs to be exactly HD resolution (1920x1080). The two child objects will have an image component each and need to be half the canvas size (960x1080).

Basic Canvas setup

One of the images on the left side and the other on the right side. See the 2 different images in each side below. I have applied color to the images so it is easier to visualize them.

Details on the position of each canvas child object

These images do not show anything from the scene yet. For that the camera view has to be projected into the canvas images.

Visualization of the scene in the canvas:

Create 2 render textures (for color image and depth image) in the Project (no Hierarchy).

renderTexture_color
renderTexture_depth

See the configuration in the following images to set them up.

Render Texture config

Note that the textures are not rendering anything yet. For that there needs to be a camera that renders to the texture, which is the next step. For the color render texture the image can be improved a bit by changing:

Anti-aliasing: 8 samples. (if image does not visualize properly reduce to none)
Filter mode: Trilinear.

The Color Format in Unity later versions (only verified in 2019.2.13f1) has changed. I am unavailable to verify in a Dimenco display as I do not have access to any now, but this should work:

For the renderTexture_color: R16G16B16A16_UNORM or R32G32B32A32_UINT
For the renderTexture_depth: DEPTH_AUTO

Camera setup (rendering to texture):

The ideal case for this is to have a single camera that renders two different textures, but Unity only allows 1 camera rendering to a single texture. To fix this problem a simple setup is created: a new empty GameObject with 3 cameras as children. Why 3? Because Unity needs a main camera, and the other two cameras are only rendering to textures.

Main Camera will be a normal camera. The configuration here is completely arbitrary as it will not affect anything in the scene, or what is displayed in the monitor. We will only see the canvas. Set culling mask to “nothing”.
Camera_color will render to target texture “renderTexture_color”.
Camera_depth will render to target texture “renderTexture_depth”.

The position and rotation of each camera (Camera_color and Camera_depth) should be exactly the same. In case the cameras need to be moved, just move the parent object.

Camera Setup

This setup renders to the textures what is obtained from each camera, but this is not shown in the canvas yet. For that, the render texture will need to using a custom shader. be projected into a material and that material will be placed into the canvas image.

Part 2 - Shaders

To be honest this is the first time I have worked with shaders, so forgive me if I do not follow best practices. The first thing is to create a predefined unlit shader.

Shader creation

This shader will be used as a baseline to modify the color and depth images. Check Unity docs for

more info on shader examples in this link. This is the default code from the unlit shader:

Shader "Unlit/NewUnlitShader"
{
 Properties
 {
  _MainTex ("Texture", 2D) = "white" {}
 }
 SubShader
 {
  Tags { "RenderType"="Opaque" }
  LOD 100

  Pass
  {
   CGPROGRAM
   #pragma vertex vert
   #pragma fragment frag
   // make fog work
   #pragma multi_compile_fog
   
   #include "UnityCG.cginc"

   struct appdata
   {
    float4 vertex : POSITION;
    float2 uv : TEXCOORD0;
   };

   struct v2f
   {
    float2 uv : TEXCOORD0;
    UNITY_FOG_COORDS(1)
    float4 vertex : SV_POSITION;
   };

   sampler2D _MainTex;
   float4 _MainTex_ST;
   
   v2f vert (appdata v)
   {
    v2f o;
    o.vertex = UnityObjectToClipPos(v.vertex);
    o.uv = TRANSFORM_TEX(v.uv, _MainTex);
    UNITY_TRANSFER_FOG(o,o.vertex);
    return o;
   }
   
   fixed4 frag (v2f i) : SV_Target
   {
    // sample the texture
    fixed4 col = tex2D(_MainTex, i.uv);
    // apply fog
    UNITY_APPLY_FOG(i.fogCoord, col);
    return col;
   }
   ENDCG
  }
 }
}

Depth shader (2DPlusDepth_Depth):

This is the easy one, as you only have to create in-between black lines for each texture. Additionally I will also add the functionality to allow for a parameter to change how strong the depth affects the 3D immersion.

It is good to realize that the fragment shader will affect every pixel using the UV coordinates. This means coordinates that go from 0 to 1. To be able to apply the in-between black lines we will transform the UV coordinates into height and width coordinates of 540x960 (As the image size required by Dimenco). It is good to remember that each pixel will run this function, so there is no need to create loops that iterate over the whole image. We only need to see if the current pixel belongs to an odd line or an even line.

First lets change the name of the shader in the first line. This will give the path and name of the shader.

Change

1	Shader "Unlit/NewUnlitShader"

for

1	Shader "Dimenco/2DPlusDepth_Depth"

The next thing is the function where the black lines are going to be written.

fixed4 frag (v2f i) : SV_Target
{

 int coord_y = i.uv.y * 540; //Gives us the current row.
 int coord_x = i.uv.x * 960; //Note: We do not use this, but it is more clear to see it.

 float mod_value = fmod(coord_y,2.0); //find whether this is an odd or even line.
 fixed4 col;

 if (mod_value == 1)
 {
  //Odd line: use the value from the texture  
  col = tex2D(_MainTex, i.uv);
 }
 else
 {
  //Even line: create a black pixel.
  col = float4(0, 0, 0, 0);
 }
 // apply fog
 UNITY_APPLY_FOG(i.fogCoord, col);
 return col;
    
}

This is the basic structure of the shader. To add a property that can modify the shader depth values in run time we do the following:

Properties
{
 _MainTex ("Texture", 2D) = "white" {}
 _DepthMod("Depth Modifier", float) = 1 //Added field
}

Pass
{
 […] //Other stuff

 sampler2D _MainTex;
 float4 _MainTex_ST;
 float _DepthMod; //Again, add here the field from Properties

 […] //Other Stuff   

 fixed4 frag (v2f i) : SV_Target
 {
    
  int coord_y = i.uv.y * 540; 
  int coord_x = i.uv.x * 960; 

  float mod_value = fmod(coord_y,2.0); 
  fixed4 col;

  if (mod_value == 1)
  {
   col = _DepthMod * tex2D(_MainTex, i.uv); //Use the new property here
  }
  else
  {
   col = float4(0, 0, 0, 0);
  }  

  UNITY_APPLY_FOG(i.fogCoord, col);
  return col;
    
 }
 ENDCG
}

The added/ modified lines have comments, while the previous comments have been removed. This way the material that uses this shader will have a variable “Depth Modifier” that will affect the depth image at runtime. From my own testing I have experienced that the value should be [1,3]. To manipulate this variable a simple script can be created. To see how to manipulate shader variables check the Unity documentation.

Color Shader (2DPlusDepth_Color):

The color shader has two distinct parts: in-between black lines and the header. For the black lines, the same procedure as before is followed. Just copy the same steps as before. First create a new shader (2DPlusDepth_Color) and change the name:

1	Shader "Unlit/NewUnlitShader"

for

1	Shader "Dimenco/2DPlusDepth_Color"

The header is a bit more tricky. For details on each component of the dimenco header, refer to the documentation in their website. I will just write down the standard header for the 3 possibilities that are needed:

Header 1: F10140800000C42DD3AF (Common to all 3 cases)

Header 2:

2D-plus-Depth: F2140000000000000000000000000000000036958221
Declipse – removed redundant data: F2149A0000000000000000000000000000006BF6C689
Declipse – full background data: F214EF0000000000000000000000000000002FF0C45F

We will only implement the header 1 and header 2 (2D-plus-Depth), which have been highlighted. The header 2 for the other two cases are also here, but they are commented. There is a “problem” with the UV coordinates, as they go [0 1] from left to right and bottom to top. The header has to be on top, so we need to check that the “row == height-1”. This can be observed in the code.

fixed4 frag (v2f i) : SV_Target
{    
 int row = i.uv.y * 540; //For the black lines
 int column = i.uv.x * 960; //For the header

 float mod_value = fmod(row,2.0);
 fixed4 col;

 if (mod_value == 1)
 {
  // sample the texture in every odd row
  col = tex2D(_MainTex, i.uv);

  //Header:
  //Blue_channel(2*(7-row)+16*column, 0)^7 = H(column)^row
  float pos_row = 16 * row;
  float pos_col = 2 * (7 - column);

  int width = 960;
  int height = 540;

  if (row == height-1 && column <= 494)
  { 
   //Only first row

   //Header 1:
   //F1,01, 40,80, 00,00, C4,2D, D3,AF: H[0],..., H[9]
   //1111 0001, 0000 0001 (F1,01) - H[0], H[1]
   //0100 0000, 1000 0000 (40,80) - H[2], H[3]
   //0000 0000, 0000 0000 (00,00) - H[4], H[5]
   //1100 0100, 0010 1101 (C4,2D) - H[6], H[7]
   //1101 0011, 1010 1111 (D3,AF) - H[8], H[9]  

   float headerA_0[8] = { 1,1,1,1, 0,0,0,1 }; //H[0]
   float headerA_1[8] = { 0,0,0,0, 0,0,0,1 }; //H[1]
   float headerA_2[8] = { 0,1,0,0, 0,0,0,0 }; //H[2]
   float headerA_3[8] = { 1,0,0,0, 0,0,0,0 }; //H[3]
   float headerA_4[8] = { 0,0,0,0, 0,0,0,0 }; //H[4]
   float headerA_5[8] = { 0,0,0,0, 0,0,0,0 }; //H[5]
   float headerA_6[8] = { 1,1,0,0, 0,1,0,0 }; //H[6]
   float headerA_7[8] = { 0,0,1,0, 1,1,0,1 }; //H[7]
   float headerA_8[8] = { 1,1,0,1, 0,0,1,1 }; //H[8]
   float headerA_9[8] = { 1,0,1,0, 1,1,1,1 }; //H[9]

   //Header 2:
   //F2,14, 00,00, 00,00, 00,00, 00,00, 00,00, 00,00, 00,00, 00,00, 36,95, 82,21: H[10],..., H[31]
   //1111 0010, 0001 0100 (F2,14) - H[10], H[11]
   //0000 0000, 0000 0000 (00,00) - H[12], H[13]
   //0000 0000, 0000 0000 (00,00) - H[14], H[15]
   //0000 0000, 0000 0000 (00,00) - H[16], H[17]
   //0000 0000, 0000 0000 (00,00) - H[18], H[19]
   //0000 0000, 0000 0000 (00,00) - H[20], H[21]
   //0000 0000, 0000 0000 (00,00) - H[22], H[23]
   //0000 0000, 0000 0000 (00,00) - H[24], H[25]
   //0000 0000, 0000 0000 (00,00) - H[26], H[27]
   //0011 0101, 1001 0101 (36,95) - H[28], H[29]
   //1000 0010, 0010 0001 (82,21) - H[30], H[31]

   float headerB_10[8] = { 1,1,1,1, 0,0,1,0 }; //H[10]
   float headerB_11[8] =  { 0,0,0,1, 0,1,0,0 }; //H[11]
   float headerB_12[8] = { 0,0,0,0, 0,0,0,0 }; //H[12]
   float headerB_13_27[8] = { 0,0,0,0, 0,0,0,0 }; //H[13]-H[27]
   float headerB_28[8] = { 0,0,1,1, 0,1,0,1 }; //H[28]
   float headerB_29[8] = { 1,0,0,1, 0,1,0,1 }; //H[29]
   float headerB_30[8] = { 1,0,0,0, 0,0,1,0 }; //H[30]
   float headerB_31[8] = { 0,0,1,0, 0,0,0,1 }; //H[31]

   /******** Other 3D inputs ********/

   //Declipse – ‘Removed redundant data’ format. Change H[12], H[28] - H[31]  
   //float headerB_12[8] =   { 1,0,0,1, 1,0,1,0 }; //H[12]
   //float headerB_28[8] =   { 0,1,1,0, 1,0,1,1 }; //H[28]
   //float headerB_29[8] =   { 1,1,1,1, 0,1,1,0 }; //H[29]
   //float headerB_30[8] =   { 1,1,0,0, 0,1,1,0 }; //H[30]
   //float headerB_31[8] =   { 1,0,0,0, 1,0,0,1 }; //H[31]

   //Declipse – ‘Full background data’ format. Change H[12], H[28] - H[31] 
   //float headerB_12[8] =   { 1,1,1,0, 1,1,1,1 }; //H[12]
   //float headerB_28[8] =   { 0,0,1,0, 1,1,1,1 }; //H[28]
   //float headerB_29[8] =   { 1,1,1,1, 0,0,0,0 }; //H[29]
   //float headerB_30[8] =   { 1,1,0,0, 0,1,0,0 }; //H[30]
   //float headerB_31[8] =   { 0,1,0,1, 1,1,1,1 }; //H[31]


   //Only even columns of first row
   if (fmod(column, 2.0) == 0)
   {
    int header = floor(column / 16);
    int headerPixel = fmod(column / 2, 8.0);

    if (header == 0) //Header A
    {
     col.b = headerA_0[headerPixel];
    } 
    else if (header == 1)
    {
     col.b = headerA_1[headerPixel];
    }
    else if (header == 2)
    {
     col.b = headerA_2[headerPixel];
    }
    else if (header == 3)
    {
     col.b = headerA_3[headerPixel];
    }
    else if (header == 4)
    {
     col.b = headerA_4[headerPixel];
    }
    else if (header == 5)
    {
     col.b = headerA_5[headerPixel];
    }
    else if (header == 6)
    {
     col.b = headerA_6[headerPixel];
    }
    else if (header == 7)
    {
     col.b = headerA_7[headerPixel];
    }
    else if (header == 8)
    {
     col.b = headerA_8[headerPixel];
    }
    else if (header == 9)
    {
     col.b = headerA_9[headerPixel];
    }
    else if (header == 10) //Header B
    {
     col.b = headerB_10[headerPixel];
    }
    else if (header == 11)
    {
     col.b = headerB_11[headerPixel];
    }
    else if (header == 12)
    {
     col.b = headerB_12[headerPixel];
    }
    else if (header >= 13 && header <= 27)
    {
     col.b = headerB_13_27[headerPixel];
    }
    else if (header == 28)
    {
     col.b = headerB_28[headerPixel];
    }
    else if (header == 29)
    {
     col.b = headerB_29[headerPixel];
    }
    else if (header == 30)
    {
     col.b = headerB_30[headerPixel];
    }
    else if (header == 31)
    {
     col.b = headerB_31[headerPixel];
    }

   }
  }
 }
 else
 {
  // black lines for every even line
  col = float4(0, 0, 0, 0);
 }
 // apply fog

 UNITY_APPLY_FOG(i.fogCoord, col);
 return col;
 
}

In case you want to change the headers, I would recommend to use this website to compute the checksum of CRC-32.

For example, in the case of modifying H[0]-H[6] the checksum has to be in H[7]-H[9]. If you use the page mentioned just in the previous paragraph select the following options

Web: http://www.sunshine2k.de/coding/javascript/crc/crc_js.html
Options:

CRC-32, Custom:

Input reflected: Uncheck; Result reflected: Uncheck.
Polynomial: 0x04C11DB7L;
Initial Value: 0x0;
Final Xor Value: 0x0;

CRC Input Data (bytes): 0xF1 0x01 0x40 0x80 0x00 0x00 (change to whichever you use in the new H[0]-H[6])

Once both shaders have been created we have to apply them into the materials to be added to the canvas.

Materials:

To visualize the images in the canvas, two new materials have to be created:

Dimenco_color: Apply the Shader “Dimenco/2DplusDepth_color”. Select the render texture “renderTexture_color” in the Detail (RGB).
Dimenco_depth: Apply the Shader “Dimenco/2DplusDepth_depth”. Select the render texture “renderTexture_depth” in the Detail (RGB).

Example of the material "Dimenco_Color"

Apply the materials to the component Image of Image_left and Image_Right GameObjects in the canvas. Change the color to white if you had it with other values.

Example of material in image component object within the canvas

Both images can be observed in each side of the display.

These materials are applied to the image left (color) and right (depth) in the canvas and everything should be alright. Once you build the project and run it in full screen mode the images should be in 3D. If you pay close attention you will spot bright blue pixels on the top left corner. These belong to the header and is normal (its the encoding that the Dimenco monitor uses to turn the image into 3D).

Remember the monitor has to be in 4K resolution and the game in HD resolution.

Sometimes there is an error saying “Releasing render texture that is set as Camera.targetTexture!”. If that happens it means the render texture is no longer attached to the camera. Attach it and build the project again, everthing should be alright. From my experience that only happens when you modify the render texture and build the project. Once you reattach and build it (without modifying the render texture again) the error does not happen again.

This is my first tutorial, so I hope everything is clear and easy to follow. In case there is a mistake, or any other input, please leave a comment.

Note: I have verified the shaders' with Unity 2019.2.13f1 and they work. I do not have a Dimenco display at hand now, but it should be working there as well.

Downloads

A simplified unity asset version of this tutorial can be found here (Unity 2019.2.13). Disclaimer: This version has not been tested using the Dimenco display.

For reference: this is the image obtained

Ideas - Game Development and Vision

Sunday, August 13, 2017

Tutorial to use a Dimenco Monitor (3D Display) in a Unity 5 project