Sunday, August 13, 2017

Tutorial to use a Dimenco Monitor (3D Display) in a Unity 5 project

Unity tutorial on using a 3D stereoscopic display with shaders



This is a small tutorial on how to use the Dimenco monitor with Unity3D. At work we managed to get one of these monitors and we were interested in using it with Unity applications, and this is how I did it. As I have not found any information on how to use the 3D display in Unity I think the community will appreciate a little bit of light into the issue.

Introduction


Let's start with a brief introduction on Dimenco displays: Dimenco monitors are 3D monitors without glasses (3D displays). This implies that at certain positions and angles the image in the monitor will be seen in 3D. The pros is that you do not need glasses, the cons is that you need to be in specific positions and cannot move much (and usually far away from the monitor). 

How does a Dimenco monitor, or most of glass-free 3D technology work? The image is divided into 2 parts: Color information and depth information. The color side has a header that tells Dimenco this is a 3D image. Both sides, color and depth, will have in between dark lines. The monitor itself will do all the merging to display the image in 3D. This tutorial will explain step by step how to achieve this. is how the image is setup for the Dimenco 3D display 



Stereoscopic view of the image

Monitor setup:

This is the easy part. Have the monitor on and connected to the PC at 4K resolution (3840x2160). Simple :)

Unity setup:

The basic idea is to have a view of the scene divided into 2 parts: color image and depth image. The key here is that the output resolution of the scene has to be normal HD resolution (1920x1080), not in 4K. 


Part 1 - Unity 

Visualization of two images side by side:

Create a Canvas with 2 child objects. The canvas needs to be exactly HD resolution (1920x1080). The two child objects will have an image component each and need to be half the canvas size (960x1080). 
Basic Canvas setup

One of the images on the left side and the other on the right side. See the 2 different images in each side below. I have applied color to the images so it is easier to visualize them. 

Details on the position of each canvas child object
These images do not show anything from the scene yet. For that the camera view has to be projected into the canvas images.

Visualization of the scene in the canvas:

Create 2 render textures (for color image and depth image) in the Project (no Hierarchy). 
  • renderTexture_color
  • renderTexture_depth
See the configuration in the following images to set them up. 


Render Texture config
Note that the textures are not rendering anything yet. For that there needs to be a camera that renders to the texture, which is the next step. For the color render texture the image can be improved a bit by changing:
  • Anti-aliasing: 8 samples. (if image does not visualize properly reduce to none)
  • Filter mode: Trilinear.
The Color Format in Unity later versions (only verified in 2019.2.13f1) has changed. I am unavailable to verify in a Dimenco display as I do not have access to any now, but this should work:
  • For the renderTexture_color: R16G16B16A16_UNORM or R32G32B32A32_UINT
  • For the renderTexture_depth: DEPTH_AUTO

Camera setup (rendering to texture):

The ideal case for this is to have a single camera that renders two different textures, but Unity only allows 1 camera rendering to a single texture. To fix this problem a simple setup is created: a new empty GameObject with 3 cameras as children. Why 3? Because Unity needs a main camera, and the other two cameras are only rendering to textures.
  • Main Camera will be a normal camera. The configuration here is completely arbitrary as it will not affect anything in the scene, or what is displayed in the monitor. We will only see the canvas. Set culling mask to “nothing”.
  • Camera_color will render to target texture “renderTexture_color”.
  • Camera_depth will render to target texture “renderTexture_depth”.
The position and rotation of each camera (Camera_color and Camera_depth) should be exactly the same. In case the cameras need to be moved, just move the parent object. 


Camera Setup
This setup renders to the textures what is obtained from each camera, but this is not shown in the canvas yet. For that, the render texture will need to using a custom shader. be projected into a material and that material will be placed into the canvas image.

Part 2 - Shaders

To be honest this is the first time I have worked with shaders, so forgive me if I do not follow best practices. The first thing is to create a predefined unlit shader. 

Shader creation
This shader will be used as a baseline to modify the color and depth images. Check Unity docs for 
more info on shader examples in this link. This is the default code from the unlit shader:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
Shader "Unlit/NewUnlitShader"
{
 Properties
 {
  _MainTex ("Texture", 2D) = "white" {}
 }
 SubShader
 {
  Tags { "RenderType"="Opaque" }
  LOD 100

  Pass
  {
   CGPROGRAM
   #pragma vertex vert
   #pragma fragment frag
   // make fog work
   #pragma multi_compile_fog
   
   #include "UnityCG.cginc"

   struct appdata
   {
    float4 vertex : POSITION;
    float2 uv : TEXCOORD0;
   };

   struct v2f
   {
    float2 uv : TEXCOORD0;
    UNITY_FOG_COORDS(1)
    float4 vertex : SV_POSITION;
   };

   sampler2D _MainTex;
   float4 _MainTex_ST;
   
   v2f vert (appdata v)
   {
    v2f o;
    o.vertex = UnityObjectToClipPos(v.vertex);
    o.uv = TRANSFORM_TEX(v.uv, _MainTex);
    UNITY_TRANSFER_FOG(o,o.vertex);
    return o;
   }
   
   fixed4 frag (v2f i) : SV_Target
   {
    // sample the texture
    fixed4 col = tex2D(_MainTex, i.uv);
    // apply fog
    UNITY_APPLY_FOG(i.fogCoord, col);
    return col;
   }
   ENDCG
  }
 }
}

Depth shader (2DPlusDepth_Depth): 

This is the easy one, as you only have to create in-between black lines for each texture. Additionally I will also add the functionality to allow for a parameter to change how strong the depth affects the 3D immersion.

It is good to realize that the fragment shader will affect every pixel using the UV coordinates. This means coordinates that go from 0 to 1. To be able to apply the in-between black lines we will transform the UV coordinates into height and width coordinates of 540x960 (As the image size required by Dimenco). It is good to remember that each pixel will run this function, so there is no need to create loops that iterate over the whole image. We only need to see if the current pixel belongs to an odd line or an even line.

First lets change the name of the shader in the first line. This will give the path and name of the shader. 

Change 
1
Shader "Unlit/NewUnlitShader"

for
1
Shader "Dimenco/2DPlusDepth_Depth"


The next thing is the function where the black lines are going to be written.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
fixed4 frag (v2f i) : SV_Target
{

 int coord_y = i.uv.y * 540; //Gives us the current row.
 int coord_x = i.uv.x * 960; //Note: We do not use this, but it is more clear to see it.

 float mod_value = fmod(coord_y,2.0); //find whether this is an odd or even line.
 fixed4 col;

 if (mod_value == 1)
 {
  //Odd line: use the value from the texture  
  col = tex2D(_MainTex, i.uv);
 }
 else
 {
  //Even line: create a black pixel.
  col = float4(0, 0, 0, 0);
 }
 // apply fog
 UNITY_APPLY_FOG(i.fogCoord, col);
 return col;
    
}


This is the basic structure of the shader. To add a property that can modify the shader depth values in run time we do the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
Properties
{
 _MainTex ("Texture", 2D) = "white" {}
 _DepthMod("Depth Modifier", float) = 1 //Added field
}

Pass
{
 [] //Other stuff

 sampler2D _MainTex;
 float4 _MainTex_ST;
 float _DepthMod; //Again, add here the field from Properties

 [] //Other Stuff   

 fixed4 frag (v2f i) : SV_Target
 {
    
  int coord_y = i.uv.y * 540; 
  int coord_x = i.uv.x * 960; 

  float mod_value = fmod(coord_y,2.0); 
  fixed4 col;

  if (mod_value == 1)
  {
   col = _DepthMod * tex2D(_MainTex, i.uv); //Use the new property here
  }
  else
  {
   col = float4(0, 0, 0, 0);
  }  

  UNITY_APPLY_FOG(i.fogCoord, col);
  return col;
    
 }
 ENDCG
}


The added/ modified lines have comments, while the previous comments have been removed. This way the material that uses this shader will have a variable “Depth Modifier” that will affect the depth image at runtime. From my own testing I have experienced that the value should be [1,3]. To manipulate this variable a simple script can be created. To see how to manipulate shader variables check the Unity documentation.

Color Shader (2DPlusDepth_Color):

The color shader has two distinct parts: in-between black lines and the header. For the black lines, the same procedure as before is followed. Just copy the same steps as before. First create a new shader (2DPlusDepth_Color) and change the name:

1
Shader "Unlit/NewUnlitShader"

for
1
Shader "Dimenco/2DPlusDepth_Color"

The header is a bit more tricky. For details on each component of the dimenco header, refer to the documentation in their website. I will just write down the standard header for the 3 possibilities that are needed:

Header 1: F10140800000C42DD3AF (Common to all 3 cases)
Header 2:
  • 2D-plus-Depth: F2140000000000000000000000000000000036958221
  • Declipse – removed redundant data: F2149A0000000000000000000000000000006BF6C689
  • Declipse – full background data: F214EF0000000000000000000000000000002FF0C45F
We will only implement the header 1 and header 2 (2D-plus-Depth), which have been highlighted. The header 2 for the other two cases are also here, but they are commented. There is a “problem” with the UV coordinates, as they go [0 1] from left to right and bottom to top. The header has to be on top, so we need to check that the “row == height-1”. This can be observed in the code.


  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
fixed4 frag (v2f i) : SV_Target
{    
 int row = i.uv.y * 540; //For the black lines
 int column = i.uv.x * 960; //For the header

 float mod_value = fmod(row,2.0);
 fixed4 col;

 if (mod_value == 1)
 {
  // sample the texture in every odd row
  col = tex2D(_MainTex, i.uv);

  //Header:
  //Blue_channel(2*(7-row)+16*column, 0)^7 = H(column)^row
  float pos_row = 16 * row;
  float pos_col = 2 * (7 - column);

  int width = 960;
  int height = 540;

  if (row == height-1 && column <= 494)
  { 
   //Only first row

   //Header 1:
   //F1,01, 40,80, 00,00, C4,2D, D3,AF: H[0],..., H[9]
   //1111 0001, 0000 0001 (F1,01) - H[0], H[1]
   //0100 0000, 1000 0000 (40,80) - H[2], H[3]
   //0000 0000, 0000 0000 (00,00) - H[4], H[5]
   //1100 0100, 0010 1101 (C4,2D) - H[6], H[7]
   //1101 0011, 1010 1111 (D3,AF) - H[8], H[9]  

   float headerA_0[8] = { 1,1,1,1, 0,0,0,1 }; //H[0]
   float headerA_1[8] = { 0,0,0,0, 0,0,0,1 }; //H[1]
   float headerA_2[8] = { 0,1,0,0, 0,0,0,0 }; //H[2]
   float headerA_3[8] = { 1,0,0,0, 0,0,0,0 }; //H[3]
   float headerA_4[8] = { 0,0,0,0, 0,0,0,0 }; //H[4]
   float headerA_5[8] = { 0,0,0,0, 0,0,0,0 }; //H[5]
   float headerA_6[8] = { 1,1,0,0, 0,1,0,0 }; //H[6]
   float headerA_7[8] = { 0,0,1,0, 1,1,0,1 }; //H[7]
   float headerA_8[8] = { 1,1,0,1, 0,0,1,1 }; //H[8]
   float headerA_9[8] = { 1,0,1,0, 1,1,1,1 }; //H[9]

   //Header 2:
   //F2,14, 00,00, 00,00, 00,00, 00,00, 00,00, 00,00, 00,00, 00,00, 36,95, 82,21: H[10],..., H[31]
   //1111 0010, 0001 0100 (F2,14) - H[10], H[11]
   //0000 0000, 0000 0000 (00,00) - H[12], H[13]
   //0000 0000, 0000 0000 (00,00) - H[14], H[15]
   //0000 0000, 0000 0000 (00,00) - H[16], H[17]
   //0000 0000, 0000 0000 (00,00) - H[18], H[19]
   //0000 0000, 0000 0000 (00,00) - H[20], H[21]
   //0000 0000, 0000 0000 (00,00) - H[22], H[23]
   //0000 0000, 0000 0000 (00,00) - H[24], H[25]
   //0000 0000, 0000 0000 (00,00) - H[26], H[27]
   //0011 0101, 1001 0101 (36,95) - H[28], H[29]
   //1000 0010, 0010 0001 (82,21) - H[30], H[31]

   float headerB_10[8] = { 1,1,1,1, 0,0,1,0 }; //H[10]
   float headerB_11[8] =  { 0,0,0,1, 0,1,0,0 }; //H[11]
   float headerB_12[8] = { 0,0,0,0, 0,0,0,0 }; //H[12]
   float headerB_13_27[8] = { 0,0,0,0, 0,0,0,0 }; //H[13]-H[27]
   float headerB_28[8] = { 0,0,1,1, 0,1,0,1 }; //H[28]
   float headerB_29[8] = { 1,0,0,1, 0,1,0,1 }; //H[29]
   float headerB_30[8] = { 1,0,0,0, 0,0,1,0 }; //H[30]
   float headerB_31[8] = { 0,0,1,0, 0,0,0,1 }; //H[31]

   /******** Other 3D inputs ********/

   //Declipse – ‘Removed redundant data’ format. Change H[12], H[28] - H[31]  
   //float headerB_12[8] =   { 1,0,0,1, 1,0,1,0 }; //H[12]
   //float headerB_28[8] =   { 0,1,1,0, 1,0,1,1 }; //H[28]
   //float headerB_29[8] =   { 1,1,1,1, 0,1,1,0 }; //H[29]
   //float headerB_30[8] =   { 1,1,0,0, 0,1,1,0 }; //H[30]
   //float headerB_31[8] =   { 1,0,0,0, 1,0,0,1 }; //H[31]

   //Declipse – ‘Full background data’ format. Change H[12], H[28] - H[31] 
   //float headerB_12[8] =   { 1,1,1,0, 1,1,1,1 }; //H[12]
   //float headerB_28[8] =   { 0,0,1,0, 1,1,1,1 }; //H[28]
   //float headerB_29[8] =   { 1,1,1,1, 0,0,0,0 }; //H[29]
   //float headerB_30[8] =   { 1,1,0,0, 0,1,0,0 }; //H[30]
   //float headerB_31[8] =   { 0,1,0,1, 1,1,1,1 }; //H[31]


   //Only even columns of first row
   if (fmod(column, 2.0) == 0)
   {
    int header = floor(column / 16);
    int headerPixel = fmod(column / 2, 8.0);

    if (header == 0) //Header A
    {
     col.b = headerA_0[headerPixel];
    } 
    else if (header == 1)
    {
     col.b = headerA_1[headerPixel];
    }
    else if (header == 2)
    {
     col.b = headerA_2[headerPixel];
    }
    else if (header == 3)
    {
     col.b = headerA_3[headerPixel];
    }
    else if (header == 4)
    {
     col.b = headerA_4[headerPixel];
    }
    else if (header == 5)
    {
     col.b = headerA_5[headerPixel];
    }
    else if (header == 6)
    {
     col.b = headerA_6[headerPixel];
    }
    else if (header == 7)
    {
     col.b = headerA_7[headerPixel];
    }
    else if (header == 8)
    {
     col.b = headerA_8[headerPixel];
    }
    else if (header == 9)
    {
     col.b = headerA_9[headerPixel];
    }
    else if (header == 10) //Header B
    {
     col.b = headerB_10[headerPixel];
    }
    else if (header == 11)
    {
     col.b = headerB_11[headerPixel];
    }
    else if (header == 12)
    {
     col.b = headerB_12[headerPixel];
    }
    else if (header >= 13 && header <= 27)
    {
     col.b = headerB_13_27[headerPixel];
    }
    else if (header == 28)
    {
     col.b = headerB_28[headerPixel];
    }
    else if (header == 29)
    {
     col.b = headerB_29[headerPixel];
    }
    else if (header == 30)
    {
     col.b = headerB_30[headerPixel];
    }
    else if (header == 31)
    {
     col.b = headerB_31[headerPixel];
    }

   }
  }
 }
 else
 {
  // black lines for every even line
  col = float4(0, 0, 0, 0);
 }
 // apply fog

 UNITY_APPLY_FOG(i.fogCoord, col);
 return col;
 
}

In case you want to change the headers, I would recommend to use this website to compute the checksum of CRC-32. 

For example, in the case of modifying H[0]-H[6] the checksum has to be in H[7]-H[9]. If you use the page mentioned just in the previous paragraph select the following options

  • Web: http://www.sunshine2k.de/coding/javascript/crc/crc_js.html
  • Options: 
    • CRC-32, Custom:
      • Input reflected: Uncheck; Result reflected: Uncheck.
      • Polynomial: 0x04C11DB7L; 
      • Initial Value: 0x0;
      • Final Xor Value: 0x0;
CRC Input Data (bytes): 0xF1 0x01 0x40 0x80 0x00 0x00 (change to whichever you use in the new H[0]-H[6])

Once both shaders have been created we have to apply them into the materials to be added to the canvas.

Materials:

To visualize the images in the canvas, two new materials have to be created:
  • Dimenco_color: Apply the Shader “Dimenco/2DplusDepth_color”. Select the render texture “renderTexture_color” in the Detail (RGB).
  • Dimenco_depth: Apply the Shader “Dimenco/2DplusDepth_depth”. Select the render texture “renderTexture_depth” in the Detail (RGB).
Example of the material "Dimenco_Color"

Apply the materials to the component Image of Image_left and Image_Right GameObjects in the canvas. Change the color to white if you had it with other values. 

Example of material in image component object within the canvas

Both images can be observed in each side of the display. 




These materials are applied to the image left (color) and right (depth) in the canvas and everything should be alright. Once you build the project and run it in full screen mode the images should be in 3D. If you pay close attention you will spot bright blue pixels on the top left corner. These belong to the header and is normal (its the encoding that the Dimenco monitor uses to turn the image into 3D).

Remember the monitor has to be in 4K resolution and the game in HD resolution. 

Sometimes there is an error saying “Releasing render texture that is set as Camera.targetTexture!”. If that happens it means the render texture is no longer attached to the camera. Attach it and build the project again, everthing should be alright. From my experience that only happens when you modify the render texture and build the project. Once you reattach and build it (without modifying the render texture again) the error does not happen again. 

This is my first tutorial, so I hope everything is clear and easy to follow. In case there is a mistake, or any other input, please leave a comment.

Note: I have verified the shaders' with Unity 2019.2.13f1 and they work. I do not have a Dimenco display at hand now, but it should be working there as well.

Downloads

A simplified unity asset version of this tutorial can be found here (Unity 2019.2.13). Disclaimer: This version has not been tested using the Dimenco display. 

For reference: this is the image obtained

Gravity Part 3: Create a force map and display it (CPU, Compute Shader and Fragment Shader)

Introduction This is the third and last post in the series I am creating, based on the logic from Physics-based Solar System Lessons : ...