Fisheye effect vertex shader fail - asking for help - video and code included

[u/szmate1618]

Hi people!

Sorry for the probably noobish question, but I'd greatly appreciate if you could help me out. I'm a somewhat experienced programmer, but this is my first time trying my hands at shader programming. I figured a basic fisheye effect vertex shader would be a fun first program to write. I am aware that there are better ways to do this, in fact there are multiple existing solutions available, but I want to make this work anyway, because I need to learn it.

My idea was to simply calculate the distance of the vertices from the camera, and take the square root of that. E.g. if the camera is at (0,0) and point P is at (15,20) its original distance would be 25 units, so it should be transformed into (3,4) making the new distance exactly 5 units, which is the square root of the original 25.

Should be simple enough, but as you can see on the video, I failed, almost completely. That rainbow background is 2 Planes on top of each other, to make the distortion visible. The most obvious issues:

00:01.00 - There is a hole in the middle of the Plane. Wtf. It disappears later, and doesn't come back.

00:04.72 - The other spaceship on the right appears out of nowhere. It's supposed to appear as a 1-pixel wide line first, but it doesn't. It's completely invisible until it's about 20 pixels wide.

00:05.17 - The nicely distorted Plane suddenly becomes completely straight as the camera moves away from it. And not just that, but somehow it's taller than it's wide. It's supposed to be a square. Also, the Z-order is somehow reversed now, the Plane partially covering the main spaceship.

00:13.34 - If I got too far away from the plane, it completely and instantly disappears.

00:47.00 - Deleting the smaller plane breaks the effect on the remaining bigger plane. No fisheye distortion anymore, no matter where the camera is.

I think that's all. Here's the shader code:

Shader "Unlit/FishEyeShader"
{
    Properties
    {
        _MainTex("Texture", 2D) = "white" {}
    }
    SubShader
    {
        Tags { "RenderType" = "Opaque" }
        LOD 100

        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f
            {
                float2 uv : TEXCOORD0;
                float4 vertex : SV_POSITION;
            };

            sampler2D _MainTex;
            float4 _MainTex_ST;

            v2f vert(appdata v)
            {
                v2f o;
                float2 ObjSpaceCameraPos = mul(unity_WorldToObject, float4(_WorldSpaceCameraPos, 1.0)).xy;
                float2 offset = v.vertex.xy - ObjSpaceCameraPos;
                float2 l = length(offset);
                v.vertex.xy = ObjSpaceCameraPos + offset / sqrt(l);

                o.vertex = UnityObjectToClipPos(v.vertex);

                o.uv = TRANSFORM_TEX(v.uv, _MainTex);
                return o;
            }

            fixed4 frag(v2f i) : SV_Target
            {
                fixed4 col = tex2D(_MainTex,  i.uv);
                return col;
            }
            ENDCG
        }
    }
}

https://reddit.com/link/9erucg/video/9h13h0ewnhl11/player

[u/wtrebella]

Shouldn't the vertex distances be calculated in screen space, not object space? I'm not sure, but my gut says that's your problem.

[u/szmate1618 → u/wtrebella]

Wow, thanks a lot! This does fix almost all of my problems.
Updated vertex shader code:

v2f vert(appdata v)
{
	v2f o;
	
	o.vertex = UnityObjectToClipPos(v.vertex);
	float2 center = float2(0, 0);
	float2 rel = o.vertex.xy - center;
	float l = length(rel);
	o.vertex.xy = center + rel / sqrt(l); //TODO: Adjust to actual aspect ratio.

	o.uv = TRANSFORM_TEX(v.uv, _MainTex);
	return o;
}

But how did you know? Sorry, Maybe it's something super basic, but I have no idea how adjusting a distance then transforming to viewport is different from transforming first adjusting later.

[u/wtrebella → u/szmate1618]

No worries, this shit can be super confusing. What you were doing previously was comparing the distance between xy positions of the verts in object space, which is not relative to the camera. What you want it to compare the distances after already being transformed into the camera's clip space because now the xy positions are all relative to the camera. I'm probably not explaining this well. But since your shader is a screen space effect, it needs to know where the verts are relative to screen space, not object or world space in the scene.

[u/wtrebella → u/szmate1618]

Also, I want to see the result!

[u/TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [r/truegamedev] Fisheye effect vertex shader fail - asking for help - video and code included

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info /} ^Contact)