RealityKit Terrain Shader

This project shows you how to use Fractal Brownian Motion (fBM) to generate an Augmented Reality (AR) Terrain using iOS RealityKit Custom Materials. RealityKit Custom Material includes a surface shader and geometry modifier, also known as fragment and vertex shaders in OpenGL.

RealityKitTerrainShader.zip

Fractal Brownian Motion (fBM)

fBM (Fractal Brownian Motion) is used extensively in computer graphics to model natural phenomenons and textures such as terrain , coastline, fire, smoke, clouds, water flow, water waves , rapids, dinosaur skin, wrinkles etc. It is quite amazing to note how such a simple random function can be used in so many areas to emulate the various forms of nature.

This project shows how you can use the fBM function to very simply generate a terrain using iOS RealityKit custom materials (which includes the surface shader and geometry modifier, also known as fragment and vertex shaders in OpenGL)

Parameters

The form of fBM function we are using has mainly the parameters

uv - texture coordinates (a value of [0,0] to [1,1], representing the four corners of an image we will like to generate)
time - in seconds (this value will animate the fBM generate image with respect to time)

Properties of the fBM

A fBM can return a single value (one dimensional value) or multiple values (e.g a 3 dimensional vector). You can view a fBM as a random number generator that depends on the current pixel position of the image. However, it is not completely random because the function varies quite smoothly with respect to its neighbours.

So if the user pass in uv-coordinate values of (0.10,0.10) and (0.11,0.10) the returned value should be quite near to each other. For a 3 dimensional vector , the value of each component will be quite similar to its neighbour’s.

Apart from varying smoothly with respect to space (i.e position or uv-coordinates), the fBM also varies quite smoothly with respect to time. So a returned value at time = 0.10 sec will be quite similar to a value at time = 0.11 sec. This gives it the property of having a smooth animation w.r.t. time.

Furthermore, the continuity of many implementations fBM is not restricted to just continuity of values, but also the continuity of its gradient as well as the continuity of (radius of) curvature. This make the fBM function very ‘smooth’ and thus very suitable for artistic renderings.

A fBM has fractal like properties. If you zoom in to small area, you will be able to notice similar patterns when the image has not been zoomed in.

Generating the Terrain

This article is mainly about using fBM to generate a terrain with RealityKit 2. The language is in Swift and Metal.

The steps involved is to first generate a 2D plane with 64x64 cells.

                               
    let plane = buildMesh(numCells: [64, 64], cellSize: 0.01)

Assign the plane with a Custom Material. This custom material will automatically associate a Surface Shader function and a Geometry Modifier function to the plane

                               
    let geometryModifier = CustomMaterial.GeometryModifier(named: "geometryModifier",
                                                            in: library)
                          
    let surfaceShader = CustomMaterial.SurfaceShader(named: "surfaceShader",
                                                        in: library)

    let customMaterials = try! CustomMaterial(from: baseMaterial,
                                                surfaceShader : surfaceShader,
                                                geometryModifier: geometryModifier)

    let plane = buildMesh(numCells: [64, 64], cellSize: 0.01)
                                      
    landEntity = ModelEntity(mesh: plane, materials : [customMaterials])

The geometryModifier and surfaceShader is currently empty functions written in the Metal language. Running the app shows an empty plane.

Add normals to the plane using the computeNormal function. At this time, we will like to introduce the fBMheightmapx function. This is the main fBM function used for generating a height map for the terrain. In normal situations, we will simply render this height map and we will get a terrain. However, for RealityKit surface shader, it is easier to add details (roughness) to the terrain using normals. This is achieved by computing normals of the height map with the computeNormal function in the surfaceShader function.

    
    //fBMheightmapx  - generates a height map at each position (uv position) of the shader.
    //This height-field has a normal at each uv coordinate position pointing away perpendicularly from the surface.
    //The normals is computed by offsetting the height-map a little in the x-direction and a little in the y-direction to compute the difference.
    //The normals computed thus does not necessary has magnitude / length of 1 (and need to be normalised to length 1).

    float3 normal = computeNormal(uvscale,time*timefactor/divfactor ,warpType,animVelocity,angle,screensize);
                            
    normal = normalize(normal);
                          
    params.surface().set_normal(normal); //set tangent space normal

With the normals set, we now run the app and we get something that look like below.

We now elevate the plane vertically at the various vertices of each cell randomly to produce a an uneven surface. The random values (elevation levels) generated for this terrain is produced directly using the fBM function we mentioned earlier (fBMheightmapx). To elevate the plane, we have to use a geometry modifier (also known as vertex shader) to modify the position of each of these vertices. To do so, we define a geometryModifier function with the following code.

                               
    float4 val =
                fBMheightmapx(uvscale,time ,warpType,animVelocity,angle);

    //here , we are preparing the values for a displacement map
    //so that only the y-value (height) is modified, but not the x and z value
    //the value 0.063 is chosen by trial and error
    float3 offset = float3(0.0,0.063*val.r,0.0);
                         
    //this function applies the fBM generated offset to the plane to make it uneven
    params.geometry().set_model_position_offset(offset);

We now get something that looks like below. The terrain is almost complete.

For the final step, wee proceed to color the terrain. The fBM function we mentioned earlier can do many things, apart from producing a height map (and its associated normals), we can also use it to color the terrain by mapping each height value with a color.

For our case, we use a modified extended version of the fBM function named fBMFlow. This function applies a color palette to the fBM generated height-map and retrieve the color at the current uv position. This color is then used to colorize the terrain.
```
                               
    float4 val = fBMFlow(uv, float2(2048,2048), float3(0,0,0), time*timefactor);
                                     
    half3 color1 = half3(halfr,halfg,halfb);

    params.surface().set_base_color(color1);
                           
```
With this done, we will get the terrain as shown below. (The image has been enhanced a little for better visibility).