Chapters
Chapters

Chapters

Hide chapters

Metal by Tutorials

Second Edition · iOS 13 · Swift 5.1 · Xcode 11

Before You Begin

Section 0: 3 chapters
Show chapters Hide chapters

Section I: The Player

Section 1: 8 chapters
Show chapters Hide chapters

Section II: The Scene

Section 2: 7 chapters
Show chapters Hide chapters

Section III: The Effects

Section 3: 10 chapters
Show chapters Hide chapters

22. Integrating with SpriteKit & SceneKit
Written by Caroline Begbie & Marius Horga

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Now that you have mastery over rendering, you can put your knowledge to use in other APIs.

SceneKit, SpriteKit and Core Image all have some integration with Metal shaders and Metal rendering.

There may be times when you don’t want to write a full-blown 3D Metal app, but you want to take advantage of SceneKit. Or perhaps all you want is a 2D layer in your Metal app — for that, you can use SpriteKit. And even though you have less control over your final output with SceneKit and SpriteKit, you can still incorporate shaders to give your games a unique look.

In this chapter, you’ll have a look at a collection of APIs that integrate with Metal. You’ll first create a toon outline and cel shader for the jet in Apple’s SceneKit template.

After that, you’ll open a scene similar to the game scene from Chapter 9, “Scene Graph.” and add to it a 2D overlay that feeds information to the player.

By creating this overlay, you’ll learn how to render a SpriteKit scene during each Metal frame rendering.

Finally, you’ll create a Core Image Metal kernel to do the blending of the SpriteKit overlay and the 3D rendered scene.

SceneKit starter project

Before creating the toon shader, you’ll first learn how to create your own custom shaders in SceneKit, and then find out how to pass data to the shaders.

Create a new project using the macOS Game template (or you can choose the iOS Game template if you prefer). Use the Product Name of Toon-SceneKit and make sure that SceneKit is specified in the Game Technology dropdown.

Note: Alternatively, you can choose to open the starter project for this chapter instead.

Build and run, and you’ll see Apple’s default game template jet animating on the screen. You can turn the jet by dragging.

This chapter assumes you have some familiarity with SceneKit, but if not, you can get the general idea of how SceneKit works by reading through GameViewController.swift, which has extensive comments.

Just like the game engine you’ve worked on throughout this book, each element is a node. You have geometry nodes such as the jet, light nodes and a camera node. These nodes are placed into a scene, and you attach the scene to a view. To animate the nodes, you run actions on them.

Open GameViewController.swift, and in viewDidLoad(), replace the rotation animation:

// animate the 3d object
ship.runAction(
  SCNAction.repeatForever(SCNAction.rotateBy(x: 0, y: 2, z: 0, 
                          duration: 1)))

With:

ship.eulerAngles = SCNVector3(1, 0.7, 0.9)

This gives you a better angle to see your toon shading.

Next, configure a different background color. Change:

scnView.backgroundColor = NSColor.black

To:

scene.background.contents = nil
scnView.backgroundColor = NSColor(calibratedWhite: 0.9, 
                                  alpha: 1.0)

This removes the procedural sky that ship.scn created, and replaces it with a light gray background.

SceneKit shaders

To run a Metal shader on a node, you create an SCNProgram object and attach it to the node.

let program = SCNProgram()
program.vertexFunctionName = "shipVertex"
program.fragmentFunctionName = "shipFragment"

if let material = ship.childNodes[0].geometry?.firstMaterial {
  material.program = program
}

#include <SceneKit/scn_metal>

struct VertexIn {
  float4 position [[attribute(SCNVertexSemanticPosition)]];
};

struct Uniforms {
  float4x4 modelViewProjectionTransform;
};

struct VertexOut {
  float4 position [[position]];
};

vertex VertexOut shipVertex(VertexIn in [[stage_in]],
                    constant Uniforms& uniforms [[buffer(1)]]) {
  VertexOut out;
  out.position = 
      uniforms.modelViewProjectionTransform * in.position;
  return out;
}

fragment half4 shipFragment(VertexOut in [[stage_in]]) {
  return half4(1, 0, 0, 1);
}

Match SceneKit names

When you’re using Metal with SceneKit, you’ll find that parameter names must match what SceneKit is expecting. Here, in the vertex function, SceneKit expects the Uniforms buffer to be named scn_node. When you give a parameter the name scn_node, you can name the struct Uniforms any name, and give the buffer index any index, and SceneKit will recognize the buffer as being the uniform values.

Send a texture to the fragment shader

In Shaders.metal, add these attributes to VertexIn:

float3 normal [[attribute(SCNVertexSemanticNormal)]];
float2 uv [[attribute(SCNVertexSemanticTexcoord0)]];

float2 uv;

out.uv = in.uv;

texture2d<float> baseColorTexture [[texture(0)]]

constexpr sampler s(filter::linear);
float4 baseColor = baseColorTexture.sample(s, in.uv);
return half4(baseColor);

material.program = program

if let url = 
    Bundle.main.url(forResource: "art.scnassets/texture",
                    withExtension: "png") {
  if let texture = NSImage(contentsOf: url) {
    material.setValue(SCNMaterialProperty(contents: texture),
                                  forKey: "baseColorTexture")
  }
}

Sending constant data to shaders

You’ve rendered a textured, unlit model, but if you want to do lighting, then you have to set it up yourself, with the usual light positions, model normals and lighting calculations.

let lightPosition = lightNode.position
material.setValue(lightPosition, forKey: "lightPosition")

float4x4 normalTransform;
float4x4 modelViewTransform;

constant SCNSceneBuffer& scn_frame [[buffer(0)]],
constant float3& lightPosition [[buffer(2)]]

float3 normal;
float4 viewLightPosition;
float4 viewPosition;

out.normal = 
    (scn_node.normalTransform * float4(in.normal, 0)).xyz;
out.viewLightPosition = 
    scn_frame.viewTransform * float4(lightPosition, 1);
out.viewPosition = scn_node.modelViewTransform * in.position;

float3 lightDirection = 
    (normalize(in.viewLightPosition - in.viewPosition)).xyz;
float diffuseIntensity = 
    saturate(dot(normalize(in.normal), lightDirection));
baseColor *= diffuseIntensity;

Toon shading

A full toon shader consists of an edge detection algorithm and a cel shader where you reduce the color palette. Generally, shading is a gradient from light to dark, but the shading in this rocket demonstrates cel shading where abrupt steps in the gradient occur:

The fwidth function

In fragment shaders, you have access to the current fragment, but you also have access to the change in slope of the fragment from neighboring fragments. Fragments pass through the rasterizer in a 2 x 2 arrangement, and the partial derivatives of each fragment can be derived from the other fragment in the group of four.

float3 v = normalize(float3(0, 0, 10)); // camera position
float3 n = normalize(in.normal);
return fwidth(dot(v, n));


float edge = step(fwidth(dot(v, n)) * 10.0, 0.4);
return edge;

Cel shading

When you render non-photorealistic toons, you generally use a minimal color range. Instead of having a smooth gradient for shading, you use a stepped flat color gradient.

if (edge < 1.0) {
  return edge;
}
float3 l = 
   (normalize(in.viewLightPosition - in.viewPosition)).xyz;
float diffuseIntensity = saturate(dot(n, l));
float i = diffuseIntensity * 10.0;
i = floor(i) - fmod(floor(i), 2);
i *= 0.1;
half4 color = half4(0, 1, 1, 1);
return color * i;


return color * pow(i, 4) * 4;


float specular = pow(max(0.0, dot(reflect(-l, n), v)), 5.0);
if (specular > 0.5) {
  return 1.0;
}

SpriteKit rendering in Metal

Note: As of the time of writing, CIContext.render hangs the app when using Xcode 11 or macOS Catalina 10.15. However, you can still do the rest of this chapter using Xcode 10 on macOS Mojave 10.14.

Create the HUD in SpriteKit

Create a new SpriteKit scene using the SpriteKit Scene template, and name it Hud.sks.

import SpriteKit

class Hud: SKScene {
  private var labelCount: SKLabelNode?
  private var label: SKLabelNode?
  
  override func sceneDidLoad() {
    label = childNode(withName: "//label") as? SKLabelNode
    labelCount = childNode(withName: "//count") as? SKLabelNode
  }
}

override func update(_ currentTime: TimeInterval) {
  print("updating HUD")
}

override var size: CGSize {
  didSet {
    guard let label = label,
      let labelCount = labelCount else { return }
    label.horizontalAlignmentMode = .left
    labelCount.horizontalAlignmentMode = .left
    let topLeft = 
        CGPoint(x: -size.width * 0.5, y: size.height * 0.5)
    let margin: CGFloat = 10
    label.position.x = topLeft.x + margin
    label.position.y = topLeft.y - label.frame.height - margin
    labelCount.position = label.position
    labelCount.position.x += label.frame.width + margin
  }
}

import SpriteKit

class HudNode: Node {
  let skScene: Hud
  
  init(name: String, size: CGSize) {
    guard let skScene = SKScene(fileNamed: name) as? Hud
      else {
        fatalError("No scene found")
    }
    self.skScene = skScene
    super.init()
    sceneSizeWillChange(to: size)
    self.name = name
  }
  
  func sceneSizeWillChange(to size: CGSize) {
    skScene.isPaused = false
    skScene.size = size
  }
}

var hud: HudNode!

hud = HudNode(name: "Hud", size: sceneSize)
add(node: hud, render: false)

hud.sceneSizeWillChange(to: size)

SKRenderer

Generally, when you create a SpriteKit app, you hook up the SKScene with an SKView. The SKView takes care of all the rendering and places the scene onto the view. SKRenderer takes the place of SKView, allowing you to control updating and rendering of your SpriteKit scene.

let skRenderer: SKRenderer
let renderPass: RenderPass

skRenderer = SKRenderer(device: Renderer.device)
skRenderer.scene = skScene
renderPass = RenderPass(name: name, size: size)

renderPass.updateTextures(size: size)

override func update(deltaTime: Float) {
  skRenderer.update(atTime: CACurrentMediaTime())
  guard let commandBuffer = Renderer.commandBuffer else {
    return
  }
  let viewPort = CGRect(origin: .zero, size: skScene.size)
  skRenderer.render(withViewport: viewPort,
                    commandBuffer: commandBuffer,
                    renderPassDescriptor: renderPass.descriptor)
}

Post-processing

Create a new Swift file named PostProcess.swift, and replace the code with:

import MetalKit

protocol PostProcess {
  func postProcess(inputTexture: MTLTexture)
}

var postProcessNodes: [PostProcess] {
  return allNodes.compactMap { $0 as? PostProcess }
}

scene.postProcessNodes.forEach { node in
  node.postProcess(inputTexture: drawable.texture)
}

extension HudNode: PostProcess {
  func postProcess(inputTexture: MTLTexture) {
    print("post processing")
  }
}

Core Image

So far in this chapter, you’ve used Metal with SpriteKit and SceneKit. There’s one other framework whose shaders you can replace with your own custom Metal shaders: Core Image.

guard let commandBuffer = Renderer.commandBuffer else { return }
let drawableImage = CIImage(mtlTexture: inputTexture)!

let filter = CIFilter(name: "CIComicEffect")!
filter.setValue(drawableImage, forKey: kCIInputImageKey)
let outputImage = filter.outputImage!

var outputTexture: MTLTexture

outputTexture = RenderPass.buildTexture(size: size,
  label: "output texture",
  pixelFormat: renderPass.texture.pixelFormat,
  usage: [.shaderWrite])

if inputTexture.width != outputTexture.width ||
  inputTexture.height != outputTexture.height {
  let size = CGSize(width: inputTexture.width,
                    height: inputTexture.height)
  outputTexture = RenderPass.buildTexture(size: size,
                    label: "output texture",
                    pixelFormat: renderPass.texture.pixelFormat,
                    usage: [.shaderWrite])
}

let context = CIContext(mtlDevice: Renderer.device)
let colorSpace = CGColorSpaceCreateDeviceRGB()
context.render(outputImage, to: outputTexture, 
               commandBuffer: commandBuffer,
               bounds: outputImage.extent, 
               colorSpace: colorSpace)

let blitEncoder = commandBuffer.makeBlitCommandEncoder()!
let origin = MTLOrigin(x: 0, y: 0, z: 0)
let size = MTLSize(width: inputTexture.width, 
                   height: inputTexture.height, 
                   depth: 1)
blitEncoder.copy(from: outputTexture, sourceSlice: 0, 
                 sourceLevel: 0,
                 sourceOrigin: origin, sourceSize: size,
                 to: inputTexture, destinationSlice: 0,
                 destinationLevel: 0, destinationOrigin: origin)
blitEncoder.endEncoding()

frameBufferOnly texture not supported for compute

metalView.framebufferOnly = false


let hudImage = CIImage(mtlTexture: renderPass.texture)!
let drawableImage = CIImage(mtlTexture: inputTexture)!

let filter = CIFilter(name: "CISourceOverCompositing")!
filter.setValue(drawableImage, forKey: kCIInputBackgroundImageKey)
filter.setValue(hudImage, forKey: kCIInputImageKey)

Core Image Metal kernels

There are a few pre-defined kernel types you can use with Core Image:

Create a Metal library for Core Image

Create a new file using the Metal template named CIBlend.metal. Include the Core Image headers:

#include <CoreImage/CoreImage.h>

extern "C" { namespace coreimage {

}}

Pre-multiplied alpha blending

In this example, the HUD texture has anti-aliasing, where the texture is partially transparent around the edge of the letters:

destination.rgb = 
    source.rgb + (destination.rgb * (1 - source.a)) 

float4 hudBlend(sample_t hudTexture, sample_t drawableTexture) {
  float4 color = 
      (1 - hudTexture.a) * drawableTexture + hudTexture;
  color = float4(srgb_to_linear(color.rgb), 1);
  return color;
}

Compiling and linking a Core Image kernel

You can build the kernel to check that it has no syntactical errors, however, because it’s wrapped up in the Core Image namespace, it won’t be available at runtime from your default Metal library.

xcrun -sdk macosx metal -c -fcikernel CIBlend.metal -o CIBlend.air
xcrun -sdk macosx metallib -cikernel CIBlend.air -o CIBlend.metallib

xcrun -sdk iphoneos metal -c -fcikernel CIBlend.metal -o CIBlend.air
xcrun -sdk iphoneos metallib -cikernel CIBlend.air -o CIBlend.metallib


let kernel: CIBlendKernel

// 1
let url = Bundle.main.url(forResource: "CIBlend", 
                          withExtension: "metallib")!
do {
  // 2
  let data = try Data(contentsOf: url)
  // 3
  kernel = try CIBlendKernel(functionName: "hudBlend", 
                             fromMetalLibraryData: data)
} catch {
  fatalError("Kernel not found")
}

let outputImage = filter.outputImage!

let hudImage = CIImage(mtlTexture: renderPass.texture)!
let extent = hudImage.extent
let arguments = [hudImage, drawableImage]
let outputImage = kernel.apply(extent: extent, 
                               arguments: arguments)!

float4 hudBlend(sample_t hudTexture, sample_t drawableTexture)

Challenge

Currently, you’re printing out “updating HUD” in the debug console. This print comes from update(_:) in the SpriteKit scene class Hud. Your challenge is to update the HUD, keeping track of the oil cans that the player has collected. To achieve this, in GameScene, on colliding with an oil can, you’ll tell HudNode to update Hud using oilcanCount. As always, you’ll find a solution in the challenge directory for this project.

Where to go from here?

In this chapter, you created two simple examples that you can experiment with further. Fragment shaders, and to a lesser extent vertex shaders, are endlessly fascinating.

21. Metal Performance Shaders
23. Debugging & Profiling
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2024 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now