Getting Started with RealityKit: Video Materials

Stream a VideoMaterial from a local or remote file + looping

TV and RealityKit logos
TV Model from apple.com

Examples

Streaming a local or remote video using AVPlayer

Animated video of an excited cat on a 3D Squircle
Animated video of an excited cat on a 3D Squircle
Streaming a Tom & Jerry URL from Gfycat 😻
// Use AVFoundation to load a video in the bundle, named glow.mp4
let asset = AVURLAsset(
url: Bundle.main.url(
forResource: "glow", withExtension: "mp4")!
)
let playerItem = AVPlayerItem(asset: asset)
// Create a Material and assign it to your model entity
let player = AVPlayer()
myModelEntity.materials = [VideoMaterial(player: player)]
// Tell the player to load and play
player.replaceCurrentItem(with: playerItem)
player.play()
// Not sure whether this URL will be valid forever…
let streamingURLString = "https://p-events-delivery.akamaized.net/2605bdtgclbnfypwzfkzdsupvcyzhhbx/m3u8/hls_vod_mvp.m3u8"
guard let url = URL(string: streamingURLString) else {
return
}
let asset = AVURLAsset(url: url)
let playerItem = AVPlayerItem(asset: asset)
// Create a Material and assign it to your model entity
let player = AVPlayer()
myModelEntity.materials = [VideoMaterial(avPlayer: player)]
// Tell the player to load and play
player.replaceCurrentItem(with: playerItem)
player.play()
let avPlayer = AVPlayer(url: url)
let vidMat = VideoMaterial(avPlayer: avPlayer)
myModelEntity.materials = [vidMat]
avPlayer.play()

Looping Playback 🔁

You may frequently want to play a looping video on a VideoMaterial, in which case the best way would be to use an AVPlayerLooper. The alternative could be adding notifications to your AVPlayer, which I’ll leave to you to search if you wish.

let avPlayerItem = AVPlayerItem(asset: AVAsset(url: url))
let queuePlayer = AVQueuePlayer()
self.avPlayerLooper = AVPlayerLooper(
player: queuePlayer,
templateItem: avPlayerItem
)
myModelEntity.materials = [VideoMaterial(avPlayer: queuePlayer)]
queuePlayer.play()

Audio Control

The only other thing to note for now is that you can control the spatial audio settings. There are three options, .nonSpatial, .spatial, and .ambient. With the default being .spatial, meaning that both the camera’s position and orientation are considered in the audio playback.

Excited about all things Augmented Reality! Developer Evangelist at Agora.io.