Video Textures, or VideoMaterials, are a new feature in Apple’s RealityKit this year (WWDC 2020), which enables you to add videos on-top of any mesh, in place of an otherwise static material. VideoMaterial uses an AVPlayer to play the video, which means it has the support of a matured class within the AVFoundation library.
To learn how to do more with RealityKit, see my on-going series here:
Examples
Streaming a local or remote video using AVPlayer
Creating a VideoMaterial and applying it is incredibly easy, here’s the code sample from Apple, altered slightly:
// Use AVFoundation to load a video in the bundle, named glow.mp4
let asset = AVURLAsset(
url: Bundle.main.url(forResource: "glow", withExtension: "mp4")!
)
let playerItem = AVPlayerItem(asset: asset)// Create a Material and assign it to your model entity
let player = AVPlayer()
myModelEntity.materials = [VideoMaterial(player: player)]// Tell the player to load and play
player.replaceCurrentItem(with: playerItem)
player.play()
The above sample uses a local file found within the Bundle, but you can also stream videos from a URL. Here’s an example of how to create the relevant AVPlayerItem which streams the main WWDC20 keynote:
// Not sure whether this URL will be valid forever…
let streamingURLString = "https://p-events-delivery.akamaized.net/2605bdtgclbnfypwzfkzdsupvcyzhhbx/m3u8/hls_vod_mvp.m3u8"guard let url = URL(string: streamingURLString) else {
return
}
let asset = AVURLAsset(url: url)
let playerItem = AVPlayerItem(asset: asset)// Create a Material and assign it to your model entity
let player = AVPlayer()
myModelEntity.materials = [VideoMaterial(avPlayer: player)]// Tell the player to load and play
player.replaceCurrentItem(with: playerItem)
player.play()
The above will make something that streams the WWDC20 Keynote like this:
In my tests, I’ve found no problem when applying an AVPlayer created in a slightly simpler way, such as this:
let avPlayer = AVPlayer(url: url)
let vidMat = VideoMaterial(avPlayer: avPlayer)
myModelEntity.materials = [vidMat]
avPlayer.play()But if anything goes wrong, revert back to the structure that Apple provided!
Looping Playback 🔁
You may frequently want to play a looping video on a VideoMaterial, in which case the best way would be to use an AVPlayerLooper. The alternative could be adding notifications to your AVPlayer, which I’ll leave to you to search if you wish.
Creating an AVPlayerLooper is best described on the Official Documentation page for it, but I’ll give an example too where it is used to create a VideoMaterial:
let avPlayerItem = AVPlayerItem(asset: AVAsset(url: url))
let queuePlayer = AVQueuePlayer()
self.avPlayerLooper = AVPlayerLooper(
player: queuePlayer,
templateItem: avPlayerItem
)myModelEntity.materials = [VideoMaterial(avPlayer: queuePlayer)]
queuePlayer.play()
An important thing to note here is that you must keep a reference of the AVPlayerLooper, otherwise it may not work all once it goes out of scope.
Audio Control
The only other thing to note for now is that you can control the spatial audio settings. There are three options, .nonSpatial, .spatial, and .ambient. With the default being .spatial, meaning that both the camera’s position and orientation are considered in the audio playback.
That’s it for VideoMaterials. I’ve seen people asking about GIFs on VideoMaterials; but actually that might not be the best idea, as video files like mp4 are usually a much smaller size, and we like that. This is why Gfycat does that as standard #randomfact.
There’s also a small bug with AVPlayerLooper in VideoMaterials at the moment, hopefully that is fixed before the end of the beta cycle this year before iOS 14 is released!
Leave a couple of 👏 below if you’ve found this helpful (or to save faries, that works too). Comment below with any thoughts, and send me a tweet if something isn’t working quite right for you, it may be an issue I’ve had previously. I’ll be posting more on here and plan to release some related apps over once iOS 14.0 is public for all.
