Getting started with RealityKit: Record + Replay AR Sessions

Part 5: Stay sat down when creating your AR apps

There’s an amazing, semi-hidden tool for developing AR apps that has been released by Apple this year. originally thought this was just something for use within Reality Composer, since Xcode 11 you can record actions and camera feed to replay in any ARKit app. This includes SceneKit, Metal, SpriteKit, Unity, and most relevant for this article; RealityKit. Personally I haven’t used Reality Composer much, but a fellow AR enthusiast here in London showed me how to use this tool a little while ago.

This would have been particularly useful when I was previously working on AR Navigation solutions, but I wouldn’t have had so much exercise if this were a feature back then!

Recording the Session

To get started, open up Reality Composer and launch either an existing project or create a new one. From there tap the type of Anchor you want to use. If you want to use the recorded session for a scene which initially looks for a horizontal plane, choose that, otherwise you can do the same for a vertical plane, image, face, or even a scanned object.

In summary, the steps are:

  1. Open Reality Composer.
  2. Open existing project or create a new one.
  3. Tap the 3 dots menu, then the </> Developer button.
  4. Tap Record AR Session.
  5. Record the session and then share to your computer.

The GIF below goes through the above steps on an iPad:

While recording the session, if you tapped on the Horizontal option when creating a new project, you’ll see some helpful boxes appearing on the floor in-front of you; these boxes will let you know how ARKit is doing with detecting the horizontal surface. This works the same way for vertical surfaces. Here’s an example:

Recording an AR Session using Horizontal anchoring

When this recording is replayed it will essentially be paused on your device, but the scene’s entities will still animate and be able to be interacted with, with some limitations which I’m still to uncover completely.

Once the video is recorded, hit the share button and send it to your computer by whatever means work best for you and open up your ARKit project using Xcode 11.

Replaying the Session

Next edit the scheme, the steps to do so are:

  1. Edit Scheme…
  2. Run > ARKit > Hit the checkbox
  3. Locate your file using the drop down menu
  4. Then load the app to your device.

here’s a GIF of how to do so.

Attaching the video to the run scheme for your device

Next when you run your app on your device it will run as though you’re scanning and moving around the same environment as you’re recording.

I have saved a few horizontal AR Sessions which I plan on re-using for multiple projects. This feature is part of Xcode 11, so is still in beta at the time of writing this, therefore you may encounter some minor issues when using this. But even if it doesn’t run perfectly every time it’s still significantly speeding up AR development for myself.

One Step Further

If your app doesn’t require any user input, and you don’t even want to look down at your phone you could mirror your screen using QuickTime. There are a ton of articles on how to do this, such as this one:

The result looks like this:

View your AR App using QuickTime

I’m expecting Apple to let this work with the simulator at some point, but until that happens this is the next best thing in my opinion.

Thanks for reading, and don’t forget to 👏 if you’ve enjoyed this article, and stay tuned for more.

--

--

--

Excited about all things Augmented Reality! Developer Evangelist at Agora.io.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Apple: No Mac-In-Touch

Blinded by the Bite of Apple’s SoC

Microsofts standing to sue over secret US data requests in question

You’re going to love this.

Oppo A53s 5G comes with MediaTek Dimensity 700 5G SoC with a 2.2 GHz frequency

Oppo A53s 5G comes with MediaTek Dimensity 700 5G SoC with a 2.2 GHz frequency

Smiling through Pandemic, Let’s flip it around, what media don’t want us to show !

How 5G will change the augmented reality devices and services

What is Virtual Reality, technology for the next reality?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Max Cobb

Max Cobb

Excited about all things Augmented Reality! Developer Evangelist at Agora.io.

More from Medium

Firebase Authentication for Swift

iOS simple painting App — demonstration in Swift, SwiftUI and RxSwift

Simple painting App preview

Classifying Sounds with the SoundAnalysis framework and custom CoreML models in Swift

Protocol and protocol inheritance in swift