Power-Up RealityKit by Adding These Missing Features
RealityKit is Apple’s Framework for simulating and rendering 3D virtual content in Augmented Reality. But there’s a few missing pieces that get in the way of creating awesome AR experiences.
As more things become possible in Augmented Reality and ARKit, it’s vital that developers like you and I are able to create and experiment in AR with ease, trying out the latest technologies without having to start from basic building blocks every time. Currently there are a lot of missing pieces in RealityKit stopping us immediately jumping into the Augmented Reality world. The tools required to build them are there, but building them from scratch can be a real turn-off.
While building apps in RealityKit it’s easy to become frustrated that something as simple as adding a tap action on a RealityKit entity, or animating something to spin in circles each requires dozens of lines of code when it shouldn’t.
The packages listed below allow developers to do those things and more with just a few lines of code. With tools for instantly creating an AR experience that’s shared between devices, to adding a 3D button in the AR space, to being able to point at an object or position in the scene to guide the user behaviour and making sure they don’t miss out on the action. These packages take out all of that hard work and let developers get on and create awesome AR apps.
You can add all of these packages with Swift Package Collections using this URL:
https://github.com/maxxfrazer/RealityKit-Package-Collection/releases/download/main-release/RealityKit-Package-Collection.jsonThe package list is currently: RealityUI, RealityGeometries, Mirador, RKPointPin, FocusEntity and MultipeerHelper. More submissions are welcome!
FocusEntity
FocusEntity is derived from Apple’s FocusNode, found in their SceneKit Augmented Reality example code from a few years ago. Add FocusEntity to your project to aid users when placing AR content, or for just scanning the environment.
Without some sort of visual aid, it’s hard to know that ARKit is picking up the scene, or knowing exactly where the centre of the screen is targeting in the real world. FocusEntity can give users an insight into how ARKit is working in their environment.
RealityUI
RealityUI is a collection of many tools and entities that will make developing with RealityKit so much more exciting. Ranging from ControlEntities, which allow users to control the environment in the AR space from the AR space, as well as additional gestures to those found already in RealityKit and some animations that are not easily achievable in RealityKit without some prodding.
Animations
If you haven’t used RealityKit much or at all, the fact that these animations are not included in RealityKit may be a surprise to you; it certainly was to me. These animations require a bit of finessing to get right on your own, but with RealityUI you can just add them directly into your Augmented Reality application.
Spin
Yes that’s right, there is no simple way to spin an entity one or many times in RealityKit, until now.
Entity.ruiSpin allows you to spin an object for as long as you like along any axis. See the wiki page for more information.
Shake
This may not be a standard animation, but I found myself needing it for an AR project I was doing, and thought others may like it too. 🎶 Shake shake shake, shake shake shake, shake your… entity.
Set the number of shakes, period of each shake, angle and axis with Entity.ruiShake. See the wiki page for more information.
Gestures
We are not working with two dimensions like most developers are used to, but instead three. Gestures can be a pain in 3D, so there are a couple of gestures in RealityUI that aim to make it that much easier for developers to build what they want to build.
The gestures that come with RealityKit (installGestures) are great for a very simple scene where you want to reorganise a small number of objects, but are limited in their offering. The gestures from RealityUI scale much better, and give you greater control.
Turn
RealityKit offers a 2 finger rotate gesture, but it can be a little awkward to use two fingers on a phone while looking at something in AR.
HasTurnTouch adds a way to rotate something in any axis you like with just one finger.
Tap
Let RealityUI easily take care of the raycasting to find taps in your RealityKit scenes with HasClick.
There’s not really a good GIF to show for this, we’ve all see a tap before. Check out the RealityUI wiki page about this tap gesture for more info.
Control Entities
The idea for these entities is to offer more ways to add controls to the 3d scene, and away from the screen space we’re used to using for 2D apps. These may become more necessary in the future with any alternative hardware (🤓), but for now can still let users interact with the AR scene without breaking the immersion.
All of these entities have natural touch controls which are (almost) automatically enabled when adding them to your scene.
See the documentation for more information on all of these.
RUISwitch
Add a 3D toggle to your RealityKit app, to easily toggle something on or off in your scene. See more.
RUISlider
This 3D slider could be used to anything you want to interpolate, like the scale of another entity, or even controlling audio volume. See more.
RUIButton
What can a button do? Anything you like! RUIButton can be tapped to trigger anything a button can do. See more.
RUIStepper
If you need something increase and decrease with just a tap in your RealityKit scene, try RUIStepper! See More.
Here’s an example of those Control Entities from above working together:
Text
It’s already possible to place text in RealityKit, but I felt it needed a little upgrade.
With RUIText you can easily create an Entity with the specified text placed with its bounding box centre at the middle of your entity.
This may seem like a moot thing to have in this package, as you can generate text already. But the main benefit is anchoring the text directly in the centre of the Entity, which is difficult and annoying to do if you want to add a lot of text to your RealityKit scene.
You can also add many ascii characters to the text element, to add some symbols/icons without needing to add downloads or 3d models to your app.
Mirador
Mirador is a fantastic new project from the mind of Andrew Hart. With Mirador you can add stunning anchors to your 3D scene to map out the landscape around you.
So far this is just the first version of Mirador, so there will be a lot to see here in the coming months while the Vision OS betas are rolled out!
RKPointPin
Sometimes you might want the user of your Augmented Reality Application to look in a general direction or at a specific object. Due to the user having 6 degrees of freedom, it can be very hard to tell people where they should be looking in an AR scene, and as a developer can feel like a waste of time to build something from scratch yourself.
With RKPointPin you can target an Entity for this UIView pin to point at, with easy to adjust properties such as colour of the pin, how direct the camera needs to point towards your entity before the pin disappears, and more.
MultipeerHelper
I’ve written in depth about MultipeerHelper before when I first created it, but essentially it’s a wrapper making the creation of multi-user AR experiences a breeze. check out the article below for a more in-depth description:
To summarise, I made this article because I’m excited to see what people will make with the Augmented Reality tools given to them. I know a few people who have been put off RealityKit due to it not being able to do nearly as much as SceneKit or Unity. I hope that these packages, and many others, can grow and turn RealityKit into something that’s even easier to make stunning AR games, tools, or experiences than it already is.
If you know of any Swift Packages that can help others create RealityKit applications, send me a message and I’ll see if they would be a good addition to this list. Or if you would like to add a feature that’s important to you, please feel free to fork any of the repositories and open a pull request!
Leave a comment or send me me a message on Twitter for any questions about these packages. But don’t forget to clap! 👏👏👏
I’ve used most of these libraries in my free app SinkAR, where you can play battleships in AR against people in the same room as you. There’s no ads or in-app purchases in this app, so try it out if you want to see an example of what you can make with RealityKit. Because of RealityKit and the above libraries, SinkAR is less than 5MB in size.
