RealityKit is Apple’s Framework for simulating and rendering 3D virtual content in Augmented Reality. But there’s a few missing pieces that get in the way of creating awesome AR experiences.
As more things become possible in Augmented Reality and ARKit, it’s vital that developers like you and I are able to create and experiment in AR with ease, trying out the latest technologies without having to start from basic building blocks every time. Currently there are a lot of missing pieces in RealityKit stopping us immediately jumping into the Augmented Reality world. The tools required to build them are there, but building them from scratch can be a real turn-off.
While building apps in RealityKit it’s easy to become frustrated that something as simple as adding a tap action on a RealityKit entity, or animating something to spin in circles each requires dozens of lines of code when it shouldn’t.
The packages listed below allow developers to do those things and more with just a few lines of code. With tools for instantly creating an AR experience that’s shared between devices, to adding a 3D button in the AR space, to being able to point at an object or position in the scene to guide the user behaviour and making sure they don’t miss out on the action. These packages take out all of that hard work and let developers get on and create awesome AR apps.
Sometimes you might want the user of your Augmented Reality Application to look in a general direction or at a specific object. Due to the user having 6 degrees of freedom, it can be very hard to tell people where they should be looking in an AR scene, and as a developer can feel like a waste of time to build something from scratch yourself.
With RKPointPin you can target an Entity for this UIView pin to point at, with easy to adjust properties such as colour of the pin, how direct the camera needs to point towards your entity before the pin disappears, and more.
FocusEntity is derived from Apple’s FocusNode, found in their SceneKit Augmented Reality example code from a few years ago. Add FocusEntity to your project to aid users when placing AR content, or for just scanning the environment.
Without some sort of visual aid, it’s hard to know that ARKit is picking up the scene, or knowing exactly where the centre of the screen is targeting in the real world. FocusEntity can give users an insight into how ARKit is working in their environment.
RealityUI is a collection of many tools and entities that will make developing with RealityKit so much more exciting. Ranging from ControlEntities, which allow users to control the environment in the AR space from the AR space, as well as additional gestures to those found already in RealityKit and some animations that are not easily achievable in RealityKit without some prodding.
RealityUI is a collection of User Interface classes for RealityKit. The classes included in RealityUI aim to offer…
If you haven’t used RealityKit much or at all, the fact that these animations are not included in RealityKit may be a surprise to you; it certainly was to me. These animations require a bit of finessing to get right on your own, but with RealityUI you can just add them directly into your Augmented Reality application.
Yes that’s right, there is no simple way to spin an entity one or many times in RealityKit, until now.
This may not be a standard animation, but I found myself needing it for an AR project I was doing, and thought others may like it too. 🎶 Shake shake shake, shake shake shake, shake your… entity.
We are not working with two dimensions like most developers are used to, but instead three. Gestures can be a pain in 3D, so there are a couple of gestures in RealityUI that aim to make it that much easier for developers to build what they want to build.
The gestures that come with RealityKit (installGestures) are great for a very simple scene where you want to reorganise furniture, but are limited in their offering. The gestures from RealityUI don’t mean you can do anything you like, but they can help you get there.
RealityKit offers a 2 finger rotate gesture, but it can be a little awkward to use two fingers on a phone while looking at something in AR.
HasTurnTouch adds a way to rotate something in any axis you like with just one finger.
Let RealityUI easily take care of the raycasting to find taps in your RealityKit scenes with
There’s not really a good GIF to show for this, we’ve all see a tap before. Check out the RealityUI wiki page about this tap gesture for more info.
The idea for these entities is to offer more ways to add controls to the 3d scene, and away from the screen space we’re used to using for 2D apps. These may become more necessary in the future with any alternative hardware (🤓), but for now can still let users interact with the AR scene without breaking the immersion.
All of these entities have natural touch controls which are (almost) automatically enabled when adding them to your scene.
See the RealityUI wiki page on Control Entities for more information on all of these.
Add a 3D toggle to your RealityKit app, to easily toggle something on or off in your scene. See more.
This 3D slider could be used to anything you want to interpolate, like the scale of another entity, or even controlling audio volume. See more.
What can a button do? Anything you like! RUIButton can be tapped to trigger anything a button can do. See more.
If you need something increase and decrease with just a tap in your RealityKit scene, try RUIStepper! See More.
Here’s an example of those Control Entities from above working together:
It’s already possible to place text in RealityKit, but I felt it needed a little upgrade.
RUIText you can easily create an Entity with the specified text placed with its bounding box centre at the middle of your entity.
This may seem like a moot thing to have in this package, as you can generate text already. But the main benefit is anchoring the text directly in the centre of the Entity, which is difficult and annoying to do if you want to add a lot of text to your RealityKit scene.
You can also add many ascii characters to the text element, to add some symbols/icons without needing to add downloads or 3d models to your app.
I’ve written in depth about MultipeerHelper before when I first created it, but essentially it’s a wrapper making the creation of multi-user AR experiences a breeze. check out the article below for a more in-depth description:
To summarise, I made this article because I’m excited to see what people will make with the Augmented Reality tools given to them. I know a few people who have been put off RealityKit due to it not being able to do nearly as much as SceneKit or Unity. I hope that these packages, and many others, can grow and turn RealityKit into something that’s even easier to make stunning AR games, tools, or experiences than it already is.
If you know of any Swift Packages that can help others create RealityKit applications, send me a message and I’ll see if they would be a good addition to this list. Or if you would like to add a feature that’s important to you, please feel free to fork any of the repositories and open a pull request!
Leave a comment or send me me a message on Twitter for any questions about these packages. But don’t forget to clap! 👏👏👏
I’ve used most of these libraries in my free app SinkAR, where you can play battleships in AR against people in the same room as you. There’s no ads or in-app purchases in this app, so try it out if you want to see an example of what you can make with RealityKit. Because of RealityKit and the above libraries, SinkAR is less than 5MB in size.