Having multiple people simultaneously seeing the same content in Augmented Reality is huge, and it’s now not only possible but very easily achievable with RealityKit.
Networking Permissions Update (iOS 14)
Since iOS 14, you need to specify that your app needs to be able to make advertising services and browse for other services on the network.
To do so, you must add NSLocalNetworkUsageDescription, along with a short description to your Info.plist; similarly than is done for Camera Usage.
You then must also add NSBonjourServices. This creates an array of services which can be created by your app. The string(s) included in this array match the serviceType mentioned in the service initialisers, preceded with an underscore, and followed with "._tcp". Below in the example we create a service named helper-test, so in the Info.plist we should add _helper-test._tcp. An example of this can be found in my repository example project here:
Intro to Browsing and Advertising MCSessions
The core functionality of connecting iOS devices starts at devices either browsing for hosts or advertising a session. For browsing you can either use the MCNearbyServiceBrowser class, which uses a delegate to decide whether to connect to any given peer. Alternatively, you can use MCBrowserViewController which displays a view where the user can choose from a list of peers.
For advertising a session you can use MCNearbyServiceAdvertiser which also uses a delegate with pre-set rules to decide whether to let others connect to the host’s session. Alternatively, you can use MCAdvertiserAssistant, which you can use to display UI pop-ups asking the host if they want to allow a user to join their session. Since I’m using this in the context of an Augmented Reality app, I would opt for the delegate options for both; otherwise we would break the AR experience. Here’s an example of how the advertiser functionality may look:
It is the browser’s job to look around for advertisers of sessions you can join and then inviting the advertiser to connect with them, and the advertiser handles those invitations to then connect the devices to the same MCSession. This connection works over wifi and bluetooth, optimizing itself the best way to do so in a similar way to Apple’s AirDrop protocol. Here’s a short example of how setting up the browser functionality might look:
From this point, the MCSession will manage the communication between all connected devices. Apple has provided a downloadable project which does exactly the above within the context of RealityKit. It uses an MCSession with RealityKit’s collaborative session API only. Therefore it does not use RealityKit’s synchronization, which is very straightforward to implement from there.
The collaborative session example above uses ARKit’s method session(_:didOutputCollaborationData:) to keep track of all the ARAnchors in the scene. The data is manually sent to all the peers, and when the peers receive it they use the update(with:) method to update the ARSession. The data in these examples is of a type ARSession.CollaborationData. The ARConfiguration’s isCollaborationEnabled must be set to true in order to send and receive these updates.
RealityKit’s SynchronizationService
Something that RealityKit offers on top of this is MultipeerConnectivityService; this is a service which synchronizes all the entities within your scene graph along with their components.
Once you have an MCSession set up you just need to create a MultipeerConnectivityService object and to tell the RealityKit scene to use that for its synchronizationService. Here’s an example of how that might look:
From here any AnchorEntity added to the scene will automatically inherit this property, and all Codable Components will also be transferred via the network.
If there is any entity which you do not want to be synchronized across the network with RealityKit automatically, you can set its synchronization to nil. This will mean no updates from this entity or any of its child entities will be synchronized.
Ownership
One important thing to note next is that every created entity has a designated owner. Only the owner of an entity can change that entity in such a way that the changes are distributed to all the other peers in the network; If someone is not already the owner of the entity they can simply request ownership of an entity, and the owner can make a request of an entity before changing any properties.
Here’s an example from a WWDC19 talk (linked below) on creating games with RealityKit, it shows how the host initially creates the two entities and sends them to the client. The host then makes an update to the top entity which is sent to the client. Shortly after the client tries to make an update to the other entity, but is denied because the client is not the owner of this entity. They then make a request to its owner (the host), it is accepted, and so the client is able to update the entity themselves as the new owner of the entity.
The base class for Entity uses a protocol called HasSynchronization, this protocol contains a function for obtaining ownership of an entity, requestOwnership. The example below is a similar setup to the request the client in the GIF above would send to the host.
After the ownership is has been transferred to the client, any changes that they now makes to the entity or any of its children entities will be reflected in the host’s scene, including transform, materials, and any other components it may have. If the host now makes any updates to entity, they will have to request ownership in the same way.
There are two ways of granting ownership of an entity. The first is very simple, it just requires the entity’s ownershipTransferMode to be switched to .autoAccept. Doing so means that whenever an ownership request is made, the current owner will automatically grant the transfer.
Alternatively, if you want to keep the transfer mode in its default of .manual, then you will need to use the SynchronizationEvents.OwnershipRequest event. For more information on subscribing to events in RealityKit, please check out my previous article:
Someone may want to keep a transfer mode manual if the ownership depends on something that only the current owner’s device knows about the scene.
One final way to give ownership of an entity to another peer would be via the giveOwnership method. This method works by just specifying the entity to transfer and the SynchronizationPeerID of the owner-to-be of the entity.
If the Entity ownership is still confusing, I’d highly recommend referring to Apple’s WWDC19 video which covers this very clearly, and can be found at around the 32:30 mark in the following video.
Simplifying MultipeerConnectivity
I have created a class largely based off some examples from Apple, which aims to simplify creating shared experiences, both with and without RealityKit.
To initialise a peer-to-peer connectivity service with an iOS project and this small library all you’ll need to do is create an object of type MultipeerHelper, setting the sessionType to .both and give it a serviceName of a string following a few rules outlined here under serviceType. Here’s an example:
self.multipeerHelp = MultipeerHelper(
serviceName: "helper-test",
sessionType: .both
)This will create both a multi-peer advertiser and browser, this is great for when you don’t care about having a specific host user, which is a great way to get started with a collaborative session.
If you then want to use RealityKit’s synchronizationService, then all you need to do is make sure that your configuration has the isCollaborationEnabled parameter set to true, and add the following line somewhere after multipeerHelp is initialized:
self.arView.scene.synchronizationService = multipeerHelp.syncServiceAt this point all the devices need to do is let ARKit synchronise their ARAnchors by seeing the same positions from similar angles, as explained in the WWDC event here:
Going further
From there, if you want to do more things over the network than just share entities, such as get notified when users join, leave, send data to each other etc. then you can use the MultipeerHelperDelegate, from the MultipeerHelper library to add callback functions. The full list of functions I have currently added to this delegate can be found here.
I made this small wrapper is because it seems that a lot of people don’t want to be creating delegates when creating shared AR experiences, they just want to get straight into it and concentrate on the Augmented Reality content. If you require something that this library does not yet offer, please feel free to open a Pull Request or, if you don’t know how to achieve it yourself, an Issue.
Thank you for reading, be sure to check out the example included in the MultipeerHelper repository, I’ll be adding updates to that as I use collaborative AR more.
Here’s an AR game I have made using Collaborative Sessions. If you have questions about any part of it, leave a comment or send me a tweet!
If you learned something, or just enjoyed this article please tap the 👏 button as many times as you can. Any questions? Reply to this story or send me a tweet. If you want to see more follow me on Medium for future articles.
