Multi-User Augmented Reality Experiences with Agora (Part 2 of 2)

What is Agora Real-time Messaging (RTM)?

Agora real-time Messaging is an SDK that enables you to send data between devices that are connected to the network. These data could contain an encoded struct, plain text, or even larger files, ranging from PDF to 3D files.

How Real-time Messaging is Used in CollaboratAR

Currently, six different types of messages will be sent across the network in CollaboratAR. They are stored in an enum to make it easy to tell them apart when sending and receiving:

enum RawRTMMessageDataType: String {

// MARK: Globe Scene

/// A session is available to display on the globe
case channelAvailable
/// Request all available sessions
case getSessionData

// MARK: Collaborative Session

/// A model has been created or transform modified
case singleCollabModel
/// Multiple models are available to display in the scene
case multiCollabModels
/// A model has been deleted
case removeCollabModel
/// Transform update for a remote user
case peerUpdate
}

Globe Scene 🌎

Sending

In the initial scene a globe is visible. If a remote user is currently in a channel or experience, we see them on the globe as a red circle with an orange centre.

struct ChannelData: Codable {
var channelName: String
var channelID: String
var position: SIMD3<Float>
var systemImage: String?
}
// channelData: ChannelData
let jsonData: Data = try! JSONEncoder().encode(channelData)
let channelDataRTM = AgoraRtmRawMessage(
rawData: jsonData,
description: RawRTMMessageDataType.channelAvailable.rawValue
)
// rtmChannel: AgoraRtmChannel
rtmChannel.send(channelDataRTM)

Receiving

Now that our message has been sent across the network, we need to use the Agora RTM Delegate methods to catch the incoming messages and interpret them as channels available to join.

let messageType = RawRTMMessageDataType(rawValue: message.text)
if messageType == .channelAvailable {
let channelData = try! JSONDecoder().decode(
ChannelData.self, from: rawMessage.rawData
)
// show channel data on globe...
}

Globe Scene Conclusion

That’s all the data that needs to be sent across Agora RTM network in the initial globe scene. All each device needs to know is what available channels there are and where to place them on the globe. No audio channels are joined when in the lobby. That comes in the next section.

Collaboration Scene

A lot more data are sent around in the collaboration scene, including the positions of all the other users, any models they add, move, and delete, as well as the data interpreted from the audio channel they are all members of.

enum CollaborationState {
case globe
case collab(data: ChannelData)
}

Sending All Models

All the entities we want to share conform to a custom protocol HasCollabModel. They are children of an entity we have a reference to, called collabBase.

guard let collabChildren = collabBase.children.compactMap{
$0 as? HasCollabModel
} else { return }
let allModelData: [ModelData] = collabChildren.map { $0.modelData }
// member: AgoraRtmMember
let jsonData = try! JSONEncoder().encode(allModelData)
let rawMessge = AgoraRtmRawMessage(
rawData: jsonData,
description: RawRTMMessageDataType.multiCollabModels.rawValue
)
self.rtmKit.send(rawMessge, toPeer: member.userId)

Receiving All Models

On the other end, the device that just joined the channel needs to receive these models and place them in the scene.

if messageType == .multiCollabModels {
let modelDatas = try! JSONDecoder().decode(
[ModelData].self, from: rawMessage.rawData
)
CollaborationExpEntity.shared.collab?.update(with: modelDatas)
}
func createModelEntity(with modelData: ModelData) {
let collabMod = CollabModel(with: modelData)
self.collabBase.addChild(collabMod)
}

Updating User Positions

All the remote augmented reality users will have their location relative to the ground square sent to the channel so that all the other users know where they are in the world. The called method is CollaborationExpEntity.updatePositions. It is called on an interval of 0.3 seconds after you have joined a channel and set the ground square. The 0.3 seconds value is arbitrary and can be altered if a faster update is desired.

if messageType == .peerUpdate {
let peerData = try! JSONDecoder().decode(
PeerData.self, from: rawMessage.rawData
)
CollaborationExpEntity.shared.collab?.update(with: peerData)
}
func update(with peerData: PeerData) {
if let child = self.findEntity(
named: peerData.rtmID
) as? PeerEntity {
child.update(with: peerData)
} else {
self.createPeerEntity(with: peerData)
}
}

Updating a Model

A similar update to .multiCollabModels is .singleCollabModel. This update is sent almost exactly the same way as .multiCollabModels, but it is a single model rather than an array.

func sendCollabData(for collabEnt: HasCollabModel) {
let jsonData = try! JSONEncoder().encode(collabEnt.modelData)
let rawMessge = AgoraRtmRawMessage(
rawData: jsonData,
description: RawRTMMessageDataType.singleCollabModel.rawValue
)
self.channels.collab?.send(rawMessge)
}
if messageType == .singleCollabModel {
let modelData = try! JSONDecoder().decode(
ModelData.self, from: rawMessage.rawData
)
CollaborationExpEntity.shared.collab?.update(with: [modelData])
}

Deleting an Entity

When deleting an entity, the enum .removeCollabModel is used, and the individual model data is sent with the following message:

// model: CollabModel, collabChannel: AgoraRtmChannel
let jsonData = try! JSONEncoder().encode(model.modelData)
let rawMessge = AgoraRtmRawMessage(
rawData: jsonData,
description: RawRTMMessageDataType.removeCollabModel.rawValue
)
collabChannel.send(rawMessge)

Testing

All of the above methods can be found in the full example project:

Other Resources

For more information about building applications using Agora Video and Audio Streaming SDKs, take a look at the Agora Video Call Quickstart Guide and Agora API Reference.

Conclusion

I hope you’ve enjoyed this post and the project that comes along with it. The idea of this project is to showcase how Agora Real-time Messaging SDK can be used to create interactive experiences with people around the world.

Excited about all things Augmented Reality! Developer Evangelist at Agora.io.