Talk About The Brand-New Microsoft Mesh Platform From A Developer's Perspective
This post is available at Medium as well.
As a Unity/Azure developer, I’m very excited to see Microsoft announced their brand-new Microsoft Mesh platform at the Ignite conference.
The Ignite keynote was created and delivered in AltspaceVR and powered by experiences built using early Mesh tools. And they released the Microsoft Mesh preview application for HoloLens 2 quickly, so the HoloLens 2 users can experience the app powered by the Mesh platform now.
However, as a developer, I’m keen to know more about this interesting platform from a developer’s perspective. So the following are the information I collected and some idea I want to share with you.
Why Microsoft Mesh?
So the first question is why does Microsoft introduce the Mesh platform to the customers/developers? And does it really necessary?
Actually, the Mesh platform is mainly built for multi-user MR apps. And there are some issues you may encounter when developing a multi-user MR app.
For example, it’s complex to synchronize actions and expressions in a geographically distributed session, and it’s difficult to bring high-fidelity 3D models/represent people with realism in an MR app. These are real challenges when you creating a multiple users app. And I think Microsoft solved some of these challenges according to the VR Ignite keynote. As you can see in the image below, James Cameron joined the keynote from New Zealand!
Why not all of these challenges are “solved”? Because as a developer, I’m still curious about the requirement of the 3D volumetric captures in the keynote.
As you can see in the video above, the 3D captures are impressive, which are very likely captured in Microsoft Mixed Reality Capture Studios. And these types of equipment are expensive, which are not suitable for all developers.
Alternatively, you can choose to use an avatar to present the person.
What is Microsoft Mesh?
Then, another question is what is Microsoft Mesh?
To be honest, when I first heard about this technology before the ignite conference, the content was created and delivered in AltspaceVR. It can be misleading to think that this technology is an application. Actually, not only AltspaceVR but also the Microsoft Mesh preview application for HoloLens 2 are Mesh-enabled apps, not the Mesh platform itself.
As you can see from the diagram below, there are some Mesh-enabled apps available and the Mesh platform supports multiple devices I think this is because lots of calculations will be done on the Azure cloud. And as an Azure/Unity developer, I pay the most attention to the middle part - Developer platform.
The Mesh developer platform is built on Azure, you can find some familiar words in the Core Platform section on the diagram, such as Identity, Graph, Infrastructure blah, blah. And Mesh provides capabilities that include the immersive presence, spatial maps, holographic rendering, and multiuser sync to address the challenges I mentioned above.
From a developer’s perspective, the most used by a developer is still the Mesh SDK, in the Toolkit section on the diagram. Mesh provides a cross-platform developer SDK, which supports Unity, native C++, and C# just like the Microsoft Mixed Reality SDK.
However, the Mesh SDK is in a limited private preview currently. I hope Microsoft can open it up to everyone as soon as possible, although it may take a few months.
How Do We Use Microsoft Mesh To Develop Our Own Products?
And the last question is how do we use the Mesh platform to develop our own products? Although the Mesh SDK is still in a private preview phase, however, from an Azure/Unity developer’s perspective, I found it’s very similar to other SDKs Microsoft provided to Unity developers before.
For the Azure part, we should create a new App registration on Azure to connect the Mesh SDK to the cloud, and we can get the client ID and tenant ID that will be used in the SDK for Unity later.
For the Unity part, in addition to the core ToolKit, it looks like Microsoft also provides packages for Avatar and UX respectively.
Then we can open a Microsoft Mesh Panel in Unity Editor, and provide the client ID & tenant ID from the App registration to connect to the cloud.
Actually, because the Mesh SDK is still in a private preview phase, so I don’t know more detail about developing with the Mesh SDK, but as I mentioned before it looks like other SDKs Microsoft provided to Unity developers before.
Conclusion
Microsoft Mesh platform is built to help developers to build their own multi-user MR apps. And from my point of view, the Mesh platform does a good job to address the multiuser sync challenge and provides an interesting avatar system.
However, I believe it will cost money to make the amazing 3D captures shown at the Ignite VR keynote, and it’s not suitable for all developers.
It’s built on Azure, standing on the shoulders of giants. But currently, the SDK is still not available for developers. I look forward to using it to develop some applications myself. AltspaceVR and the Microsoft Mesh preview application for HoloLens 2 are cool, but it also needs more cases to prove its maturity.
In short, the Microsoft Mesh platform is a promising solution for multi-user MR applications, and I hope Microsoft can open it up to everyone without waiting too long. : )
Thanks for reading and hope it’s helpful.
Have a chat at https://www.linkedin.com/in/chenjd/