Exploring Windows Mixed Reality, switching between 2D / 3D and embedding Web Views

Friday. July 28, 2017 - 4 mins

In these days, I’m exploring the options for switching between Unity 3D and XAML 2D views integrating the access to UWP APIs for content hosted on Web Views.

This scenario could be particularly useful if an app needs to reuse existing code, perhaps available in a website, with the requirement to access the Windows Runtime when executed in Windows Mixed Reality devices and activated from a Unity 3D scene.

Creating the Unity project

To start, I created a new Unity scene and imported the HoloToolkit package downloaded from here.

I applied the Scene and Project settings from the HoloToolkit|Configure menu and added the following prefabs available from the imported package:

• HoloLens Camera
• InputManager
• Cursor

Then, I added a simple cube which, when gazed and air-tapped, permits to trigger the view switching.

I defined a new script TapBehaviour to capture the event and call the 2D view:

using System;
using HoloToolkit.Unity.InputModule;
using UnityEngine;
using UnityEngine.VR.WSA.Input;

public class TapBehaviour : MonoBehaviour, IInputClickHandler {

    // Use this for initialization
    void Start () {

    }

    // Update is called once per frame
    void Update () {
		
    }

    public void OnInputClicked(InputClickedEventData eventData)
    {
        var contentPage = AppViewManager.Views["ContentPage"];
        if (contentPage != null)
        {
            contentPage.SwitchAndConsolidate(contentPage);
        }
    }
}

The interesting part here is the one using the AppViewManager for switching views: after some research, I decided to use this class from GitHub as a starting point for handling the logic via CoreApplicationView.

I created a new script in Unity (AppViewManager.cs) and added the source code from the GitHub repository.

In this way, it is possible to retrieve the list of registered views or a specific one using for instance

AppViewManager.Views["ContentPage"];

And then simply transition to the new page by calling the Switch() or SwitchAsync() methods.

Creating and registering the 2D views

The 2D view named ContentPage needed to be defined and registered in the UWP project generated by Unity using the File|Build Settings|Build command after setting the UWP Build type to XAML to be able to define additional views in my project

Then, I opened the project in Visual Studio and added a new page called ContentPage.xaml containing a WebView and a button to switch back to Unity 3D

<Page
    x:Class="MixedReality2D3D.ContentPage"
    IsTabStop="false"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d"
    Background="#FFFFFF">

    <Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height="90*"/>
            <RowDefinition Height="10*"/>
        </Grid.RowDefinitions>
        <WebView Source="http://www.davidezordan.net/samples/windows10/hostedwebappssample/page.html" />
        <Button Grid.Row="1" Content="Click me" Click="Button_Click" Width="300"
                HorizontalAlignment="Center"
                HorizontalContentAlignment="Center" />
    </Grid>
</Page>

And the code makes use again of the AppViewManager previously imported in Unity:

private async void Button_Click(object sender, RoutedEventArgs e)
{
    var contentPage = AppViewManager.Views["MainPage"];
    await contentPage?.SwitchAsync();
}

We are now making use of two pages MainPage and ContentPage, respectively hosting the 3D and 2D content. Before using them, I needed to register both in the App.cs class right after initialising the Unity player:

private async Task InitialiseAppViewManager()
{
    // Capture the "Main" view which becomes 3D when Unity starts
    var mainView = await AppViewManager.CreateFromCurrentDispatcherAsync("MainPage");

    // Create a new 2D view for the "Content" page.
    var contentPage = await AppViewManager.CreateNewAsync("ContentPage", typeof(ContentPage));
}

Accessing UWP APIs from the hosted web page

The Web View is actually showing content from this web page hosted in my personal web-site which tries to access the UWP APIs if hosted with elevated permissions:

var networkElement = document.querySelector("#networkElement");
var deviceFamilyElement = document.querySelector("#deviceFamilyElement");

var status = "Sorry about that :( No Windows Runtime here";
var deviceFamily = "Unknown Device Family";

if (typeof Windows !== "undefined") {
    if (Windows.Networking.Connectivity.NetworkInformation.getInternetConnectionProfile().getNetworkConnectivityLevel() ==
        Windows.Networking.Connectivity.NetworkConnectivityLevel.internetAccess) {
        status = "Happy days! You have access to internet and the Windows Runtime :)";
    }

    var family = Windows.System.Profile.AnalyticsInfo.versionInfo.deviceFamily;
    if (family) {
        deviceFamily = "Device family: " + family;
    }
}

To enable Windows Runtime access, I modified the Package.appxmanifest and added the following under the Application section:

<uap:ApplicationContentUriRules>
  <uap:Rule Match="http://www.davidezordan.net/" WindowsRuntimeAccess="all" Type="include"/>
</uap:ApplicationContentUriRules>

After this step, I was able to launch the app using an HoloLens device or emulator:

And then switch to the 2D view hosting the web page accessing UWP APIs on HoloLens by tapping the cube:

And return to the Unity 3D view using the "Click me" button.

As usual, the source code is available for download on GitHub.

Davide Zordan

Senior Software Engineer

Tweet Share Share