In this post we’re going to dive into ARKit, we’ll find out what it is and get started with building our first augmented reality experience in ARKit.

ARKit is Apple’s toolkit for building augmented reality experiences on iOS devices. It was initially released at WWDC 2017, then ARKit 2.0 was released at WWDC 2018.

Before we jump into any code it’s important to understand what ARKit is and why we need it. Augmented Reality on mobile devices is hard, it’s hard because of the heavy calculations/triangulations/mathematics. It’s also very hard to do AR without killing the users battery or reducing the frame rates. ARKit takes care of all the hard parts for you hence allowing you to use a clean and simple API.

The Basics of ARKit

In order to Augment our Reality then we need to be able to track reality, eg how do we map the world so that we know what it looks like in a digital form. Devices like HoloLens have special sensors specifically designed for AR tracking. Mobile devices don’t have anything specifically designed for world tracking, but they do have enough sensors that when combined with great software we can track the world.

ARKit takes advantage of the sensors (camera, gyroscope, accelerometer, motion) already available on the device. As you can see in the diagram below ARKit is only responsible for the processing and basically this means sensor reading and advanced mathematical calculations. The rendering can be handled by any 2D/3D rendering engine, which includes SceneKit as you see below but majority of the apps will be using a 3D engine like Unreal or Unity.

Understanding the World

The primary function of ARKit is to take in sensor data, process that data and build a 3D world. In order to do this ARKit uses a truckload of mathematical calculations, we can simplify and name some of the methods ARKit is using.

The diagram below shows Inertial Odometry, Inertial Odometry takes in motion data for processing. This input data is processed at a high frame rate.

The diagram below shows Visual Odometry, Visual Odometry takes in Video data from the camera for processing. The processing of visual data is done at a lower framerate and this is due to the fact processing the visual data is CPU intensive.

ARKit then combines the odometries to make what’s called Visual Inertial Odometry. This will have the motion data processed at a high framerate, the visual data processed at a lower framework and differences between the processing normalised. You can see Visual Inertial Odometry in the diagram below.

Triangulation allows the world mapping

In a very simple explanation triangulation is what allows ARKit to create a model of the world. In a similar way to humans, so as the phone is moved around ARKit will do calculations against the differences allowing ARKit to essentially see in 3D. A digital map of the world is created.

As you can see below a world map is created within ARKit.

Augmenting Reality (with Anchor Points)

As the world is mapped ARKit will create and update Anchor Points, these anchor points allow us to add items in reference to the anchor point. As you can see in the diagram below ARKit has added Anchor points and we’ve placed an object (3D vase) near the anchor point. As the devices is moved around these anchor points are updated, so it’s important that we track these changes and update our augmentations of the world.

As I mentioned before ARKit only does the processing and provide the data. It’s up to us to render objects in the 3D world. Below shows how you’ll combine both the video captured and then overlay a 3D rendering. As we move both the video capture and the 3D rendering is updated.

Tracking Options/Features.

ARKit has a few different tracking options and features that I will go over below.

Orientation Tracking

This is the most basic type of tracking available in ARKit, this will track your orientation within the world. It will not track your location in physical space, in essence it’s like your standing still and can view 360 degrees in the world.

World Tracking

This option in ARKit is the most common, in this scenario ARKit tracks and builds a complete world maps and allows you to move freely within the world. It’s important to note that World Tracking includes the majority of features in ARKit including Plane Detection, Maps, Image Tracking and Object Detection.

Plane Detection

As we’re moving around the 3D world we need to know about the different surfaces in the world this is where plane detection comes in. The first release of ARKit included only horizontal orientation, in ARKit 2 we now have the option of both vertical and horizontal orientation. In the image below you can see the floor being detected as a plane.

Saving & Loading Maps 

In ARKit 2.0 we can now save, load and share the world map. In ARKit 1.0 the map was internal and only ever kept around for a single users session. This meant that in ARKit 1.0 you could not save sessions (maps) to be resumed later or share. This allows for a few scenarios including multiplayer games and the ability to save world maps.

Below is a video of a multi player game that leverages ARKit 2.0 map sharing feature.

Image Tracking

Imaging tracking allows your app to easily detect an image in the real world, this for example might be a photo or a business card or a dvd case. Once you’ve detected this image then you’ll be able to easily augment the reality around image. Normally a task like this would be really difficult but again ARKit makes this really easy for us, the only steps we need to take are a reference (eg the image we need to track) with the physical size, once we set that up we use a single option to set turn on the feature and add the reference images into ARKit.

Below I’ve included a reference video that leverages Image Tracking. While it looks very impressive the application below could be implemented with ARKit 2.0 without a huge amount of effort, around 80-100 lines of code.

Object Detection

Object Detection (ARKit 2.0 feature) allows us to both scan and detect 3D objects. I think the best way to understand this would be a simple video.

Building your first ARKit Experience

To give you a feel for how easy it is to build a ARKit experience I’m going to take you through a simple application in which you can see in the video below. As you move the phone around a plane is detected (which ARKit does for you), we place a Node on that surface then if a users taps the Node we add a box on top of that node where the user tapped.

Let’s jump into some code to see how easy it is to get started with ARKit.

The first thing we do in this app is create a SceneView and add it as a SubView to the visible ViewController, as we see below.

The next step we need is to call the run command on the scene with the world tracking configuration, as we see below.

As we move the phone around and surfaces are detected the DidAddNode method is called by ARKit. As you can see below if the anchor is a ARPlaneAnchor we then add our PlaneNode, which is the blue we see in the video.

Then if a user touches the PlaneNode we then add a Cube on top of where the user just touched.

That’s it, all we need to do for our first AR Experience. You can see the full code file below or get all the code from https://github.com/rid00z/ARKitExample

using System;
using System.Collections.Generic;
using System.Linq;
using ARKit;
using ARKitExample.Nodes;
using Foundation;
using SceneKit;
using UIKit;

namespace ARKitExample
{
    public partial class ViewController : UIViewController
    {
        private readonly ARSCNView sceneView;

        protected ViewController(IntPtr handle) : base(handle)
        {
            this.sceneView = new ARSCNView
            {
                AutoenablesDefaultLighting = true,
                //DebugOptions = ARSCNDebugOptions.ShowFeaturePoints,
                Delegate = new SceneViewDelegate()
            };
            this.View.AddSubview(this.sceneView);
        }

        public override void ViewDidLoad()
        {
            base.ViewDidLoad();

            this.sceneView.Frame = this.View.Frame;
        }

        public override void ViewDidAppear(bool animated)
        {
            base.ViewDidAppear(animated);

            this.sceneView.Session.Run(new ARWorldTrackingConfiguration
            {
                AutoFocusEnabled = true,
                PlaneDetection = ARPlaneDetection.Horizontal,
                LightEstimationEnabled = true,
                WorldAlignment = ARWorldAlignment.GravityAndHeading
            }, ARSessionRunOptions.ResetTracking | ARSessionRunOptions.RemoveExistingAnchors);
        }

        public override void ViewDidDisappear(bool animated)
        {
            base.ViewDidDisappear(animated);

            this.sceneView.Session.Pause();
        }

        public override void TouchesEnded(NSSet touches, UIEvent evt)
        {
            base.TouchesEnded(touches, evt);

            if (touches.AnyObject is UITouch touch)
            {
                var point = touch.LocationInView(this.sceneView);
                var hits = this.sceneView.HitTest(point, ARHitTestResultType.ExistingPlaneUsingExtent);
                var hit = hits.FirstOrDefault();

                if (hit == null) return;

                var cubeNode = new CubeNode(0.05f, UIColor.White)
                {
                    Position = new SCNVector3(
                        hit.WorldTransform.Column3.X,
                        hit.WorldTransform.Column3.Y + 0.1f,
                        hit.WorldTransform.Column3.Z
                    )
                };

                this.sceneView.Scene.RootNode.AddChildNode(cubeNode);
            }
        }

        class SceneViewDelegate : ARSCNViewDelegate
        {
            private readonly IDictionary<NSUuid, PlaneNode> planeNodes = new Dictionary<NSUuid, PlaneNode>();

            public override void DidAddNode(ISCNSceneRenderer renderer, SCNNode node, ARAnchor anchor)
            {
                if (anchor is ARPlaneAnchor planeAnchor)
                {
                    var planeNode = new PlaneNode(planeAnchor);
                    node.AddChildNode(planeNode);
                    this.planeNodes.Add(anchor.Identifier, planeNode);
                }
            }

            public override void DidRemoveNode(ISCNSceneRenderer renderer, SCNNode node, ARAnchor anchor)
            {
                if (anchor is ARPlaneAnchor planeAnchor)
                {
                    this.planeNodes[anchor.Identifier].RemoveFromParentNode();
                    this.planeNodes.Remove(anchor.Identifier);
                }
            }

            public override void DidUpdateNode(ISCNSceneRenderer renderer, SCNNode node, ARAnchor anchor)
            {
                if (anchor is ARPlaneAnchor planeAnchor)
                {
                    //this.planeNodes[anchor.Identifier].Update(planeAnchor);
                }
            }
        }
    }
}

I’ve actually had this project around since this start of the year but I never blogged it. I did this code for my speaking session at Microsoft Ignite in Australia.

Warning – This code is a playground there’s lots of play code and not reflective of a production application. This project is for only used for learning about Xamarin.Forms features.

In this blog post I’ll go over some of the features want to demo with this app, with some screenshots. Over the coming weeks I plan to demonstrate that anyone who says you cannot build fast and beautiful apps in Xamarin.Forms is simply wrong, lying or ignorant. The deal is that you just need to know how Xamarin.Forms work.

Post-Layout Animations/Translations – Right Slide Bar

One of the most powerful features of Xamarin.Forms are the Post-Layout translations. 

People are so surprised when I tell them this is done without any custom renders, let’s take a look at how.

In this case I make use of the Grid in Xamarin.Forms, in actual fact the sidebar is layered over the listview which in the background. The Grid only has a single column.

<Grid>
  <FriendList ... (300 Set Width)
  <Header ...(Full Screen Width)
  <ListView ...(Full Screen Width)
</Grid>

So the FriendList normally sits on top of the List but what we actually do is translate the X coordinate so it sits off the screen, just after the page size allocated.

protected override void OnSizeAllocated(double width, double height)
{
    base.OnSizeAllocated(width, height);

    friendList.TranslationX = this.Width;
}

Then the awesomely simple part, when a user taps the menu button we just translate the views.

void FriendListTapped(object sender, System.EventArgs e)
{
    if (friendList.TranslationX == Width) // determine if open
    {
        friendList.TranslateTo(Width - 300, 0, 100); //open the friendlist
        myListView.TranslateTo(-300, myListView.TranslationY, 100); //move the listview to allow for friendlist
        header.TranslateTo(-300, header.TranslationY, 100); //move the header to allow for friendlist
    }
    else
    {
        friendList.TranslateTo(Width, 0); //close the friendlist
        header.TranslateTo(0, header.TranslationY, 100); //put the header back
        myListView.TranslateTo(0, myListView.TranslationY, 100);  //put the listview back
    }
}

Disappearing NavigationBar

This one was a little harder and it’s still not working perfectly.

In this case our grid has 2 rows, 80 at the top for the header size and a * for the rest.

<Grid.RowDefinitions>
    <RowDefinition Height="80"></RowDefinition>
    <RowDefinition Height="*"></RowDefinition>
</Grid.RowDefinitions>

On the default state the header just sits in the first row.

<Grid x:Name="header" BackgroundColor="{StaticResource BlueHeaderColor}" >

While the ListView actually uses the Grid.RowSpan=”2″ so it goes over multiple rows, I had to do this because the height of the listview needs to be the full height of the screen once the header disappears. I also make use of the ability to set post-layout values in Xaml, setting the TranslationY to 60 on the ListView (this moves it down into the correct position by default).

<local:AdvancedListView BackgroundColor="#D7DBDF" HasUnevenRows="true" TranslationY="60" Grid.RowSpan="2" 			
        x:Name="myListView" 
		ItemsSource="{Binding Items}" ItemTemplate="{StaticResource homeWallListTemplateSelector}" 
		SeparatorVisibility="None"
		> 
</local:AdvancedListView>

So when we want to hide the navigation bar we must hook into the Scrolled event on the listview and translate the Y values on the views depending on the scroll position. This part is really just for a demo, it’s not thought out and very prototype.

void MyListView_Scrolled(Point obj)
{
    var headerHeight = 60;

    if (obj.Y <= 0)
    {
        header.TranslationY = 0;
        myListView.TranslationY = headerHeight;
    }
    else if (obj.Y <= headerHeight)
    {
        header.TranslationY = -obj.Y;
        myListView.TranslationY = -obj.Y + headerHeight;
    }
    else if (obj.Y > headerHeight) //scrolled 
    {
        header.TranslationY = -headerHeight;
        myListView.TranslationY = 0;
    }
}

Like Animations

One other thing I like about this app is the animated likes buttons.

In this case I cheated a little bit and used a Animated Gif, it worked ok. I also used a few other features.

As with all the other cases I leveraged the Grid. You’ll also notice that I used a few extra controls, AdvancedFrame from FreshEssentials to allow for the rounded edges and a GifImageViewControl.

<ContentView Grid.Row="5" Grid.Column="1" Grid.ColumnSpan="4" x:Name="likeArea"  Opacity="0">
    <fe:AdvancedFrame Padding="15, 4" HasShadow="true" OutlineColor="#EEEEEE" CornerRadius="70" Corners="all"  InnerBackground="White"  >
        <local:GifImageViewControl  Aspect="AspectFill" Source="facebookreactionsanimation.gif"></local:GifImageViewControl>
    </fe:AdvancedFrame>
</ContentView>

As with all the other samples we simply leverage some Translations and in this case we also leveraged the FadeTo animation.

async void Handle_Clicked(object sender, System.EventArgs e)
{
    if (likeArea.TranslationY == -35)
    {
        await likeArea.TranslateTo(0, 0, 100, Easing.Linear);
        await Task.Delay(50);
        await likeArea.FadeTo(0, 50);
    }
    else
    {
        await likeArea.FadeTo(1d, 20);
        await likeArea.TranslateTo(0, -35, 100, Easing.Linear);
    }
}

If you want to check out the code base please find it here: https://github.com/XAM-Consulting/FacebookForms

Thanks