Detecting Images

Search for images in the real world and use them for a variety of AR effects.

Overview

An ARSession can be configured to search for specified images in the real world and surface them as image anchors. This anchor can then be used for a variety of effects such as:

  • Making a character seem to turn 3D and jump out of the image.

  • Making a treasure chest appear in a specific real world location near a landmark (for example, in a mural).

  • Making a static image “animate” by creating a virtual overlay.

An ARSession will update an image anchor over time if its initial estimate is wrong or if the image moves, but these updates won’t necessarily be smooth or consistent because the algorithm is designed for detection, not tracking.

The code shown in this example is taken from the ImageDetection scene from the ARDK-Examples Unity package.

Enabling Image Detection

Creating the Image Set

The ARSession needs to be told up front what images to detect. This is done by creating an IARReferenceImage for each image and including them in the ARWorldTrackingConfiguration.DetectionImages that is passed to IARSession.Run.

ARReferenceImages are created by calling ARReferenceImageFactory.Create, which has overrides allowing a reference image to be created from various sources.

Creating From a .jpg.bytes Reference

For this, add the .bytes file extension to your .jpg image. On Mac you can add the extension under Right Click -> Get Info -> Name & Extension.

using Niantic.ARDK.AR.ReferenceImage;
using Niantic.ARDK.Extensions;

private ARImageDetectionManager _imageDetectionManager;
public TextAsset _imageAsBytes;

// ...

byte[] rawByteBuffer = _imageAsBytes.bytes;
IARReferenceImage _yetiImage =
      ARReferenceImageFactory.Create
      (
        "yeti",
        rawByteBuffer,
        rawByteBuffer.Length,
        _physicalImageWidth
      );

// Add image to the manager.
_imageDetectionManager.AddImage(_yetiImage);

Creating From a .jpg Filepath

To provide a persistent data path for the file on any device, …you can create a Assets/StreamingAssets folder in your Unity project.

string _imagePath = "Yeti.jpg";

// The contents of Assets/StreamingAssets are copied to device when installing an app.
string filePathImageBytes = Path.Combine(Application.streamingAssetsPath, _imagePath);

// Create an ARReferenceImage from the local file path.
IARReferenceImage _yetiImage =
  ARReferenceImageFactory.Create
  (
    "yeti",
    filePathImageBytes,
    _physicalImageWidth
  );

// Add image to the manager.
_imageDetectionManager.AddImage(_yetiImage);

Creating the Image Set Asynchronously

Loading and processing images can take enough time that ARDK provides asynchronous versions of all the ARReferenceImageFactory functionality, enabling other UI elements to continue updating while your images load. We provide async methods for all versions of ARReferenceImageFactory.Create using a callback pattern.

Here is an example for the .jpg.bytes example on how to use the callback to add the IARReferenceImage to be tracked, once the creation is finished.

byte[] rawByteBufferAsync = _imageAsBytes.bytes;
ARReferenceImageFactory.CreateAsync
(
  "yeti",
  rawByteBufferAsync,
  rawByteBufferAsync.Length,
  _physicalImageWidth,
  arReferenceImage =>
  {
    _yetiImage = arReferenceImage;
    _imageDetectionManager.AddImage(_yetiImage);
  }
);

Using ARImageDetectionManager

To simplify the process of enabling image detection and creating GameObjects to track the real world images, we provide a Manager you can add to your scene. The ARImageDetectionManager uses a reference to a .jpg.bytes file.

Picking the Best Images

To help an ARSession surface image anchors with optimal speed and accuracy, use reference images that have lots of non-repetitive visual features. On Android there is a strict requirement that the image be at least 300x300 pixels.

Using Image Anchors

Subscribing to Anchor Events

Once an image has been detected in the environment, an ARImageAnchor will be created for it. It will first show up in the ARSession.AnchorsAdded callback, and then in future ARSession.AnchorsUpdated callbacks.

This is an example of attaching a game object to detected images anchors. The game object is added within the OnAnchorsAdded callback and its position updated whenever the anchor moves in the OnAnchorsUpdated callback. When an ImageAnchor is removed from a session, the attached game object will be destroyed.

using Niantic.ARDK.AR.Anchors;
using Niantic.ARDK.Extensions;

public GameObject _prefab;
private Dictionary<Guid, GameObject> _detectedImages = new Dictionary<Guid, GameObject>();

private void Start()
{
    ARSessionFactory.SessionInitialized += SetupSession;
}

private void SetupSession(AnyARSessionInitializedArgs arg)
{
    // Add listeners to all relevant ARSession events.
    var session = arg.Session;
    session.SessionFailed += args => Debug.Log(args.Error);
    session.AnchorsAdded += OnAnchorsAdded;
    session.AnchorsUpdated += OnAnchorsUpdated;
    session.AnchorsRemoved += OnAnchorsRemoved;
}

private void OnAnchorsAdded(AnchorsArgs args)
{
    foreach (var anchor in args.Anchors)
    {
        if (anchor.AnchorType != AnchorType.Image)
            continue;

        IARImageAnchor imageAnchor = (IARImageAnchor) anchor;
        string imageName = imageAnchor.ReferenceImage.Name;

        GameObject gameObjectOnImage = Instantiate(_prefab);
        gameObjectOnImage.name = "Image-" + imageName;
        _detectedImages[anchor.Identifier] = gameObjectOnImage;

        UpdatePlaneTransform(imageAnchor);
    }
}
private void OnAnchorsUpdated(AnchorsArgs args)
{
    foreach (var anchor in args.Anchors)
    {
        if (!_detectedImages.ContainsKey(anchor.Identifier))
            continue;

        IARImageAnchor imageAnchor = (IARImageAnchor)anchor;
        UpdatePlaneTransform(imageAnchor);
    }
}

private void UpdatePlaneTransform(IARImageAnchor imageAnchor)
{
    Guid identifier = imageAnchor.Identifier;

    _detectedImages[identifier].transform.position = imageAnchor.Transform.ToPosition();
    _detectedImages[identifier].transform.rotation = imageAnchor.Transform.ToRotation();

    Vector3 localScale = _detectedImages[identifier].transform.localScale;
    localScale.x = imageAnchor.ReferenceImage.PhysicalSize.x;
    localScale.z = imageAnchor.ReferenceImage.PhysicalSize.y;
    _detectedImages[identifier].transform.localScale = localScale;
}

private void OnAnchorsRemoved(AnchorsArgs args)
{
    foreach (var anchor in args.Anchors)
    {
        if (!_detectedImages.ContainsKey(anchor.Identifier))
            continue;

        Destroy(_detectedImages[anchor.Identifier]);
        _detectedImages.Remove(anchor.Identifier);
    }
}

AR Session RunOptions

When an IARReferenceImage is added to the ImageDetectionManager, the AR Session is re-run with the updated configuration. When removing an IARReferenceImage from the ImageDetectionManager, the AR Session is also re-run and will stop tracking the ImageAnchor and triggering OnAnchorsUpdated callbacks for this image.

Depending on the selected RunOptions of the AR Session, different things happen when a session is re-run with an updated configuration. When “Remove Existing Anchors” is selected, all anchors including ImageAnchors will be removed from the session and have to be re-added by detecting the images again.

Note

Android and ARCore do not support keeping anchors between session re-runs. On Android, existing anchors will still be removed when the RunOption “None” is selected.

Mock Image Anchors

If you want to iterate on an image detection experience in the Unity Editor, refer to the instructions for adding mock image anchors on the Mock Mode page.

Best Practices

Images To Use

The quality of image tracking is highly dependant on the image being tracked. What makes an easily tracked image isn’t immediately intuitive. You’re looking for an image that:

  • Has many image features. Areas of high visual contrast. Something solid color or with a gradient won’t have the features you need.

  • Doesn’t have repeating patterns or features. A chess board has sharp visual features at the intersections of squares, but the visual features repeat constantly making them less useful for placing the image.

ARCore and ARKit both provide tools for determining the quality of a given image. You should use those tools if you need a way to verify an assumption about whether an image will be good or bad for tracking.

Placing Virtual Content

Image detection does not cleanly track moving images, so the anchor is prone to changing its position suddenly. It updates in discrete jumps rather than continuously updating from its old position to its new one, and the angle of the image plane can change more dramatically than plane anchors tend to, making the orientation especially inconsistent. These issues are prominent whenever an image moves, but can also cause problem for stationary images as errors in the image detection are corrected. Depending on the type of virtual content you’re placing, there are ways to work around these issues.

  • For detachable 3D content, like an astronaut who jumps out of a picture and then floats around, it is best to create a new ARAnchor at the image anchor’s position and attach the virtual object to that anchor instead. That anchor should update its pose more smoothly and consistently than the image anchor.

  • Similarly, add a new ARAnchor for content that isn’t near the image but uses it as a landmark. For example, once the ARSession detects a movie poster, place an ARAnchor a meter in front of the image anchor and attach your virtual content to that.

  • For flat 2D content that is directly tied to the image, like a virtual picture frame, the image anchor should be used directly. In this case, being close to the most up-to-date detected location of the image is more important than being consistent to where the virtual object was last frame. This will look poorly tracked if the image moves, but should fix itself once the image stops moving.

  • 3D content that is directly tied to the image, like a virtual character who reaches out of the image but is still connected directly to it, will most spotlight the weaknesses of image detection. However, creative techniques can mitigate these issues.

    • Any small changes in the image anchor’s orientation will cause the character reaching out of it to move a lot, as anything far from the image’s center is on a long lever arm that turns small rotational changes into large translational changes. So, if the image is known to be on a wall, attach the virtual object to the image anchor’s position but ignore its orientation.

    • If it’s important that the virtual object seamlessly mesh with the image, consider entirely occluding the real world image with a virtual copy of it.