interface IARFrame (Niantic.ARDK.AR.IARFrame)

Overview

A video image, with position-tracking information, captured as part of an AR session. More…

interface IARFrame: IDisposable {
    // properties

    ReadOnlyCollection<IARAnchor> Anchors;
    IARCamera Camera;
    IImageBuffer CapturedImageBuffer;
    IntPtr[] CapturedImageTextures;
    IDepthBuffer Depth;
    IDepthPointCloud DepthPointCloud;
    ARFrameDisposalPolicy? DisposalPolicy;
    IARLightEstimate LightEstimate;
    ReadOnlyCollection<IARMap> Maps;
    IReadOnlyList<Detection> PalmDetections;
    IARPointCloud RawFeaturePoints;
    ISemanticBuffer Semantics;
    float WorldScale;

    // methods

    Matrix4x4 CalculateDisplayTransform(
        ScreenOrientation orientation,
        int viewportWidth,
        int viewportHeight
    );

    IDataBufferFloat32 CopySemanticConfidences(string channelName);

    ReadOnlyCollection<IARHitTestResult> HitTest(
        int viewportWidth,
        int viewportHeight,
        Vector2 screenPoint,
        ARHitTestResultType types
    );

    void ReleaseImageAndTextures();
};

Detailed Documentation

A video image, with position-tracking information, captured as part of an AR session. A running AR session continuously captures video frames from the device camera. For each frame, the image is analyzed together with data from the device’s motion sensing hardware to estimate the device’s real-world position.

Properties

ReadOnlyCollection<IARAnchor> Anchors

The list of anchors representing positions tracked or objects at a point in time.

IARCamera Camera

Information about the camera position, orientation, and imaging parameters used to capture the frame.

IImageBuffer CapturedImageBuffer

The CPU-side image.

In Android, initiates a copy from the native GPU texture into CPU-accessible memory.

Note

May be null.

IntPtr[] CapturedImageTextures

One or more native GPU textures.

In iOS, this will be two textures, together they represent an image of format YCbCr.

In Android, this will be a single texture of format BGRA.

Note

Not supported in Remote Debugging.

IDepthBuffer Depth

The depth buffer.

Note

May be null.

IDepthPointCloud DepthPointCloud

3D positions generated based on the Depth Buffer.

Note

May be null

ARFrameDisposalPolicy? DisposalPolicy

Gets or sets a value telling the retain policy of this frame. If unset (that is, null) uses the value set at the session that created this frame.

IARLightEstimate LightEstimate

An estimate of lighting conditions based on the camera image.

Note

May be null, e.g. when not tracking.

ReadOnlyCollection<IARMap> Maps

The list of maps from the Computer Vision system

IReadOnlyList<Detection> PalmDetections

Gets the detection rectangle and confidence where palms may be.

Note

This is an experimental feature. Experimental features should not be used in production products as they are subject to breaking changes, not officially supported, and may be deprecated without notice

IARPointCloud RawFeaturePoints

Raw 3D positions that are used in scene understanding.

Note

May be null.

Note

Not currently supported in Remote Debugging.

ISemanticBuffer Semantics

The semantic segmentation buffer.

Note

May be null.

float WorldScale

The scaling factor applied to this frame’s data.

Methods

Matrix4x4 CalculateDisplayTransform(
    ScreenOrientation orientation,
    int viewportWidth,
    int viewportHeight
)

Returns an affine transform for converting between normalized image coordinates and a coordinate space appropriate for rendering the camera image onscreen.

Note

Returns a pre-calculated value in Remote Debugging.

Parameters:

orientation	The current interface orientation.
viewportWidth	Viewport width, in pixels.
viewportHeight	Viewport height, in pixels.

IDataBufferFloat32 CopySemanticConfidences(string channelName)

Acquires per-pixel confidences for the specified semantic class. The confidence values get copied to a floating point CPU buffer.

Note

May be null.

Parameters:

channelName

The semantics channel to get the confidences for.

Returns:

Floating point CPU buffer containing per-pixel confidences when called on key-frames.

ReadOnlyCollection<IARHitTestResult> HitTest(
    int viewportWidth,
    int viewportHeight,
    Vector2 screenPoint,
    ARHitTestResultType types
)

Using a screen location, find points on real-world surfaces and objects in the camera view.

Parameters:

viewportWidth	Width of the screen in pixels.
viewportHeight	Height of the screen in pixels.
screenPoint	A 2D point in screen (pixel) space.
types	The types of results to search for. Certain values are not supported on some platforms. See ARHitTestResultType documentation for details.

Returns:

An array of hit test results in order of closest to furthest. May be zero-length.

void ReleaseImageAndTextures()

Releases the captured image and textures.

Due to Unity’s Garbage Collector (GC), we can’t be sure when our memory will be deallocated even if there are no more references to it. This method serves as a way to release the largest memory suck in this class the images and textures.

This might clear some dictionaries that cache accessed values, which is not thread safe, so this should only be called from the Unity main thread