FOVE SUPPORT CENTER

Articles in this section

See more

The FOVE Eye Tracker: Every Field Explained

Avatar
Jeff
  • Updated
Follow

日本語版はこちらへ

Introduction

In this article we describe all the data available from the FOVE Eye Tracker. There are two general ways to collect data described here:

The two are equivalent in terms of data reported but since the Unity Gaze Recorder is integrated with the Unity Game Engine, it has some extra things included. Since it outputs a CSV, the Unity Gaze Recorder can't capture eye data (the Unity plugin or SDK can). The underlying FOVE SDK is engine agnostic.

The Gaze Recorder writes a CSV out automatically without the need for any programming. For this reason, it's recommended for researchers and anyone who wants to easily record data.

The FOVE runtime service manages multiple client applications simultaneously and handles multiplexing of data. This means you have one program rendering content, and one or more entirely different programs recording data. Or you can have a single program rendering VR content and recording data. It is recommended to multi-thread such a program, because the VR content runs at the screen frame rate (70hz for FOVE0), and the Eye Tracking data updates at the camera frame rate (120hz for FOVE0).

All eye tracker data is uploaded as soon as the camera image comes in and the eye tracker has completed processing. When using the SDK, you can sync your thread's main loop to the camera by using the Headset::waitAndFetchNextEyeTrackingData API call.

See output of data from an actual run on this public Google Sheet.

 

Coordinate Systems

In the Gaze Recorder you can choose between three coordinate systems for gaze ray calculation: World, Local and HMD. The SDK always uses HMD coordinates.

The choice of the coordinate system affects the following fields of the Gaze Recorder output:
Combined Gaze Ray
Eye Ray Left / Right

The "Eye Ray" for an eye consists of the coordinate of the eyeball center and the direction of the gaze from there, and the "Combined Gaze Ray" is a certain average of the left/right "Eye Rays".

The systems all use Cartesian coordinates in meters (except where noted below), but differ in the choice of a reference point / origin.

 

HMD coordinate system - the gaze ray vectors are given relative to the headset. They depend only on the position and orientation of the user's eyes relative to the headset, and do not include head rotation or body motion of the user. The HMD coordinate system is useful for studies when the stimulus is fixed with respect to the user.

In this system +X is to the right, +Y is upward, and the +Z axis is forward, all with respect to the user's face. Thus a direction of (0, 0, 1) indicates directly forward with no vertical or horizontal component.

The origin of this system is the midpoint between the rotation centers of the user's eyes. Thus (0.0315, 0, 0) is the position of the right eye if the user's IOD is 63mm (IOD is explained below).

Local coordinate system - the gaze vectors are given relative to a fixed real world position. They take into account the headset rotation and translation (translation is available only if position tracking is enabled). You can set the origin of the Local coordinate system to the current headset position and orientation using Tare Position and Tare Orientation buttons in the FOVE Debug Tool, or programmatically via the Headset::tareOrientationSensor and Headset::tarePositionSensor. The Local coordinate system is useful for studies that aim to relate eye movements to the head or body motion.

World coordinate system - the gaze vectors are given relative to the origin of the Unity virtual space. Besides the headset rotation and translation the gaze vectors incorporate the transform of the Unity object parent to the Fove Interface (such as Fove Rig). The World coordinate system is useful for studies that correlate user gaze to the virtual objects.

 

In short, given that the point p⃗ in the HMD coordinate system is p⃗_hmd, the headset orientation/translation matrix is H, and the local-to-world matrix is U, then p⃗_local = H * p⃗_hmd, and p⃗_world = U * H * p⃗_hmd.

 

Gaze Rays in the HMD coordinate system

Each gaze ray is defined by a pair of vectors: a position and a direction. The position indicates the start point of the ray, which then continues indefinitely in the given direction. The start point of the ray is the center of the eye for both of the eye rays, and varies for the combined gaze. The direction is always expressed as a unit vector (magnitude of 1). 

gaze_recorder_rays_and_coords.png

Error Fields

In the SDK API, each function call returns an error code alongside the data.

In the Gaze Recorder, every data item has a corresponding column for errors. A blank entry in that column indicates no error.

In most cases, if there's an error, then the data should be disregarded, considered as junk. However, there are some errors regarding data reliability, which may indicate that the data is partially reliable. Please note the the following error codes:

  • Data_LowAccuracy - data item was calculated, however the value can be of lower accuracy due to unfavorable tracking conditions. The value can be used with caution.
  • Data_Unreliable - data item was not calculated because tracking has been lost, or the tracking conditions were unacceptable. The value should not be used.
  • Data_Uncalibrated - data item requires a calibration that was not performed. The value should not be used.

Errors that don't begin with Data_ indicate other issues, such as if the Service is not running, or usage errors of the API.

 

FOVE Eye Tracking Fields

Combined Gaze Ray - the general purpose gaze ray. It's generated by combining the data from the left and right gazes, also taking into account their reliability. This is the recommended field to use when you simply need "the direction of the users gaze", because it will always return the most reliable data, and the user doesn't need to handle cases like detecting if one eye is closed, etc.

When both eye's gaze rays are reliable, the combined ray is an average of the two. When one of the rays is unreliable the combined ray equals the other eye's ray.

The origin of the ray is important to take into account, especially when the user is converged on a near object. Because sometimes the right or left gaze is used, the origin can move between those two eyes. When both eyes are tracking well, the origin is the midpoint between the center of the two eyes, which is (0, 0, 0) in the headset coordinate system.

The ray origin is returned in meters, the ray direction is a directional unit vector.

Gaze Depth - distance to the estimated point where the left and right gaze rays would intersect. In reality the eye rays usually do not actually intersect because of various inaccuracies. Put simply, it is the distance to the object the user is looking at, along the Combined Gaze Ray. FOVE's algorithm is more advanced than simply finding the closest point between two rays, so we recommend using this field and not trying to do it manually.

This distance is always returned in meters.

Eye Ray - individual gaze rays for left and right eye. Each ray consists of a position vector, that represents the starting point of the ray, given in meters from the origin of the coordinate system, and a directional unit vector along the direction of the gaze.

  • SDK: Headset::getGazeVector
  • Gaze Recorder: Eye Ray [left/right] [dir/pos] [x/y/z]
  • License: Basic, Pro, or Enterprise

Eye State - can be Open, Closed or Not detected. Closed state means that the eye is found on the eye camera image and it is closed, for example if the user is blinking. Not detected state means that the eye is not detected at all, for example if the headset is not mounted or positioned so poorly that the eyes are not visible on the eye camera.

  • SDK: Headset::getEyeState
  • Gaze Recorder: Eye State [left/right]
  • License: Basic, Pro, or Enterprise

Pupil Radius - estimated physical radius of the pupil.

In the SDK this is in meters (SDK always uses meters for consistency).

In the Gaze Recorder this is in millimeters (which is a more natural unit for researchers).

  • SDK: Headset::getPupilRadius
  • Gaze Recorder: Pupil Radius (millimeters) [left/right]
  • License: Basic, Pro, or Enterprise

Gazed Object - the name of the object in the scene that is being gazed at.

In the SDK, this is the integer ID of an object registered via Headset::registerGazableObject. If the client does not register any scene objects via the API, then this will always be the invalid object ID.

In the Gaze Recorder, this field is the name of the Scene Object being gazed at. You need to add the GazableObject component to any objects you want to be tracked.

This is part of the FOVE SceneAware system, and allows the client to avoid having to do it's own gaze casts. The client is free to do it's own ray tests to determine gazed objects, for example if it wants to do occlusion tests. However, allowing FOVE to handle gaze object testing is recommended because we can handle edge cases, heuristics, noise, etc in a central place rather than having each app do with it.

Eye Torsion - angle of the eye rotation around the line of sight, in degrees. Torsion data is reliable only when the user is looking forward. A special calibration is required for the torsion data to be available.

A value of 0 means no rotation with relation to calibration. From there, the values represent a clockwise rotation as seen from the camera image, currently with a range limit from -17 to 17 degrees. Since the reference is the camera, slippage of the headset will induce change in the output, so it’s important to ensure the headset is mounted well when using torsion data.

User Presence - whether the user is wearing the headset.

In FOVE0 this is handled by lack of detection of an eye in the camera image. False positives are possible, so please test when you use this field. In future hardware, this may be checked by proximity sensor or other means.

User Attention Shift - whether the user is currently performing a saccade

IPD - interpupillary distance. This is constantly updated in real time as the eyes rotate. When the user is converged on a close object, the IPD becomes smaller.

In the SDK this is in meters (SDK always uses meters for consistency).

In the Gaze Recorder this is in millimeters (which is a more natural unit for researchers).

IOD - interocular distance, or the estimated distance between rotational centers of left and right eyes.

This is a biometric parameter of the user that is essentially fixed per person. The value itself coming out of the eye tracker may vary over time, as the eye tracker continually updates its estimation.

FOVE specifically distinguishes IPD and IOD because our IPD value updates in real time as the distance between the pupils change. Outside of FOVE, IPD is used sometimes to refer only to the maximum (non-converged) distance between the pupils. We use it to refer to the current distance at any given time, including during convergence.

IOD can generally be thought of as the maximum value of IPD, because human eyes typically do not diverge, but this is not necessarily true. The difference in visual-to-optical axis can affect this. Furthermore, the two values are independently calculated within the FOVE eye tracker, and thus may have inaccuracies in different directions, so this relation should not be directly relied on.

In the rendering system, the FOVE0 uses a fixed rendering IOD of 63mm (see Headset::getRenderIOD) because this is the fixed distance between the center of the lenses. This may change with future hardware that includes a hardware adjustment. This means users with a larger IOD will perceive objects as slightly closer, and users with a smaller IOD will perceive objects as farther. Generally, this effect is minor and can be ignored, but it's worth mentioning.

In the SDK this is in meters (SDK always uses meters for consistency).

In the Gaze Recorder this is in millimeters (which is a more natural unit for researchers).

gaze_recorder_ipd_iod.png

Eyeball Radius - estimated distance between the cornea surface and the rotational center of the eye.

In the SDK this is in meters (SDK always uses meters for consistency).

In the Gaze Recorder this is in millimeters (which is a more natural unit for researchers).

Screen Gaze - estimated point on the physical screen of the headset where the line of sight for an individual eye intersects it. These points are defined in a separate coordinate system different from the HMD / Local / World systems. Each eye has its own coordinate system with an origin in the center of the HMD screen portion devoted to the eye, x axis to the right and y axis to the top. The values are scaled to the [-1,1] range.

This value can be easily derived from the Gaze Vectors by projecting them through the camera frustum (see Headset::getProjectionMatricesLH), but is provided as a convenience for users who simply want a "2D Gaze Point".

Eye Shape - 2D outline of the eyelids on the eye camera image, defined as a sequence of points in pixels, with (0,0) corresponding to the top-left corner of the image. Point 0 is the inner corner of the eye (closer to the nose), point 6 is the outer corner of the eye (farther from the nose). Points 0 to 6 correspond to the lower eyelid, going from the inner corner towards the outer corner. Points 6 to 11 correspond to the upper eyelid, going from the outer corner towards the inner corner.

  • SDK: Headset::getEyeShape
  • Gaze Recorder: Eye Shape [left/right] point [0-11] [x/y]
  • License: Pro or Enterprise

mceclip0.jpg

 

Pupil Shape - 2D outline of the pupil on the eye camera image, defined as an ellipse with a center and size in pixels starting from the top-left corner of the image, and a rotation angle of the ellipse axes in degrees. The angle does not represent eye torsion, please see the eye torsion section for that.

  • SDK: Headset::getPupilShape
  • Gaze Recorder: Pupil Shape [left/right] [center x/center y/size x/size y/angle]
  • License: Pro or Enterprise

Eye Image - uncompressed image of the eyes. 

The left and right eye images are stitched together into one image. You may combine multiple frames acquired through this method to create a video.

This data is returned internally with a standard 54-byte bitmap header. You can directly write it to a .BMP file and it will be recognized by other programs. However, this means you need to skip ahead a little if you want to read the actual pixel data.

The Unity and Unreal plugins do it for you, but when using the SDK directly, we highly recommend reading the header rather than assuming the data format and size, because it may change. One particular thing to look out for is that the height may be negative, indicating a top-down bitmap.

Please note that this image can contain biometric data regarding the user. We encourage handling it with care.

  • SDK: Headset::getEyesImage
  • Gaze Recorder: Not part of the Gaze Recorder (but available in the Unity plugin separately)
  • License: Basic, Pro or Enterprise

eyes.jpg

 

Unity Gaze Recorder Specific Fields

Application Time - time in seconds from the start of the application

User Mark - indicates that user pressed a button. This field can be used for storing any external application-specific events

 

Related articles

Comments

0 comments

Please sign in to leave a comment.