Subject 00

Subject 01 

Subject 02 

Subject 03 

Subject 04 

Subject 05 

Subject 06 

Subject 07 

Subject 08 

Subject 09 

Table of All Subjects 

Survey Data 

See also: notes on file structures and code for parsing data

Note about finger-tracking data: The Manus gloves may not have been working reliably for all subjects in the current set, so we recommend caution if using the finger-tracking data. The gloves have since been repaired, and we will record data from more subjects soon.

Notes about finger-tracking and tactile data: To provide some data where the hand is visible, Subjects 06 through 09 did not wear finger-tracking gloves or tactile hand sensors. We have also recently upgraded the custom tactile sensors, so tactile data from future subjects is expected to be more robust.

General note: We plan on recording from at least 10 more subjects in the coming months, and are still finalizing how data and processing scripts are presented, so feel free to pass along any comments or suggestions about what would be most beneficial for your research! We're happy to help however we can.

Data for all subjects (see subject-specific sections for more information)

Subject Data visualization Wearable sensors First-person video [with gaze overlay] First-person video [no gaze overlay] Global: RGB cameras Global: depth camera Global: microphones
S00 YouTube Split 1
Split 2 		Split 1
Split 2 		Split 1
Split 2 		Split 1 videos: C01 | C02 | C03 | C04 | C05
Split 1 times : C01 | C02 | C03 | C04 | C05
Split 2 videos: C01 | C02 | C03 | C04 | C05
Split 2 times : C01 | C02 | C03 | C04 | C05 		[coming soon]
 Split 1 audio: M01 | M02
Split 1 times: M01 and M02
Split 2 audio: M01 | M02
Split 2 times: M01 and M02
S01 YouTube Split 1
(only
calibration) 		[unavailable] [unavailable] Split 1 videos: C01 | C02 | C03 | C04 | C05
Split 1 times : C01 | C02 | C03 | C04 | C05 		[coming soon]
 Split 1 audio: M01 | M02
Split 1 times: M01 and M02
S02 YouTube Split 1
Split 2
Split 3
Split 4 		Split 1
Split 2
Split 3
Split 4 		Split 1
Split 2
Split 3
Split 4 		Split 1 videos: C01 | C02 | C03 | C04 | C05
Split 1 times : C01 | C02 | C03 | C04 | C05
Split 2 videos: C01 | C02 | C03 | C04 | C05
Split 2 times : C01 | C02 | C03 | C04 | C05
Split 3 videos: C01 | C02 | C03 | C04 | C05
Split 3 times : C01 | C02 | C03 | C04 | C05 		[coming soon]
 Split 1 audio: M01 | M02
Split 1 times: M01 and M02
Split 2 audio: M01 | M02
Split 2 times: M01 and M02
S03 YouTube Split 1
Split 2 		Split 1
Split 2 		Split 1
Split 2 		Split 1 videos: C01 | C02 | C03 | C04 | C05
Split 1 times : C01 | C02 | C03 | C04 | C05
Split 2 videos: C01 | C02 | C03 | C04 | C05
Split 2 times : C01 | C02 | C03 | C04 | C05 		[coming soon]
 Split 1 audio: M01 | M02
Split 1 times: M01 and M02
S04 YouTube Split 1 Split 1 Split 1 Split 1 videos: C01 | C02 | C03 | C04 | C05
Split 1 times : C01 | C02 | C03 | C04 | C05 		[coming soon]
 Split 1 audio: M01 | M02
Split 1 times: M01 and M02
S05 YouTube Split 1
Split 2 		Split 1
Split 2 		Split 1
Split 2 		Split 1 videos: C01 | C02 | C03 | C04 | C05
Split 1 times : C01 | C02 | C03 | C04 | C05 		[coming soon]
 Split 1 audio: M01 | M02
Split 1 times: M01 and M02
S06 YouTube Split 1
Split 2 		Split 1
Split 2 		Split 1
Split 2 		[coming soon]
 [coming soon]
 [coming soon]
S07 YouTube Split 1 Split 1 Split 1 [coming soon]
 [coming soon]
 [coming soon]
S08 YouTube Split 1 Split 1 Split 1 [coming soon]
 [coming soon]
 [coming soon]
S09 YouTube Split 1
Split 2
Split 3 		Split 1
Split 2
Split 3 		Split 1
Split 2
Split 3 		[coming soon]
 [coming soon]
 [coming soon]

Data for Subject S00


Note: Chapters in the YouTube progress bar denote activity labels.


Notes

  • Recording was restarted when Xsens required recalibration.
  • Recording splits for wearable sensors and eye-tracking:
    • (1) calibration
    • (2) all activities
  • Recording splits for environment-mounted cameras:
    • (1) calibration
    • (2) all activities
  • Recording splits for environment-mounted microphones:
    • (1) calibration
    • (2) all activities

Wearable Sensor Data

Eye-Tracking First-Person Video (may need to right-click on a link to Save)

Environment-Mounted RGB Cameras

Environment-Mounted Depth Camera

  • Table Camera: recording

Environment-Mounted Microphones

Data for Subject S01


Note: Chapters in the YouTube progress bar denote activity labels.


Notes

  • Due to a technical snafu, only environment-mounted sensor data is available for all activities after calibration and peeling cucumbers (the wearable sensor data was lost).
  • Recording splits for wearable sensors and eye-tracking:
    • (1) all available data
  • Recording splits for environment-mounted cameras:
    • (1) all data
  • Recording splits for environment-mounted microphones:
    • (1) all data

Wearable Sensor Data

Environment-Mounted RGB Cameras

Environment-Mounted Depth Camera

  • Table Camera: recording

Environment-Mounted Microphones

  • Overhead (Omnidirectional): recording split 1
  • Sink (Directional): recording split 1
  • Timestamps for both microphones: recording split 1

Data for Subject S02


Note: Chapters in the YouTube progress bar denote activity labels.


Notes

  • Recording was restarted when Xsens required recalibration.
  • Recording splits for wearable sensors and eye-tracking:
    • (1) calibration
    • (2) peel, slice, spread
    • (3) open/close jar, clean, set table
    • (4) stack, load/unload dishwasher
  • Recording splits for environment-mounted cameras:
    • (1) calibration, peel, slice, spread
    • (2) open/close jar, clean, set table
    • (3) stack, load/unload dishwasher
  • Recording splits for environment-mounted microphones:
    • (1) calibration, peel, slice, spread, open/close jar, clean, set table
    • (2) stack, load/unload dishwasher

Wearable Sensor Data

Eye-Tracking First-Person Video (may need to right-click on a link to Save)

Environment-Mounted RGB Cameras

Environment-Mounted Depth Camera

  • Table Camera: recording

Environment-Mounted Microphones

Data for Subject S03


Note: Chapters in the YouTube progress bar denote activity labels.


Notes

  • Recording splits for wearable sensors and eye-tracking:
    • (1) calibration, peel, slice
    • (2) all remaining
  • Recording splits for environment-mounted cameras:
    • (1) calibration, peel, slice
    • (2) all remaining
  • Recording splits for environment-mounted microphones:
    • (1) all data

Wearable Sensor Data

Eye-Tracking First-Person Video (may need to right-click on a link to Save)

Environment-Mounted RGB Cameras

Environment-Mounted Depth Camera

  • Table Camera: recording

Environment-Mounted Microphones

  • Overhead (Omnidirectional): recording split 1
  • Sink (Directional): recording split 1
  • Timestamps for both microphones: recording split 1

Data for Subject S04


Note: Chapters in the YouTube progress bar denote activity labels.


Notes

  • Recording splits for wearable sensors and eye-tracking:
    • (1) all data
  • Recording splits for environment-mounted cameras:
    • (1) all data
  • Recording splits for environment-mounted microphones:
    • (1) all data

Wearable Sensor Data

Eye-Tracking First-Person Video (may need to right-click on a link to Save)

Environment-Mounted RGB Cameras

Environment-Mounted Depth Camera

  • Table Camera: recording

Environment-Mounted Microphones

  • Overhead (Omnidirectional): recording split 1
  • Sink (Directional): recording split 1
  • Timestamps for both microphones: recording split 1

Data for Subject S05


Note: Chapters in the YouTube progress bar denote activity labels.


Notes

  • Recording splits for wearable sensors and eye-tracking:
    • (1) calibration
    • (2) all activities
  • Recording splits for environment-mounted cameras:
    • (1) all data
  • Recording splits for environment-mounted microphones:
    • (1) all data

Wearable Sensor Data

Eye-Tracking First-Person Video (may need to right-click on a link to Save)

Environment-Mounted RGB Cameras

Environment-Mounted Depth Camera

  • Table Camera: recording

Environment-Mounted Microphones

  • Overhead (Omnidirectional): recording split 1
  • Sink (Directional): recording split 1
  • Timestamps for both microphones: recording split 1

Data for Subject S06


Note: Chapters in the YouTube progress bar denote activity labels.


Notes

  • Recording was restarted after peeling potatoes, just to double check that the Myos were still streaming correctly after being re-synced
  • Recording splits for wearable sensors and eye-tracking:
    • (1) calibration, peel, slice
    • (2) all remaining
  • Recording splits for environment-mounted microphones:
    • (1) all data

Wearable Sensor Data

Eye-Tracking First-Person Video (may need to right-click on a link to Save)

Environment-Mounted RGB Cameras

  • RGB Camera 1
    • RGB Camera 2
      • RGB Camera 3
        • RGB Camera 4
          • RGB Camera 5

            Environment-Mounted Depth Camera

            • Table Camera: recording

            Environment-Mounted Microphones

            • Overhead (Omnidirectional): recording
            • Sink (Directional): recording
            • Timestamps for both microphones: recording

            Data for Subject S07


            Note: Chapters in the YouTube progress bar denote activity labels.


            Notes

            • Recording splits for wearable sensors and eye-tracking:
              • (1) all data
            • Recording splits for environment-mounted microphones:
              • (1) all data

            Wearable Sensor Data

            Eye-Tracking First-Person Video (may need to right-click on a link to Save)

            Environment-Mounted RGB Cameras

            • RGB Camera 1
              • RGB Camera 2
                • RGB Camera 3
                  • RGB Camera 4
                    • RGB Camera 5

                      Environment-Mounted Depth Camera

                      • Table Camera: recording

                      Environment-Mounted Microphones

                      • Overhead (Omnidirectional): recording
                      • Sink (Directional): recording
                      • Timestamps for both microphones: recording

                      Data for Subject S08


                      Note: Chapters in the YouTube progress bar denote activity labels.


                      Notes

                      • Recording splits for wearable sensors and eye-tracking:
                        • (1) all data
                      • Recording splits for environment-mounted microphones:
                        • (1) all data

                      Wearable Sensor Data

                      Eye-Tracking First-Person Video (may need to right-click on a link to Save)

                      Environment-Mounted RGB Cameras

                      • RGB Camera 1
                        • RGB Camera 2
                          • RGB Camera 3
                            • RGB Camera 4
                              • RGB Camera 5

                                Environment-Mounted Depth Camera

                                • Table Camera: recording

                                Environment-Mounted Microphones

                                • Overhead (Omnidirectional): recording
                                • Sink (Directional): recording
                                • Timestamps for both microphones: recording

                                Data for Subject S09


                                Note: Chapters in the YouTube progress bar denote activity labels.


                                Notes

                                • Recording was restarted after the initial eye-tracking calibration, due to Xsens requiring recalibration.
                                • The second recording includes the remaining calibrations and all activities.
                                • Recording was restarted before the concluding calibrations to ensure data was saved correctly.
                                • Recording splits for wearable sensors and eye-tracking:
                                  • (1) eye-tracking calibration
                                  • (2) remaining starting calibrations, all activities
                                  • (3) ending calibrations
                                • Recording splits for environment-mounted microphones:
                                  • (1) all data

                                Wearable Sensor Data

                                Eye-Tracking First-Person Video (may need to right-click on a link to Save)

                                Environment-Mounted RGB Cameras

                                • RGB Camera 1
                                  • RGB Camera 2
                                    • RGB Camera 3
                                      • RGB Camera 4
                                        • RGB Camera 5

                                          Environment-Mounted Depth Camera

                                          • Table Camera: recording

                                          Environment-Mounted Microphones

                                          • Overhead (Omnidirectional): recording
                                          • Sink (Directional): recording
                                          • Timestamps for both microphones: recording


                                          Survey Data

                                          Subject Pool:

                                          The subject pool recruited so far comprises 10 subjects. It is 70% male, 90% right-hand dominant, 70% right-eye dominant, and 27.3±3.7 years old. Note that the subject pool continues to grow, and that the dataset aims to reach 25 subjects over the next few months.

                                          Levels of Expertise:

                                          Subjects self-reported their levels of expertise on various tasks related to the activities explored in the dataset.

                                          Experience Wearing Sensors:

                                          Subjects rated their experience wearing each of the sensors. Comments on the body tracking system were highly variable, largely depending upon the strap tightness. Subjects generally found the eye tracker and the muscle sensor to be unobtrusive. The gloves and the tactile sensors could be improved and streamlined in the future to reduce their impact on task performance.

                                          Overall Workload:

                                          Across all subjects, the raw workload estimate using the NASA TLX averaged 3.40±1.62 (lower numbers indicate lower workloads, on a scale from 0 to 10, inclusive). This indicates that the workload was relatively low despite the mild sensor obtrusiveness discussed above.

                                          Visions of Robot Assistants:

                                          There was significant variability among desired roles for robot assistants. This spans both autonomous and collaborative cases, although there are a few tasks such as loading or unloading the dishwasher where an autonomous assistant would be greatly valued over a collaborative robot. Some considerations provided by the subjects include safety (especially regarding a robot wielding sharp objects), how much time the task would take to do manually, and a desire for the robot to anticipate what task or tool will be needed next.