iSET - the "Interactive Social-Emotional Toolkit" - is a system for recording and live annotation of video to teach emotion recognition to persons with Autism Spectrum Disorders.
iSET is being developed at the MIT Media Lab in the Affective Computing Group by Mish Madsen, Micah Eckhardt, Rana el Kaliouby, Matthew Goodwin, and Rosalind Picard. Questions may be sent to Mish Madsen.
The iSET system consists of custom-built software for video recording, annotation, and playback, written in C++ and Open CV, running on a handheld ultramobile Samsung computer.
1) Hold the iSET system (which automatically records video) so that it records your conversation partner's face. Tap customizable emotion-label buttons with thumbs to add labels to your video.
Default Emotion Labels:
Simple (from Ekman): angry, disgusted, afraid, sad, surprised, happy
Complex (from Baron-Cohen Mind Reading DVD): disagreeing, thinking, unsure, interested, concentrating, agreeing.
2) Replay your conversation on the iSET software. Delete or add labels; use the timeline to easily skip between labeled parts of the video.
Default Emotion Labels:
Simple (from Ekman): angry, disgusted, afraid, sad, surprised, happy
Complex (from Baron-Cohen Mind Reading DVD): disagreeing, thinking, unsure, interested, concentrating, agreeing.
2) Replay your conversation on the iSET software. Delete or add labels; use the timeline to easily skip between labeled parts of the video.
back to top
Where is iSET currently being used?
iSET is currently part of a pilot intervention at the Groden School in Rhode Island. The participant groups and overall protocol for this intervention are as seen below:
The following metrics are being used as pretests and posttests for the ASD subject groups (and as pretests for the neurotypical subject group): SRS, K-Bit, TAS-20, ADOS, Benton Face Test, AU recognition test, Mind Reading DVD clips (iSET intervention emotions) and iSET-recorded clips of iSET intervention emotions.
The following metrics are being used as pretests and posttests for the ASD subject groups (and as pretests for the neurotypical subject group): SRS, K-Bit, TAS-20, ADOS, Benton Face Test, AU recognition test, Mind Reading DVD clips (iSET intervention emotions) and iSET-recorded clips of iSET intervention emotions.
back to top
What are the goals of the iSET intervention?
The iSET intervention aims to teach children and adolescent with autism spectrum disorders how to process, understand, and naturally respond to nonverbal communication conveyed by facial expressions.
back to top
What other projects are associated with the iSET intervention?
The VidL project (VidL.media.mit.edu), also developed in the Affective Computing group, is being used to label and organize the video collected through the iSET intervention.
The iCalm system is used to collect autonomic nervous system data, signaling levels of excitement or stress, and is also part of the iSET intervention.
What work has been published about iSET?
Some of our publications are listed here.