Robust, Markerless Tracking of People and Gestures for Interactive Systems (REMUS)

In-House Development

  • Development period: since 2008
  • Code base: about 300.000 lines of code

    • In the REMUS project we are realizing own ideas. By cooperating with the Fraunhofer IGD Rostock we keep up with the latest developments and research results in the field. Our experiences in optical driver assistance systems help us to implement innovative algorithms performing in real-time using minimal resources.

    Problem Description

    The vision of natural man-machine interaction gets along without any supplementary devices such as pointing devices or special markers. Many potential applications, for instance around digital signage, are only imaginable that way.

    Gestures represent one of the humans’ most sophisticated means to interact. Gestures can be observed optically by a machine. Prior to gesture recognition, however, many sub-problems need to be addressed. It starts by creating a 3D-environmental model from 2D images. Based on the 3D-model we are able to perform effective separation of people, their heads as well as their hands even if the environment is quickly changing.

    Project Results

    REMUS Demonstrator Enlarge

    In the course of the project algorithms have been developed that extract the following information using a stereo camera setup:

    • Varying number of people in a configurable observation cube
    • Estimation of 3D-pose and velocity-vectors of heads and hands of people inside an observation cube
    • Stable selection of a person obtaining the interaction focus out of a group of people
    • Supply of image regions containing the interacting head and hand to be used in further processing steps.

    The Fraunhofer IGD Rostock implemented, based on the extracted interacting hand region, a processing chain delivering static as well as dynamic hand gestures.

    All algorithms are executed in a data-flow oriented framework developed by FORTecH in the course of the project. The framework provides features useful to a broader class of real-time image processing solutions:

    • Synchronization of arbitrary, possibly sporadic data sources to a common cycle time
    • Integration of an arbitrary number of processing algorithms supplied as COM components
    • A generic tracking component able to keep track of many alternative hypotheses in parallel
    • Support for synchronous as well as asynchronous data flow elements
    • Adaptive scaling to different numbers of processor cores
    • Optimal performance even under system overload situations
    • Robust handling of missing sensor data

    A demonstrator has been presented at our booth at the embedded world 2011 and at the exhibition accompanying the Go-3D 2010 as well as the Go-3D 2011.

    Parts of the REMUS algorithmic library have been used successfully for educational projects by the University of Applied Sciences Augsburg.