MediaPipe Gesture Recognition

Demo on Android Device

Unable to display the GIF. It is a demonstration showing the results of MediaPipe gesture recognition.

Overview

Provides the recognized hand gesture results along with the landmarks of the detected hands.

• Input
  • Image
  • Video
  • Stream
• Output
  • Categories of hand gestures
  • Handedness of detected hands
  • Landmarks of detected hands in image coordinates
  • Landmarks of detected hands in world coordinates

Categories of Hand Gestures

• 0: Unrecognized gesture, label: Unknown
• 1: Closed fist, label: Closed_Fist ✊
• 2: Open palm, label: Open_Palm 👋
• 3: Pointing up, label: Pointing_Up ☝️
• 4: Thumbs down, label: Thumb_Down 👎
• 5: Thumbs up, label: Thumb_Up 👍
• 6: Victory, label: Victory ✌️
• 7: Love, label: ILoveYou 🤟

Landmarks of Detected Hands

Unable to display the image, it is a diagram illustrating the keypoint localization of 21 hand-knuckle coordinates within the detected hand regions.

Task Configuration Options

• runningMode
  • Options: IMAGE, VIDEO, LIVE_STREAM.
• numHands
  • Default is 1.
• minHandDetectionConfidence
• minHandPresenceConfidence
• minTrackingConfidence
  • This is the bounding box IoU threshold between hands in the current frame and the last frame. If the tracking fails, Gesture Recognizer triggers hand detection. Otherwise, the hand detection is skipped.
  • Gesture Recognizer uses tracking to avoid triggering palm detection model on every frame, which helps reduce the latency of Gesture Recognizer.
• cannedGesturesClassifierOptions
• customGesturesClassifierOptions
• resultListener
  • Only be used when running mode is LIVE_STREAM.
• errorListener

Key Components of Code

• Sample code
• Key Components
  • GestureRecognizerHelper.kt - Initializes the gesture recognizer and handles the model and delegate selection.
  • MainActivity.kt - Implements the application, including calling GestureRecognizerHelper and GestureRecognizerResultsAdapter.
  • GestureRecognizerResultsAdapter.kt - Handles and formats the results.

Custom Model

If you want to improve or alter the capabilities of the models provided in this task, you can use Model Maker to modify the existing models. Custom models used with MediaPipe must be in the .task format, which is a model bundle file. You should consider using Model Maker to modify the provided models for this task before building your own.

For more information about customizing a model for this task, see Customize models for Gesture Recognizer.

Reference

• MediaPipe gesture recognition overview
• Gesture recognition guide for Android
• Android sample code
• Hand gesture recognition model customization guide