Monday, April 2, 2012

Accelerometer Based Gesture Recognition - Integration With Windows Paint

This article delineates, yet another application of Gesture Recognition Framework by integrating it with Windows Paint application. The idea behind this work is to provide economic solutions for smart classes which makes interactive teaching very imminent. The application is just not confined by few use cases, but further enhancements will lead to  opportunities for wider applications like gesture to text conversion, presentations involving multiple participants, image viewer control etc. 

Project Outline

Paint Brush is an accelerometer controlled painting application, which enables user to draw basic line diagrams in 2D space . Inputs for this framework are the gestures generated from user's handheld device, as the user tilts/waves his cell phone in the air.
The project currently is in it's primitive stage of development which renders user's gestures only as horizontal and vertical lines, but further enhancements makes it feasible to draw various geometrical shapes, curves and free hand drawing as well.

Demonstration 



Domain Model

 

Implementation

Accelerometer based Paint Brush application consists four modules,
  • Accelerometer module on mobile device
The accelerometer module running on user’s mobile will initiate Accelerometer hardware and start listening to accelerometer output data which returns the acceleration due to gravity of the device in three dimensions. This is helpful in understanding the inclinations of the device due to tilting gestures and also the direction in which device is moving. This information is fed to Gesture Recognition Algorithm which processes raw accelerometer data values into useful line patterns which is drawn within Paint Brush window.
  • Bluetooth server running on user's PC
Bluetooth server running on user’s PC will publish text/file transfer services, and these services are discovered by Bluetooth client application running on user’s device. Once the service is discovered the client will establish a connection with server and start pushing accelerometer data to server socket. Server will continue to receive Bluetooth client data using a series of “Receive” function calls.
  • Bluetooth client running on mobile device
Bluetooth client running on mobile device will search for peer Bluetooth servers publishing various services, In this case the client will search for file/text transfer services. After discovering the service client makes a connection with Bluetooth server and starts sending accelerometer data to server using series of “send” function calls.
  • Gesture Recognition Framework for Painting
Bluetooth server module feeds painting framework with Gestures pumped continuously from user's handheld device. The framework gets hold of target painting application on which these gestures are executed to achieve required functionality. Each gesture is translated distinct draw event  which ultimately renders desired shapes on the screen.

Future Enhancements
  • Enable drawing of curves, and other geometrical shapes.
  • Text rendering
  • Multicolor support
  • Adjustable brush/pencil size.
Suggestions and feedback on this work is most appreciated. I'm reachable at mail2vijaydr@gmail.com

Thursday, March 15, 2012

Accelerometer Based Gesture Recognition - Integration with NFS on Desktop

Project Outline
  
With the innovations in telecom domain and sensor technologies, the handheld devices of these days are equipped with various sensors which provide sophisticated functionalities and makes gadgets very powerful and easy to work with.

Accelerometer is one of the sensors which are commonly available on handheld devices like mobile phones, tablet PCs, notebooks etc. With the integration of gesture recognition framework for mobile applications, it provides intuitive ways of interaction with a rich user experience and it also makes things simple but interesting. Accelerometer can be used for wide range of applications and games on mobile devices and can be integrated with desktop applications.

The goal of the project is to build a gesture recognition framework based on accelerometer and provide seamless integration with any of the desktop applications. With the integration of this framework for desktop applications user can remotely play games, create drawing, control key event based applications. And the framework is so robust and flexible that this being a generic framework , can be integrated with any target application it really doesn't matter whether the application is new, or legacy , whether the application exposes APIs or not, no matter what this framework can be integrated with any target application and this doesn’t even need any changes from application side.

Demonstration
 

Domain Model



Implementation Details
 
Racing games is one of the applications of accelerometer's integration with desktop applications. Wherein  user will be able to control racing objects using gestures from his handheld device.  Also user can steer the car to  left, right, front, back just by corresponding gestures from his mobile device.

NFS racing game with Gesture Recognition Framework comprises of four modules.
  • Accelerometer module on mobile device
The accelerometer module running on user’s mobile will initiate Accelerometer hardware and start listening to accelerometer output data which returns the acceleration due to gravity of the device in three dimensions. This is helpful in understanding which way the user is tilting the phone, and this information is fed to Gesture Recognition Algorithm which processes raw accelerometer data values into useful steering directions. And this steering information will control racing objects on user's PC.  
  • Bluetooth server running on user’s PC
    Bluetooth server running on user’s PC will publish text/file transfer services, and these services are discovered by Bluetooth client application running on user’s device. Once the service is discovered the client will establish a connection with server and start pushing accelerometer data to server socket. Server will continue to receive Bluetooth client data using a series of “Receive” function calls.
    • Bluetooth Client running on mobile device
    Bluetooth client running on mobile device will search for peer Bluetooth servers publishing various services, In this case the client will search for file/text transfer services. After discovering the service client makes a connection with Bluetooth server and starts sending accelerometer data to server using series of “send” function calls. 
    •  Gesture Recognition Framework
    The Gesture Recognition Framework running on user's PC will receive gesture information from Bluetooth Server module. The intelligence is built into this framework so that it knows the ways to interact and take control of target application. Once the framework gets control of the target application user can remotely operate on the target application using his handheld device as joystick.

    Future Enhancements
    • Multi player options
    • NFC integration
    • Enabling tap events along with Gestures for enhanced control.
    Suggestions and feedback on this project is most appreciated and encouraging for me. Also if any of the developers out there interested in working on these future enhancements can reach me out at mail2vijaydr@gmail.com

    Wednesday, March 14, 2012

    Live Your Dream and Share Your Passion

    Innovations and technological advancements have never seized to amaze me, every single innovation of today was once a distant dream of mankind. Hats off to the magnanimous efforts of those fanatic scientists and research scholars who lived the dream of making this earth, a better place to live.


    The Holstee Manifesto Lifecycle Video from Holstee on Vimeo.

    It isn't about being flabbergasted by looking at breakthrough innovations and expressing astonishment of  " How did they do it ? ",  but rather how can we build better systems and enhance existing ones for the love of serving mankind. In these lines,  I have been working on couple of hobby projects and here is my efforts to present my work to you to have a hands on and to provide your suggestions for improvements and feedback.

    I express my gratitude to all my colleagues, mentors and technical forum members who have had and have been assisting me in these works with their valuable suggestions and feedback.

    Share This Article