A computer vision based virtual mouse

1. A Vision Based Application for Virtual Mouse Interface Using Finger -
Tip
Presented By:
Gaurav Kr. Tiwari – DDU8362000025
Abhishek Srivastava – DDU8362000002
Shubham Yadav – DDU8362000058
Under the guidance of
Mr. Anshuman Srivastava(Associate Professor)
Dept. of Computer Application
ITM College of Management, Gida, Gorakhpur
Deen Dayal Upadhyaya Gorakhpur University, Gorakhpur

2. Introduction
Introduction of our project goes here…
As computer technology continues to develop, people have smaller and smaller electronic devices and want to use
them ubiquitously (everywhere). There is a need for new interfaces designed specifically for use with these smaller
devices. The importance of computers has increased to a great extent these days. Then it can be used for general
purpose or at work places. Even there is a demand of more and more application-based devices, where the latest
example can be of smart phones.
Our system basically used image processing, Object detection to control the mouse activities such as its
movement, Left-Click, Right-Click. The user will simply hold his finger up, place it in the viewing area of the webcam
and thus controlling the mouse by finger-tip also the workspace required could be reduced.

3. Continue…
Introduction about Computer-Vision goes here…
Computer-Vision:
Computer vision is a field of Artificial Intelligence (AI) that enables computers and system to derive meaningful
information from digital images, videos and other visual inputs and take actions or make recommendations based
on that information.
Tools used for Computer-Vision: Open-CV
Open-CV:
• It stands for “Open Source Computer Vision”.
• It supports Python, Matlab, Java, etc.

4. motivation
Motivation of our project goes here…
• Firstly, the technological motivation is to build a vision based application for virtual
mouse interface using Finger-Tip.
• Secondly, It reduce hardware cost by eliminating use of mouse.
• Thirdly, It convenient for users not comfortable with touchpad.

5. Literature review
S. No. Author Name Description
1. 1. Monali Shetty,
2. Christina A. Daniel,
3.Manthan K. Bhatkar,
Department of Computer
Science, Fr. CRCE , Mumbai,
India.
This paper proposed a Computer vision-based
mouse cursor control system which uses hand
gestures.
2. 1. Aviral Gupta , MTech. Scholar,
2. Dr. Neeta Sharma , Assistant
Professor, Department of Computer
Science & Engineering,
Noida International University, India.
In this paper, author identified an alternative to
mouse command especially with reference to
cursor controlling applications.
3. 1. Aykut Erdem,
2. Erkut Erdem,
Middle East Technical
University, Ankara, Turkey.
In this research paper the author describes a
computer vision based mouse which can control
and command the cursor of a computer using a
camera.

6. Proposed methodology
1. Capturing the real time video: For the framework to work we require a sensor to distinguish the hand
developments of the client. The webcam of the PC is utilized as a sensor. The webcam catches the
settled of the PC is utilized as a sensor. The webcam catches the constant video at a settled casing rate
and determination which is controlled by the equipment of the camera. The casing rate and
determination can be changed in the framework if required.

7. continue
2. Flipping of Image:At the point when the camera catches a picture, it is upset. This implies on the off
chance that we move the shading pointer towards the left, the picture of the pointer moves towards the
privilege and the other way around. It’s like a picture acquired when we remain before a mirror. To
maintain a strategic distance from this issue we have to vertically flip the picture.

8. continue
3. Finger-tip Detection: This is the most critical stride in the entire procedure. The finger up , down, fist
shading article is recognized by subtracting the flipped shading smothered channel from the flipped
Gray- Scale Image. This makes a picture which contains the recognize protest as a fix of dim
encompassed by dark space.
4. Performing Clicking Actions: The control activities of the mouse are performed by controlling the
banners related with the mouse catches. The client needs to perform hand motions keeping in mind
the end goal to make the control activities. Because of the utilization of shading pointers, the
calculation time required is lessened. Besides the framework ends up noticeably impervious to
foundation clamor and low light conditions.

9. Implementation and result
Module No. Module Contents Month
1. Basic Video Operations using Open CV.
March
2. Advanced Image Operations using Open CV.
April
3. Object Tracking using Open CV.
May
4. Detect Location of finger-tip using OpenCV.
June
5. Computer Vision Based Mouse.
July

10. Module - 1
In module 1, we have done the Basic Video Operations using Open CV.
Basic Video Operations:-
1. Capture video in Open CV:-
In this we capture the video using the camera of our PC. To capture a video
we use the Video Capture Function.
Video Capture Function:-
• Syntax:- cv2.videoCapture(Argument)
Steps to capture a video :-
• Use cv2.videoCapture() to get a video capture object for the camera.
• Set up an infinite while loop and use the read() method to read the
frames using the above created object.
• imshow() method to show the frames in the video.
• Breaks the loop when the user clicks a specific key like ESC.
2. Play video in OpenCV
3. Save video as a file using OpenCV

11. Module - 2
In module 2, we have done the Advanced Image Operations using OpenCV.
Advanced Image Operations:-
1. Smoothing an image:-
In this we have done how to blur and smooth our images. Image blurring is useful for removing noise from an
images.
2. Simple Thresholding of Image using OpenCV:-
In this we have done the thresholding of an image. Thresholding is used to create the Binary images from Gray
scale images which can further be used for Object detection and Identification.

12. Module - 3
In module 3, we have done Object Tracking and contours using OpenCV.
1. Object Tracking:- Object tracking is an algorithm that tracks the displacement of one of several
particular objects using cameras to capture a scene.
Pipeline for Object Tracking:-
Image Acquisition
Image Preparation
Gesture Recognition
Object Identification

13. continue
1. Contours in Open CV:-
In this we find and draw the contour in an Image and calculate the features of
contour in OpenCV. Contours can be explained simply as a curve joining all the
continuous points(along the boundary), having same color or Intensity. The
contour are a useful tool for shape analysis and object detection and
recognition.
Syntax:- cv2.findContours(img,mode,method)
where,
img:- Source image.
mode:- Contour retrieval mode.
method:- Contour approximation method.

14. Module - 4
In module 4, we have done the Tracking of a Finger-tip and detect the location of an object using OpenCV.
1. Tracking of a Finger-tip using OpenCV :-
In this we have done the tracking of a Finger-tip using OpenCV. Here we use
our fingers for tracking.
2. Detect location of finger-tip :-
In this, we have done how to detect the location of finger-tip.

15. Module - 5
In module 5, we have done Computer Vision based Mouse.
1. Computer Vision based Mouse :-
A computer vision based mouse is a system to control the cursor without
any physical device. Here we have done the Tracking markers and their
motion for CV based mouse and also Translate the marker motion to
Cursor motion.
• Two fingers up- Cursor movements.
• Put down the index finger- Left click.
• Put down the middle finger- Right click.
• Fist- Drag
• Palm- Stop the cursor
• Left Pinch- Brightness Control
• Right Pinch- Scroll

16. continue
Pipeline for Computer vision based Mouse:-
Image Acquition
Image Preparation
Gesture Recognition
Fingers Motion Tracking
Cursor Control

17. continue
2. Cursor control using PyautoGUI :-
Pyautogui is a python module for programmatically controlling the mouse and
keyboard of our computer. In this we have done how to move the mouse and
track its position on the screen using pyautogui.
Pyautogui is also have a FAILSAFE feature. The FAILSAFE feature was stop the program if
we quickly move the mouse for up and left. Here we can disabled this feature by setting
pyautogui.FAILSAFE = False.
Syntax :-
pyautogui.FAILSAFE = False

18. continue
Result :-
The proposed paper is on controlling of mouse functions using finger-tips. The functions are Mouse
Movement, Left Clicking, Right Clicking, Drag & drop, Brightness control, Scroll up & down, etc. In this system,
the user can show two finger (Index and Middle) for movement of cursor and putting one of the finger down
to performing clicking actions. Our main aim was to reduce the hardware components. Although the
application can be run in different environment.
Mouse events and Evaluation :-
Gesture Input Mouse Events Accuracy
Two fingers up Movement ~ 85%
Putting index finger down Left click ~ 80%
Putting middle finger down Right click ~ 80%
Left pinch Scroll ~ 60%
Right pinch Brightness control ~ 60%
Fist Drag ~ 85%

19. conclusion
The proposed system is used to control the mouse cursor and implement its function using a real-time
camera. We implemented Mouse Movement , Selection of the Icons and its Functions like Right, Left Click,
Drag-Drop, scroll. This system is based on image comparison and motion detection technology to do mouse
pointer movements and selection of icon. From the results, we can expect that if the algorithms can work
in all environments then our system will work more efficiently.
This system could be useful in presentations and to reduce work space. Future work of this
project includes making the Finger tip detector module invariant to illumination changes and 3D pose
estimation of panel which can be used for Augmentation reality of 3D objects.

20. Reference
1. Monali Shetty , Christina A. Daniel , Manthan K. Bhatkar, “Virtual mouse using Object tracking”,
Department of Computer Science, Fr. CRCE, India. ICCES(2020).
2. Deeksha Verma , Dr. Pankaj Sharma, “Vision based computer mouse controlling using Hand Gesture”,
Department of Computer Science & Technology, SIT, Mathura, India. IJESRT(2018).
3. Yash Velaskar , Akshay Dulam, “Computer vision based hand gesture interfaces”, Department of
Information Technology, Atharva College of Engineering, Mumbai, India. IJIRCCE(2017).
4. Sandeep Thakur, Rajesh Mehra, Buddhi Prakash, “Vision based computer mouse control using hand
gesture.” Department. of Electronics & Communication Engineering, National Institute of
Technical Teachers’ Training & research Chandigarh, UT, India. IEEE (2015).

21. Thank you