O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Augmented Reality based Product Identification and Advertising System - Final Year Project (Udara Alwis)

1.912 visualizações

Publicada em

This was my Final Year Project thesis for my Software Engineering Degree at Staffordshire University, UK.
An Augmented Reality based Real-time Products and Advertisements identification System developed upon, Real-Time Image Processing, Machine Learning, Computer Vision Technologies, SURF (Speeded Up Robust Feature Detection) Feature detection algorithm, Android, Cloud technology. The .Net platform for the Server side implementation and Java Android development for Client side development was being used.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Augmented Reality based Product Identification and Advertising System - Final Year Project (Udara Alwis)

  1. 1. AUGMENTED REALITY BASED PRODUCT IDENTIFICATION AND ADVERTISING SYSTEM - UDARA ABHILASH ALWIS (CB003982)
  2. 2. INTRODUCTION
  3. 3. PROBLEM BACKGROUND Problem Statement There are huge product varieties in the market and advertising has a great impact in today’s life. Customers are always looking for extended information when they buy products or even regarding an advertisement, whereas it is very crucial to provide users with updated and accurate information regarding them. QR Codes are being deprecated in modern end consumerism due to its disadvantages and real time visual recognitions technologies are coming up with powerful mobile devices. Augmented Reality is becoming the next mass media advertising in today’s world.
  4. 4. PROBLEM BACKGROUND • Product information could change anytime • Advertisements can not display all the information at once • Customers are looking for more extended information • No proper method to look for information, time waste • Augmented Reality as the next mass advertising media • Extinction of QR Codes from end user consumerism
  5. 5. PROBLEM OBJECTIVES • Identify Product or Advertisement • View the extended information, and related advertisements or products • Implement Augmented Reality based concepts in the application • Insert Products and Advertisements information as Administrators • Simulate Product or Advertisement Identification process in admin system.
  6. 6. SOLUTION OUTLINE • Capture image when the user points at a product or an advertisement. • Identify the product or the advertisement from the server side. • Provide extended information about the product or the advertisement to the user. • Present the information in according to Augmented Reality principles. • Allow admins to add new information to the System.
  7. 7. MAIN CHALLENGES • Implementing Augmented Reality into the Mobile Application. • Getting the identification results from the server from the fastest way possible. • Implementing an Image Matching Algorithms. • Identifying a product or an advertisement by its Brand Logo. • Adding and maintaining Product and Advertisement information
  8. 8. RESEARCH
  9. 9. RESEARCH • User Requirements Investigation • Customers • Difficulty of Finding information • Having to wait for a Salesperson to show up • QR Codes, useless • A New Experience • Advertisers • The search for new creative advertising strategies • Modern Adverts focus simplicity, therefore less information • Visual Recognition over QR Code Recognition
  10. 10. RESEARCH • Similar Systems Study • Aurasma • Vuforia Qualcomm • Layar • The IBM Augmented Reality Personal Shopping Assist • Conclusions • Used for Advertising and real time Image target identification • Only available for US and Europe based countries • Cloud technologies • No Local implementation
  11. 11. RESEARCH • Augmented Reality • Mobile Augmented Reality • Mobile Application Development Platform • Apple • Android • Windows Phone • Mobile Video Stream Frame Monitoring • Image Processing
  12. 12. RESEARCH • Image Enhancements • Spatial Domain Enchantement • Image Negatives • Contrast Stretching • Histogram Processing • Frequency Domain Enchantement • Smoothing Frequency-Domain Filters • Low pass Filtering • High pass Filtering • Computer Vision
  13. 13. RESEARCH • Feature Extraction • SIFT - Scale Invariant Feature Transforms • SURF – Speeded Up Robust Features • Matching • 3D Object Detection in SURF • FAST - Corner Detection • FREAK - Fast Retina Key point Detection • BRISK - Binary Robust Invariant Scalable Key-points • Character Recognition and Reading
  14. 14. RESEARCH • Selection of APIs and Tools • Image Processing and Image Matching APIs • OpenCV • MatLab • Simple CV • Character Recognition APIs • Tesseract • ABBY OCR
  15. 15. RESEARCH • Augmented Realty APIs and SDKs • Vuforia • FastCV • Aurasma • Layar
  16. 16. DEVELOPMENT METHODOLOGY Feature Driven Development • Optimized for Scalable Systems • Executes on iterative development cycles • Suited for time intensive application • Features list that is being built at the beginning of the development method cycle • Feature by Feature • Each feature gets much better focus
  17. 17. ANALYSIS • Augmented Realty APIs and SDKs • Mobile Augmented Reality • Mobile Application Development Platform • Apple • Android • Windows Phone • Mobile Video Stream Frame Monitoring • Image Processing
  18. 18. RESEARCH
  19. 19. RESEARCH • Similar Systems Study Analysis • Augmented Reality • Image Processing and Recognition • Cloud Computing • User Experience • Availability for 3rd Parties • Not available for 3rd World Countries • No actual implementation in Sri Lanka (Experimental)
  20. 20. RESEARCH • Mobile Application Development Analysis – Android • Mobile Platform Usage • Developer Support and Features • Multitasking and Heavy Weight Processing Support • Image Processing • User Friendliness and UI Features
  21. 21. RESEARCH • Computer Vision Analysis • Robust • Faster • High Accuracy • Feature Extraction Analysis - SIFT, SURF, FREAK and BRISK • SURF • Speed and Accuracy • Implementation Support
  22. 22. REQUIREMENT SPECIFICATION
  23. 23. REQUIREMENT SPECIFICATION • Functional Requirements • Ability to identify Products or Advertisements • Augmented Reality should be implemented • Provide Related Products and Advertisements • Automatically Sort and categorize related Products and Advertisements
  24. 24. REQUIREMENT SPECIFICATION • Non-functional requirements • User friendly and easy to use Interface. • An Interface suitable for Augmented Reality based UI designs. • Mobile application should be smooth and quickly responsive. • Recognize Products and advertisements from the fastest way possible.
  25. 25. DESIGN
  26. 26. DESIGN • Use case – Mobile Application
  27. 27. DESIGN • Use case – Admin Server System
  28. 28. DESIGN • Class Diagram – Admin Server System
  29. 29. DESIGN • Activity Diagram – SURF Image Matching
  30. 30. DESIGN • Sequence Diagram – Brand Logo Identification
  31. 31. DESIGN • Sequence Diagram – Product Identification
  32. 32. DESIGN • Sequence Diagram – Mobile Application
  33. 33. IMPLEMENTATION • Image matching using SURF • Character Recognition and key words identification • Mobile Application UI development • Mobile Application Backend • Hosting a Web Service
  34. 34. IMPLEMENTATION • SURF Feature Extraction and Matching • Extract Feature Points and Descriptors • Brute Force Match and Vote for Feature Point Matches
  35. 35. IMPLEMENTATION • Further Voting and Filtering • Draw Matching Feature Points and Return Results
  36. 36. IMPLEMENTATION • SURF Feature Extraction and Matching Sample
  37. 37. IMPLEMENTATION • Experimentations • In need of Experimentation and Modifications • Could use “mask” Matrix • Started to give even greater matched points count of incorrect images
  38. 38. IMPLEMENTATION • Experimentation Conclusions • Get the number of matched feature points count • Calculate the Area of the formed Polygon • Identify whether the formed Polygon is a proper Polygon • Calculate the percentage Ratio of the Detected Key points of both images • Adjust the Uniqueness threshold
  39. 39. IMPLEMENTATION • Determining Best Match
  40. 40. IMPLEMENTATION • Threshold of an Image Matching Process • Image Pre-Processing Usage • Character Recognition and key words identification
  41. 41. IMPLEMENTATION • Mobile Image Processing • Image Resizing and Scaling
  42. 42. IMPLEMENTATION • Image Encoding and Decoding – Base64
  43. 43. IMPLEMENTATION • Web Service Connectivity
  44. 44. IMPLEMENTATION • Hosting a Web Service • Runtime Exception – Heavy EmguCV Libraries • Cv.Invoke Exceptions • Server Out Of Memory Exceptions
  45. 45. IMPLEMENTATION • Dummy Web Service
  46. 46. IMPLEMENTATION • Overall Admin System Implementation • Object Oriented Methodologies • Surf Execution Class
  47. 47. IMPLEMENTATION • Product Identification and Advertisement Identification Process • Find the Brand Logo Information • Load the Product Images under that Brand Logo • Take the given image through that Product Images list to find the best match and retrieve information.
  48. 48. TESTING • Unit Testing • Functional Testing • Performance Testing
  49. 49. TESTING • Unit Testing - Server System Test Case 1 Test Name : Recognize Given Product Image’s Brand Logo at Server Simulation Test Criteria Check if the Given Product’s Brand type is identified. Input / Action Performed Drag and Drop a Product Image and Click on “Search and Detect” Expected Results System should identify the brand logo and Display the Brand Information Actual Results The System successfully identified the Brand Type and showed the information of the Brand. Test Case 2 Test Name : Recognize Given Product Image at Server Simulation Test Criteria Check if the given Product’s front view Image is Recognized and Show the information of the Product Input / Action Performed Drag and drop Product Image and click on “Search and Detect” button. Expected Results System should recognize the Product and show information about it as well as Related Products. Actual Results Product was Successfully Recognized and information were showed.
  50. 50. TESTING • Unit Testing - Server System Test Case 3 Test Name : Determine when a given Image of a Product or an Advertisement cannot be Identified Test Criteria Check if the given Threshold condition is able to determine whether a given image can be Recognized or not. Input / Action Performed Insert a random unrelated Product/advertisement Image to the Matching Algorithm and execute matching algorithm Expected Results The algorithm should determine that the given Image cannot be Recognized based on the Threshold condition. Actual Results The Matching process successfully determines the given Image cannot be identified.
  51. 51. TESTING • Unit Testing – Server System Test Case 4 Test Name : Automatically Recognize the Brand Type of the Product/Advert image when adding a new Product/Advert info Test Criteria Check if the System is able to Automatically Recognize the Brand Type of a given Product or Advertisement Image when Inserting their information. Input / Action Performed Drag and drop the Product/Advert Image and click on “Auto Detect” Expected Results The matching algorithm will detect the Brand logo of the Product/Advert Image and identify the Brand. Actual Results The system successfully identifies the Brand Type by detecting the brand logo.
  52. 52. TESTING • Unit Testing – Mobile Application Test Case 5 Test Name : Retrieve Recognition results from the Web Service Test Criteria Check if the Application is able to retrieve the identification result of a capture image Input / Action Performed Click on “Scan and Detect” button Expected Results After sending the image to server application should wait for the response and retrieve the results. Actual Results The Application send the image to the server and waits till the response is being given, while the other UI Controls are being disabled. When the results are retrieved, they are being displayed successfully.
  53. 53. TESTING • Performance Testing Case Expected Accuracy Actual Result Capturing Image and Sending to Web 100% 100% service Retrieving Product/Advert Recognition Results 100% 100% Capturing product/advert image from a different angle and recognizing it 100% 50% Overall Recognition Results 90% 70% 푃표푠푖푡푖푣푒 푅푒푠푢푙푡푠 푁푢푚푏푒푟 표푓 푇푒푠푡푠 × 100%
  54. 54. CRITICAL EVALUATION • Efficiency • Execute all the heavy processing in Server • Holding the Pre-Processed – SURF Features Extracted Images in the Memory • Mobile Application to Server Communication – Data compression and Base64 Encoding • Utilizing the SURF based Image Matching process to achieve more accuracy and Speed • Utilizing the Recognition Process • Automated Brand Logo detection Process
  55. 55. CRITICAL EVALUATION • Ease of Use • Accuracy • Increased User Satisfaction • Maintainability and Extendibility • The Centralized Database • Web Service Implementation
  56. 56. CRITICAL EVALUATION • System Utility and Usability • Consumers, Advertisers and Marketers • Providing Access for Extended information • Information are updated and centralized • implement creative strategies into Augmented Reality • intuitive way to recognize products for end users rather than QR Codes • easily upgraded and extended due to the architecture • The interface has been designed in a very minimalistic, simple, and user friendly
  57. 57. CRITICAL EVALUATION • Further Enhancements • Further Developing the Image Matching Process from ground up to use even more efficient Algorithms such as FREAK or BRISK Feature Extraction algorithms • Adopt an Augmented Reality API such as Vuforia • High end Cloud Service which will radically increase the processing power. • Adopt a new lightweight API Library for Image Processing
  58. 58. DEMONSTRATION
  59. 59. Q & A
  60. 60. THANK YOU !

×