Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a From Paper to Power using Azure Form Recognizer (Azure Sydney UG 2020)
Semelhante a From Paper to Power using Azure Form Recognizer (Azure Sydney UG 2020) (20)
Último
Último (20)
From Paper to Power using Azure Form Recognizer (Azure Sydney UG 2020)
- 1. Sydney Azure User Group From Paper to Power using Azure Form Recognizer Join the Conversation @AzureSydneyUG @jernej_kavka v3
- 2. SSW Senior Software Architect Jernej Kavka (JK) @Jernej_kavka github.com/jernejk jkdev.me linkedin.com/in/jernejkavka/ .NET Core and EF Core dev at day Cognitive Services and ML.NET dev at night Join the Conversation @AzureSydneyUG @jernej_kavka
- 3. Join the Conversation @AzureSydneyUG @jernej_kavka
- 4. Cognitive Studio – https://cognitivestudio.dev
- 5. What do we do now? Demo Azure Form Recognizer Table of Contents
- 6. Current flow of collecting feedback
- 7. We present
- 8. Ask participants to fill out the feedback form
- 9. Admins scans papers and upload to SharePoint
- 10. Make a summary of the feedback
- 11. Shreds the feedback forms to protect attendees privacy ☺
- 12. Later when time permits, enters them into CRM
- 13. Manual labour • ~1 hour - Scanning and quick summary • ~1-3 hours – Manually entering data for a single event Join the Conversation @AzureSydneyUG @jernej_kavka
- 14. We have a lot of events! Join the Conversation @AzureSydneyUG @jernej_kavka
- 15. Brisbane Full Stack User Group Join the Conversation @AzureSydneyUG @jernej_kavka
- 16. Melbourne .NET User Group
- 17. Sydney .NET User Group
- 18. 1 – 2 days training in 3 cities
- 19. Hack Days
- 20. It’s a lot of work • Up to 100 attendees per event • Up to 5 cities “road-trip presenting” • Up to 10 events per month • Hours stacks up quickly with manual labour Join the Conversation @AzureSydneyUG @jernej_kavka
- 21. We need to scan papers as soon as they come
- 22. Or they will stack up really quickly Join the Conversation @AzureSydneyUG @jernej_kavka
- 23. Making forms digital • Works well on virtual events! • Doesn’t work on a physical event • Participants don’t like copying links to their phones • Most don’t like scanning QR codes • It can work if you promise rewards (expensive 💸) Join the Conversation @AzureSydneyUG @jernej_kavka
- 24. Works really good but not enough feedback
- 25. What do we do now? Demo Azure Form Recognizer Table of Contents
- 26. Azure Form Recognizer • Allows recognize and tag text in a form • By default it supports • US receipts • Tables Join the Conversation @AzureSydneyUG @jernej_kavka
- 27. Custom Form Recognizer • Train using your own forms • Uses various cognitive services behind the scenes • OCR • Positioning of the labelled text • Is done in 2 phases • Training • Usage Join the Conversation @AzureSydneyUG @jernej_kavka
- 28. Getting started • Azure account • https://azure.microsoft.com/en-us/free/ • PowerShell Core • https://github.com/powershell/powershell • Azure CLI • https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest • Azure Storage Explorer • https://azure.microsoft.com/en-au/features/storage-explorer/ Join the Conversation @AzureSydneyUG @jernej_kavka
- 29. Quick start • Download automation script • https://gist.github.com/jernejk/fdb42e032a9568d42dc8c2f05bd1fc1 3 • Get started in 2 mins rather than 30+ minutes Join the Conversation @AzureSydneyUG @jernej_kavka
- 30. Join the Conversation @AzureSydneyUG @jernej_kavka
- 31. Label, train and test data
- 32. Logic Apps
- 33. MS Flow/Power Automate allows adding upload buttons on your phone
- 34. Cognitive Studio – Form Recognizer coming soon
- 35. What have we learned? • Hand writing is a challenge for AI and humans • Form Recognizer works great with printed text • Initial setup is a bit lengthy • Don’t use doted lines over solid line • Checkboxes are not yet supported Join the Conversation @AzureSydneyUG @jernej_kavka
- 36. What about Check/Radio boxes?
- 37. “How hard can it be?” Anonymous
- 38. Azure Custom Vision • Use object recognition • Upload 15+ images • Manually draw boundaries for • Unchecked checkbox/radio box • Checked checkbox/radio box Join the Conversation @AzureSydneyUG @jernej_kavka
- 39. Uploading images Join the Conversation @AzureSydneyUG @jernej_kavka
- 41. After training the number don’t look that good… Maybe it will work?
- 42. Ok, it got 3 right….
- 43. Plan B: More focused training
- 44. Almost all radio boxes are marked while bad data is skipped
- 45. Looks better…
- 46. It’s… actually pretty good ☺
- 47. It still need some work, though…
- 48. Ok… Maybe I need even more data?
- 49. Plan C: Custom Form Recognizer + ML.NET
- 50. Form Recognizer can give us location of all sentences, no training required!
- 51. Step 1 – Find all labels for radio boxes
- 52. Step 2 – Define a unit based on relative distance 1 relative unit
- 53. Step 3 – Calculate relative positions based on that unit 6.5 relative unit
- 54. Step 4 - Mess around with numbers until you find most of the radio boxes on most forms
- 55. Step 5 – Cut radio boxes. Now, I got ~15,000 training images to work with!
- 56. Step 6 – Do ML.NET Image Classification (mostly because it’s free and I like it)
- 57. Step 7 – Train based on confirmed correct images
- 58. Step 8 – Train! Finally, the results I’ve been waiting for! 🎉
- 59. Step 9 – Test! A few hundred random radio boxes worked perfectly!
- 60. About 80%+ of forms seems to work correctly!
- 61. Some forms needed a bit of “Math and statics magic” when data wasn’t available
- 62. And some need a bit more work (less than 10%!)
- 63. Summary • Recognizing data on Forms is easy • Recognizing radio/checkboxes is not • Microsoft is already working on a solution • It took me 40+ hours for semi-generic solution • I’ll open source images and ML.NET model Join the Conversation @AzureSydneyUG @jernej_kavka
- 64. Thank you! @Jernej_kavka https://bit.ly/AzureFormRecognizer info@ssw.com.au www.ssw.com.au Sydney | Melbourne | Brisbane | Canberra