O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Smart Web Apps with Azure and AI as a Service

15.068 visualizações

Publicada em

Smart homes, smart phones, even smart stones… Today users expect everything to be smart and web sites to be tailored to their needs, and intelligent enough to serve within less taps. The huge advancements in machine learning and big data in recent years made that possible. One of the most complete and advanced services that is a step in front of the competition, and allows developers to add AI to their products, is Azure Cognitive Services. This session will be about how computer vision, natural language processing, speech and intent recognition could allow building smart apps with enhanced experience and be more engaging, personal and relevant.

Publicada em: Software
  • Seja o primeiro a comentar

Smart Web Apps with Azure and AI as a Service

  1. 1. Nov 17, 2018Sofia var title = “Building Smart Web Apps with Azure and AI as a Service”; var info = { name: “Ivelin Andreev”, otherOptional: “Building apps for enhanced, personal and relevant UX” };
  2. 2. Nov 17, 2018 • Software Architect @ o 16+ years professional experience • Microsoft Azure MVP • External Expert Horizon 2020 • External Expert Eurostars-Eureka, InnoFund Denmark • Business Interests o Web Development, SOA, Integration o IoT, Machine Learning, Computer Intelligence o Security & Performance Optimization • Contact ivelin.andreev@icb.bg www.linkedin.com/in/ivelin www.slideshare.net/ivoandreev Who is in front of you?
  3. 3. Nov 17, 2018 Thanks to our Sponsors: General Sponsor: Gold Sponsors: Technological Partner: Hosting Partner: Silver Sponsors: Lunch Sponsor: Innovation Sponsor: Bronze Sponsors: Infotainment Sponsor:
  4. 4. Nov 17, 2018 agenda(); • Artificial Intelligence as a Service • Computer Vision • Custom Vision • Language Understanding • Custom Decision • Other Cognitive Services • Demo
  5. 5. Nov 17, 2018 Users’ expectations: • Engaging experience • Effortless interaction • High performance • Relevant content Businesses aim: • Provide high value • Faster and at low cost o Data science talent o Powerful infrastructure o Continuous improvement The developer role is to bridge the gap:
  6. 6. Nov 17, 2018 Artificial Intelligence as a Service (AIaaS) Def: Artificial intelligence off the shelf • Bots and NLP – commands and guidance • Cognitive APIs – speech, vision, translation, knowledge • ML frameworks – build own model w/o infrastructure (i.e. Azure ML Service) • Fully managed ML – templates, deployment, drag-drop (i.e. Azure ML Studio) • Innovation w/o upfront costs and expertise • Usability – easy learning curve • Scalability – start PoC, grow big • Flexi cost – know what you pay for • Share data with vendors • Data regulations (i.e. GDPR) • Reduced transparency • Breaking changes
  7. 7. Nov 17, 2018 AIaaS market expected to grow from $1.5Bil (2018) to 10.9Bil (2023) (ResearchAndMarket Apr’ 2018) 1 year ago this was not as achievable as it is now.
  8. 8. Nov 17, 2018 • Microsoft and open source • Contribute in Code https://opensource.microsoft.com/ • Contribute in Localization o Democratic mode and voting o Crowdsourcing with tool support https://github.com/Microsoft/Localization/wiki Microsoft are Open (more than ever before) 830+ Projects o VS Code o VSCode tools for AI o SQL tools on Linux o TypeScript o CNTK o .NET Core o SignalR o ONNX
  9. 9. Nov 17, 2018 Computer Vision – Why? What it does • Advanced algorithms for processing images for information Use cases • Tag visual features (2000+ objects and actions) • Description in human readable language • Face detection with tagging of age, gender and bounding rectangle • OCR in 25 languages (no Bulgarian ), Handwriting (preview) • Categorization on hierarchical taxonomies • Domain modeling – recognize domain members (i.e. celebrity, family ) Pricing • Free - 5000 req./month, $1-$2.5 per 1’000 req. (10 TPS)
  10. 10. Nov 17, 2018 Sample: Computer Vision for Fire Detection Cloud Vision (Google) Computer Vision (Microsoft)
  11. 11. Nov 17, 2018 Face API, Verify API, Identify API – Why? What it does • Detect and analyze facial attributes • Authentication against two faces or group Use cases • Security - verify the same person (Verify API) • Person identification – find closest match (Identify API) • Find known persons - similar faces (Face API) • Image tagging – age, gender, emotion, hair, makeup, accessories (Face API) Pricing • Free – 30’000 req/month (20 TPM) • Standard - $1 per 1’000 req. (10 TPS)
  12. 12. Nov 17, 2018 Identify API – How? Create person group • Create group (i.e. Family) and get unique personGroupID Create person • Create person (name, userData) on personGroupID Add face to person in group • Upload image (URL and byte[] supported) for personGroupID and personID Train group • Train prior to identification; retrain on changes (!= Status.Running) Identify • Upload image and get unique faceID • Identify candidates on personGroupID and faceID
  13. 13. Nov 17, 2018 [{"faceId": "abaff639-9ed4-43d6-b519- 670804552941", faceRectangle": { "width": 220, "height": 242, "left": 256, "top": 256 }, "faceLandmarks": {…}, "faceAttributes": { "age": 67.0, "gender": "male", "smile": 0.0, "facialHair": { "moustache": 0.9, "beard": 0.9, "sideburns": 0.9 }, "glasses": "sunglasses", "headPose": { "roll": -0.8, "yaw": -11.7, "pitch": 0.0 }, "emotion": { "anger": 0.0, "contempt": 0.0, "disgust": 0.0, "fear": 0.0, "happiness": 0.0, "neutral": 0.673, "sadness": 0, "surprise": 0.326 }, "hair": { "bald": 0.0, "invisible": true, "hairColor": [] }, "makeup": { "eyeMakeup": false, "lipMakeup": false }, "occlusion": { "foreheadOccluded": true, "eyeOccluded": false, "mouthOccluded": false }, "accessories": [ {"type":"headWear","confidence":1.0}, {"type":"glasses","confidence":1.0}], … } } ] Face API - Sample Response
  14. 14. Nov 17, 2018 Computer Vision is an easy to use pretrained image classifier. Use Custom Vision to create custom classifications.
  15. 15. Nov 17, 2018 customvision.ai (preview) – Why? What it does • Build, deploy, improve custom image classifiers (on tags) Use cases • Image classification – assign tags to each image • Object detection – return the coordinates of object in image • Execute on mobile device (ONNX, CoreML, Tensorflow) Pricing • Free (2 TPS, 2 projects, 5’000 images/project) • Standard (10 TPS; 100 projects; $1 per 1’000 req., SLA 99.9%) API- and SDK-based (train, predict, test)
  16. 16. Nov 17, 2018 Domains (General, Food, Landmarks, Retails, Adult, Compact) Create project Upload and tag • It is easier to upload by the tags (50+ images/tag) • Use different angles, background, size and balance Train the classifier • As easy as • Set training iteration as default Performance • Precision – correct prediction TP / (TP + FP) • Recall – proper identification TP / (TP + FN) customvision.ai (preview) – What?
  17. 17. Nov 17, 2018 Prediction API Request customvision.ai (preview) – How? Negative image handling • Resemblance: Automatic negative image handling (0% score expected) • No resemblance: Build class “Other” Prediction API Response
  18. 18. Nov 17, 2018 Language Understanding
  19. 19. Nov 17, 2018 LUIS.ai – Why? What it does • Apply custom ML intelligence to conversational natural language • Predict meaning to complete a task / command Use cases • Chat bot; Intent recognition LUISGen • CMD tool for strongly typed interfaces in C# and TS Pricing • Free (5 TPS) • Standard (50 TPS; Text - $1.50 per 1’000; Speech - $5.50 per 1’000) 12 languages supported
  20. 20. Nov 17, 2018 LUIS.ai – What? Concepts • Intents – tasks or actions user wants to perform • Utterances – input from user the app needs to interpret • Entities – key words and phrases for domain (prebuilt, simple, list, hierarchy, composite, RegEx) o Hierarchies – i.e. Name::FirstName, Name::LastName • Roles – contextual subtypes of entity for patterns (i.e. {Location:origin} to {Location:destination}) Build LUIS Model • Start with 10-15 training utterances per intent (Authoring API or LUIS portal) Publish (incl. sentiment) Active Learning • Utterances added to review list (low top score, top 2 are too close)
  21. 21. Nov 17, 2018 LUIS.ai – How? (Best Practices) Different intents, different vocabulary (i.e. Book flight / Book hotel) Phrase lists • {Jan, Feb, Mar…}; Synonyms (i.e. want, require, need) Patterns “None” intent for subject outside of application domain Don’t add examples of the same format • Buy a ticket to Seattle • Buy a ticket to Paris -> Reserve a seat to Paris Don’t create intent for different entities (i.e. every destination)
  22. 22. Nov 17, 2018 LUIS.ai – How? Endpoint https://<region>.api.cognitive.microsoft.com/luis/v2.0/apps/<appID>?verbose=true&subscription- key=<YOUR_KEY>&<optional-name-value-pairs>&q=<user-utterance-text> • Send encoded utterance with “q” parameter (i.e. Here+is+my+c.v.+for+the+data+scientist+job) JSON { "query": "Here is my c.v. for the data scientist job", "topScoringIntent": { "intent": "ApplyForJob", "score": 0.9826467 }, "intents": [ {"intent": "ApplyForJob", "score": 0.9826467}, {"intent": "GetJobInformation", "score": 0.0218927357} … ], "entities": [ { "entity": “data scientist", "type": "Job", "startIndex": 24, "endIndex": 37, "score": 0.5714122 } ] }
  23. 23. Nov 17, 2018 Custom Decision
  24. 24. Nov 17, 2018 Custom Decision (experimental) – Why? What it does • Learn behavioural patterns of users • Recommend relevant experience or content Use cases • Personalize product / website • Personalize video content • Optimize Ad placement on pages • Rank shopping item recommendations 18 languages supported
  25. 25. Nov 17, 2018 Custom Decision (experimental) – What? Handles end-to-end ML process with reinforcement learning • Explore and collect data • Train and deploy models User is presented with a list of content items (i.e. on front page) • Specify an RSS feed to define Action Set and refresh time • Convert content into ML features (text, images, video, sentiment) • Use other services at the backend (Entity Linking, Text Analytics, Emotion/Face, Vision) • Get all content and returning in ranked order (Ranking) • Callback function (defined by developer) renders the content in UI in ranked order When a user makes selection • Clicks are reported by custom code and measure response (Reward)
  26. 26. Nov 17, 2018 • Register application to receive App ID • Define RSS feed Ranking API • JSONP formatted response to bypass Same-origin-policy Reward API Custom Decision (experimental) – How?
  27. 27. Nov 17, 2018 Cognitive Labs - in Preview Gesture o Productivity SDK to allow interaction with technology using gestures o Requires Intel RealSense camera ($89) Conversation Learner o Build and teach bots and intelligent agents from sample interaction o Based on LSTM NN and learn based on prototypical dialogs o Custom code attached to callbacks Entity Linking o Recognize named entities based on context and provide Wikipedia ref. o Part of Text Analytics API https://labs.cognitive.microsoft.com/
  28. 28. Nov 17, 2018 Cognitive Labs - Private Preview Anomaly Finder o Monitor data over time, adapt and detect anomalies based on statistical models URL Preview o JSON with description and a link to preview image o isFamilyFriendly flag – indicates adult, pirated or illegal content Local Insights o Score the attractiveness of a location based on proximity to amenities (public transport, parks, cinemas or other) Answer Search o Answer question in natural language with information from the web Event Tracking o Cluster news that report things happening on Earth to event
  29. 29. Nov 17, 2018 Takeaways • Cognitive services documentation https://docs.microsoft.com/en-us/azure/cognitive-services/welcome • Custom Decision service documentation https://docs.microsoft.com/en-us/azure/cognitive-services/custom-decision-service • LUIS Samples https://github.com/Microsoft/LUIS-Samples/tree/master/examples • Cognitive Services Online Demos https://azure.microsoft.com/en-us/services/cognitive-services/directory/vision/ • Computer Vision (Sample JS Code) https://github.com/Azure-Samples/cognitive-services-javascript-computer-vision-tutorial
  30. 30. Nov 17, 2018 Thanks to our Sponsors: General Sponsor: Gold Sponsors: Technological Partner: Hosting Partner: Silver Sponsors: Lunch Sponsor: Innovation Sponsor: Bronze Sponsors: Infotainment Sponsor:

×