An introduction to Data Quality Services. DQS enables to discover, build, and manage knowledge about your data. Use that knowledge to perform data cleansing, matching and profiling. We will explore the numerous features and capabilities of Data Quality Services and its integration with SSIS with the DQS Cleansing Transform. Data Quality Services in SQL Server 2012
The Health and Social Care Information Centre is hosting a series of road shows jointly with the Electronic Staff Record (ESR) Central Team and Health Education England to highlight developments in NHS workforce information, data standards and data quality.
Here are the slides presented at the first event, held at the Royal Marsden NHS Foundation Trust on 1st October 2015.
Data quality is all about collaborative working with a shared purpose and this is the main driver behind our road shows during 2015/16. Any efforts to improve data quality should have mutual benefits and should provide a platform for discourse between all involved. Collectively we can ensure that the data that is used to inform decisions about the workforce at local, regional and national level is as accurate as possible. Good data quality can't guarantee good decisions are made, but poor data quality will definitely increase the likelihood of poor decisions and poor outcomes.
For more information about future events, please contact the team mailto:workforce.dq@hscic.gov.uk <mailto:workforce.dq@hscic.gov.uk>
a2c Boston Big Data Meet-up: Agile Data Warehouse Designa2c
Preview this Big Data Seminar, and request the complete audio and animated download featuring Agile Data Warehouse Design - a step-by-step method for data warehousing / business intelligence (DW/BI) professionals to better collect and translate business intelligence requirements into successful dimensional data warehouse designs. The method utilizes BEAM✲ (Business Event Analysis and Modeling) - an agile approach to dimensional data modeling that can be used throughout analysis and design to improve productivity and communication between DW designers and BI stakeholders. a2c's Practice Director of Information Services and Author Jim Stagnitto and CTO John DiPietro designed this presentation to provide an overview of Agile Warehouse Design that will facilitate communication between Data Modelers and Business Intelligence Stakeholders in a fun and informative one hour session. Demystify this process and find out what the 96 Data Scientists who attended November's Boston Big Data Meet-up are talking about.
“Excellent presentation. It is good to hear meaningful …information about new developments in how Agile methodologies can be applied to DW/BI work. Big Kudos to the presenters and organizers. Thanks, I found it very useful and enjoyable.”- Ramon Venegas
“Extremely useful to understand how to apply Agile approach to DWH; how create a framework where model changes are welcome, and bring users to the process of DWH modeling.” – Alfredo Gomez
An introduction to Data Quality Services. DQS enables to discover, build, and manage knowledge about your data. Use that knowledge to perform data cleansing, matching and profiling. We will explore the numerous features and capabilities of Data Quality Services and its integration with SSIS with the DQS Cleansing Transform. Data Quality Services in SQL Server 2012
The Health and Social Care Information Centre is hosting a series of road shows jointly with the Electronic Staff Record (ESR) Central Team and Health Education England to highlight developments in NHS workforce information, data standards and data quality.
Here are the slides presented at the first event, held at the Royal Marsden NHS Foundation Trust on 1st October 2015.
Data quality is all about collaborative working with a shared purpose and this is the main driver behind our road shows during 2015/16. Any efforts to improve data quality should have mutual benefits and should provide a platform for discourse between all involved. Collectively we can ensure that the data that is used to inform decisions about the workforce at local, regional and national level is as accurate as possible. Good data quality can't guarantee good decisions are made, but poor data quality will definitely increase the likelihood of poor decisions and poor outcomes.
For more information about future events, please contact the team mailto:workforce.dq@hscic.gov.uk <mailto:workforce.dq@hscic.gov.uk>
a2c Boston Big Data Meet-up: Agile Data Warehouse Designa2c
Preview this Big Data Seminar, and request the complete audio and animated download featuring Agile Data Warehouse Design - a step-by-step method for data warehousing / business intelligence (DW/BI) professionals to better collect and translate business intelligence requirements into successful dimensional data warehouse designs. The method utilizes BEAM✲ (Business Event Analysis and Modeling) - an agile approach to dimensional data modeling that can be used throughout analysis and design to improve productivity and communication between DW designers and BI stakeholders. a2c's Practice Director of Information Services and Author Jim Stagnitto and CTO John DiPietro designed this presentation to provide an overview of Agile Warehouse Design that will facilitate communication between Data Modelers and Business Intelligence Stakeholders in a fun and informative one hour session. Demystify this process and find out what the 96 Data Scientists who attended November's Boston Big Data Meet-up are talking about.
“Excellent presentation. It is good to hear meaningful …information about new developments in how Agile methodologies can be applied to DW/BI work. Big Kudos to the presenters and organizers. Thanks, I found it very useful and enjoyable.”- Ramon Venegas
“Extremely useful to understand how to apply Agile approach to DWH; how create a framework where model changes are welcome, and bring users to the process of DWH modeling.” – Alfredo Gomez
CRM magic with data migration & integration (Presentation at CRMUG Summit 2013)Daniel Cai
This is the deck that I presented to CRMUG Summit 2013 in Tampa. During the session, I tried to discuss various options that you may have for Microsoft Dynamics CRM data migration and integration, including some best practices that you can leverage. This deck is an updated version of my XrmVirtual presentation on Apr 9, 2013.
Top 5 TSQL Improvements in SQL Server 2014Boris Hristov
SQL Server 2014 comes with dozens of improvements in various areas. In this presentation we will discuss and see how the new release can make the life of each and every developer easier and what are the top 5 T-SQL enhancements that we can use in our day-to-day work.
With the introduction of SQL Server 2012 data developers have new ways to interact with their databases. This session will review the powerful new analytic windows functions, new ways to generate numeric sequences and new ways to page the results of our queries. Other features that will be discussed are improvements in error handling and new parsing and concatenating features.
Business Redefined – Managing Information Explosion, Data Quality and ComplianceCapgemini
Capgemini is innovating to deliver maximum value to customers by utilizing the latest technologies and thinking.
An example of Capgemini combining technology and thinking is our Data Warehouse Optimization (DWO) solution, enabling a business to balance the needs of archiving against the needs of access to legacy information. DWO leverages Informatica technologies and Hadoop storage to provide a robust and cost effective solution, effectively archiving data into Hadoop and retaining access to query the data.
This is just one of our recent innovations - others include Data Quality as a Service and a new approach to Data Masking.
Presented by Malay Baral at Informatica World 2014.
This exam measures your ability to accomplish the technical tasks listed below. The percentages indicate the relative weight of each major topic area on the exam. https://www.pass4sureexam.com/70-461.html
Introduction to Master Data Services in SQL Server 2012Stéphane Fréchette
What is Master Data Services? Why is it important? - Will discuss Master Data Services capabilities, it's underlying architecture. Will demo creating a model, using SQL Server 2012 MDS add-in for Microsoft Excel, creating hierarchies, business rules and exposing/integrating data with other interfaces (Data Warehouse)
Microsoft for BI and DW: Using the Right Tool for the JobSenturus
Learn the capabilities and best use cases for Power BI, SQL Server, SharePoint, Azure and Office. View the webinar video recording and download this deck: http://www.senturus.com/resources/microsoft-for-bi-and-dw/.
You'll also want to check out a Microsoft tool matrix that guides you in choosing the right tool for the job: http://www.senturus.com/wp-content/uploads/2015/11/Microsoft-BI-DW-Tool-Matrix-Senturus.pdf.
Knowing how the tools work together allows you to build an efficient, integrated BI solution. Information includes a review of product features and benefits, discusses use cases and demonstrate product capabilities.
Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.
SSDN Technology is a training institute located in Delhi Gurgaon, NCR & India which offer best MCSA - SQL SERVER 2012 training by our experienced trainer. We are providing live project training with full lab facility. For more details for a bright future call us at +91-9999-111-686.
http://www.ssdntech.com/sql-server-training.aspx
During this session we will look into Windows 10 for the Enterprise.
Let’s explore the new management capabilities and choices.
Let’s understand the Windows 10 deployment infrastructure and mechanisms.
Let’s discover new Windows 10 features and improvements.
You are eager to learn about Windows 10 and want to gather early-stage info about this exciting Operating System… ?
Well you know what to do! See you there!
Compliance settings, formerly known as DCM, remains one of the often unexplored features in Configuration Manager. During this session we will walk through the new capabilities and improvements of this feature in ConfigMgr 2012, discuss implementation details, and demonstrate how you can start using it to fulfill actual business requirements.
Discover what’s new in Windows 8.1 regarding interface, settings, deployment, security, … How will Windows 8.1 fit in your enterprise? How do you upgrade? All answers are here!
CRM magic with data migration & integration (Presentation at CRMUG Summit 2013)Daniel Cai
This is the deck that I presented to CRMUG Summit 2013 in Tampa. During the session, I tried to discuss various options that you may have for Microsoft Dynamics CRM data migration and integration, including some best practices that you can leverage. This deck is an updated version of my XrmVirtual presentation on Apr 9, 2013.
Top 5 TSQL Improvements in SQL Server 2014Boris Hristov
SQL Server 2014 comes with dozens of improvements in various areas. In this presentation we will discuss and see how the new release can make the life of each and every developer easier and what are the top 5 T-SQL enhancements that we can use in our day-to-day work.
With the introduction of SQL Server 2012 data developers have new ways to interact with their databases. This session will review the powerful new analytic windows functions, new ways to generate numeric sequences and new ways to page the results of our queries. Other features that will be discussed are improvements in error handling and new parsing and concatenating features.
Business Redefined – Managing Information Explosion, Data Quality and ComplianceCapgemini
Capgemini is innovating to deliver maximum value to customers by utilizing the latest technologies and thinking.
An example of Capgemini combining technology and thinking is our Data Warehouse Optimization (DWO) solution, enabling a business to balance the needs of archiving against the needs of access to legacy information. DWO leverages Informatica technologies and Hadoop storage to provide a robust and cost effective solution, effectively archiving data into Hadoop and retaining access to query the data.
This is just one of our recent innovations - others include Data Quality as a Service and a new approach to Data Masking.
Presented by Malay Baral at Informatica World 2014.
This exam measures your ability to accomplish the technical tasks listed below. The percentages indicate the relative weight of each major topic area on the exam. https://www.pass4sureexam.com/70-461.html
Introduction to Master Data Services in SQL Server 2012Stéphane Fréchette
What is Master Data Services? Why is it important? - Will discuss Master Data Services capabilities, it's underlying architecture. Will demo creating a model, using SQL Server 2012 MDS add-in for Microsoft Excel, creating hierarchies, business rules and exposing/integrating data with other interfaces (Data Warehouse)
Microsoft for BI and DW: Using the Right Tool for the JobSenturus
Learn the capabilities and best use cases for Power BI, SQL Server, SharePoint, Azure and Office. View the webinar video recording and download this deck: http://www.senturus.com/resources/microsoft-for-bi-and-dw/.
You'll also want to check out a Microsoft tool matrix that guides you in choosing the right tool for the job: http://www.senturus.com/wp-content/uploads/2015/11/Microsoft-BI-DW-Tool-Matrix-Senturus.pdf.
Knowing how the tools work together allows you to build an efficient, integrated BI solution. Information includes a review of product features and benefits, discusses use cases and demonstrate product capabilities.
Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.
SSDN Technology is a training institute located in Delhi Gurgaon, NCR & India which offer best MCSA - SQL SERVER 2012 training by our experienced trainer. We are providing live project training with full lab facility. For more details for a bright future call us at +91-9999-111-686.
http://www.ssdntech.com/sql-server-training.aspx
During this session we will look into Windows 10 for the Enterprise.
Let’s explore the new management capabilities and choices.
Let’s understand the Windows 10 deployment infrastructure and mechanisms.
Let’s discover new Windows 10 features and improvements.
You are eager to learn about Windows 10 and want to gather early-stage info about this exciting Operating System… ?
Well you know what to do! See you there!
Compliance settings, formerly known as DCM, remains one of the often unexplored features in Configuration Manager. During this session we will walk through the new capabilities and improvements of this feature in ConfigMgr 2012, discuss implementation details, and demonstrate how you can start using it to fulfill actual business requirements.
Discover what’s new in Windows 8.1 regarding interface, settings, deployment, security, … How will Windows 8.1 fit in your enterprise? How do you upgrade? All answers are here!
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
2. WHO AM I
• BI consultant @ Ordina
• member of SQLUG.be
• MCTS, MCITP in SQL Server 2008
• working with Microsoft BI for over 2 years
• beer and comic books enthusiast
• married with children…
3. INTRODUCTION
data quality?
Data are of high quality "if they are fit for their intended uses in
operations, decision making and planning" (J. M. Juran).
- Wikipedia on Data Quality
• achieved through people, technology & processes
• can be measured with various dimensions
• accuracy
• consistency
• completeness
• duplicates (uniqueness)
• timeliness
• validness
• bad data = bad business
4. INTRODUCTION
Data Quality Issue Sample Data Problem
Standard Are data elements consistently Gender code = M, F, U in one system and Gender
defined and understood? code = 0, 1, 2 in another system
Complete Is all necessary data present ? 20% of customers’ last name is blank,
50% of zip-codes are 99999
Accurate Does the data accurately A supplier is listed as ‘Active’ but went out of
represent reality or a verifiable business six years ago
source?
Valid Do data values fall within Temperature recordings should be between
acceptable ranges? -100°C and +100°C
Unique Data appears several times Prince, The Artist formerly known as Prince, The
Artist, … are they the same person?
5. INTRODUCTION
Monitoring Cleansing
Tracking and monitoring Amend, remove or enrich
the state of Quality data that is incorrect or
activities and Quality incomplete. This includes
of Data correction, standardization
and enrichment.
Monitoring Cleansing
Profiling Matching
Profiling
Matching
Analysis of the data
Identifying, linking or
source to provide insight
merging related entries
into the quality of the
within or across sets of data.
data and help to identify
data quality issues.
6. OUTLINE
• introduction
• overview of data quality services
• building a knowledge base
• data cleansing & matching
• SSIS integration
• conclusion
7. OVERVIEW OF DQS
Data Quality Services (DQS) is a
Knowledge-Driven data quality solution,
enabling IT Pros and data stewards to easily
improve the quality of their data
8. OVERVIEW OF DQS
Knowledge-
Based on a Data Quality Knowledge Base (DQKB)
Driven
Semantics Data Domains capture the semantics of your data
Knowledge
Acquires additional knowledge the more you use it
Discovery
Open and Support use of user-generated knowledge and IP
Extendible by 3rd party reference data providers
Compelling user experience designed for increased
Easy to use productivity
9. OVERVIEW OF DQS
• easy installation
• pre-installation checks
o SQL Server 2012 database engine (server)
o .NET 4.0 & IE 6.0 or higher (client)
• installation of DQS using SQL Server set-up
• post-installation tasks
o run DQSInstaller.exe
o grant DQS roles to users
o enable TCP/IP
10. OUTLINE
• introduction
• overview of data quality services
• building a knowledge base
• data cleansing & matching
• SSIS integration
• conclusion
11. BUILDING A KNOWLEDGE BASE
Knowledge
Management
Build Discover / Explore Data / Connect
Integrated Knowledge
Profiling
Base
Use
DQ Projects
12. BUILDING A KNOWLEDGE BASE
Values
Composite
Domains
Domains
Represent
3rd party the data type
Reference
Data Domains Knowledge
Rules & Base
Relations
Matching
Policy
15. BUILDING A KNOWLEDGE BASE
• iterative process
• knowledge discovery
• gather knowledge from
o Excel
o SQL Server
• profiling of data
o not the same as SSIS profiling task!
• automatically detects anomalies
16. BUILDING A KNOWLEDGE BASE
• domain management
• knowledge about fields is kept in domains
• data steward can
o create rules
o assign synonyms and corrections
o create term based relations (str. street)
o link domains together into
composite domains
• import knowledge from
o reference data (e.g. Azure Marketplace)
o other knowledge bases
17. OUTLINE
• introduction
• overview of data quality services
• building a knowledge base
• data cleansing & matching
• SSIS integration
• conclusion
18. DATA CLEANSING & MATCHING
• cleansing • St. --> street (corrected)
• why? • Microsot --> Microsoft (corrected)
o identifies incomplete or incorrect data • john.doe@hotmail (invalid)
o standardizes and enriches data by using • 0472/34672 (invalid)
domain values, domain rules and reference data
• Verbeek --> Verbeeck (suggested)
• DQS cleansing
o create a knowledge base or select an existing one
o create a data quality project
o 2-step process
– computer assisted cleansing
– interactive cleansing
o export results
19. DATA CLEANSING & MATCHING
• matching • Prince
• The Artist Formerly Known
• why? •
As Prince
The Artist
o identify duplicates with the data source
•
o create consolidated view of data
• Jon Doe, High Street 13, NY,
• DQS matching doe@gmail.com
o build a matching policy in KB John Doe, High Str, NY,
o matching training doe@gmail.com
o create matching project
o choose survivors
DQ Client – Match Results
21. DATA CLEANSING & MATCHING
• create a cleansing project
• uses knowledge gathered in a DQS knowledge base
• simple user-friendly process
• profile results
22. DATA CLEANSING & MATCHING
• create a matching project
• uses a matching policy created
in a knowledge base
• eliminates duplicates
• profile results
• the more knowledge that is added the better results will be
o tip: clean-up the data first using a cleansing project
• choose survivors at the end
• export results into .csv
or SQL Server
23. OUTLINE
• introduction
• overview of data quality services
• building a knowledge base
• data cleansing & matching
• SSIS integration
• conclusion
24. SSIS INTEGRATION SSIS Data Flow
Knowledge
Base
SSIS Package
Source + Data correction
Values/Rules Mapping Component Destination
Reference Data
Definition
26. SSIS INTEGRATION
• cleaning as a batch process
• only cleaning, matching is (not yet?) possible
• composite domains are supported
27. OUTLINE
• introduction
• overview of data quality services
• building a knowledge base
• data cleansing & matching
• SSIS integration
• conclusion
28. CONCLUSION
Knowledge-driven Easy To Use Open & Extendible
Rich Knowledge Base Focus on productivity and Focus on cloud-based
Continuous improvement user experience Reference Data
and knowledge acquisition Designed for business users User-generated knowledge
Build once, reuse for Out-of-the-box knowledge Integration with SSIS
multiple DQ improvements
29. RESOURCES
• DQS Team Blog @ MSDN
http://blogs.msdn.com/b/dqs/
• DQS documentation @ MSDN
http://msdn.microsoft.com/en-us/library/ff877917(v=sql.110).aspx
• SQL Server 2012 Resource Center (nice How-To videos)
http://msdn.microsoft.com/en-us/sqlserver/ff898410.aspx
• DQS Forum @ MSDN
http://social.msdn.microsoft.com/Forums/en-
US/sqldataqualityservices/threads
• TechEd presentation about DQS by Elad Ziklik
http://channel9.msdn.com/Events/TechEd/NorthAmerica/2011/DBI207