A Beginners Guide to Building a RAG App Using Open Source Milvus
Big data and its potential in integrity and operational reliability
1. PIMS of London
The Structured Approach to Asset Management
Big Data and its Potential for Integrity
Management and Operational Reliability
Richard Matthews
Operations Director
PIMS of London Ltd
2. What is Big Data?
The Structured Approach to Asset Management
Data is a torrent flowing in every area of the global economy
Companies churn out a huge volume of transactional data
Millions of networked sensors embedded in the physical
world sense, create and communicate data
Sensors, PCs and smart-phones allow billions of people
around the world to contribute to Big Data
However, big data can play a significant economic role to the
benefit of industry, national economies & citizens and can
create significant value by enhancing productivity and competitiveness in the global
economy
Digital data is now everywhere – in every sector and every economy, every organisation and
every digital user
Recent research shows that we are on the leading edge of a tremendous wave of
innovation, productivity and growth driven by Big Data
Big Data is now relevant for business leaders across every sector, as products, services and
their competitiveness stand to benefit from its application
Most large organisations are currently in the early stages of Big Data development efforts –
many piloting ways to mine existing data more effectively and to improve visualisation &
analytics in order to improve outcomes and decision making
3. Capturing the Value of Big Data
Pipeline Services
The Structured Approach to Asset Management
$300 billion
potential annual value to US health care (more than double the
total annual health care spending in Spain)
€250 billion
potential annual value to Europe’s public sector administration
(more than GDP of Greece)
$600 billion
potential annual consumer surplus from using personal location
data globally
60%
potential increase in retailers’ operating margins possible with big
data
4. What is Big Data?
The Structured Approach to Asset Management
Characterising the dimensions of Big Data:-
Volume – data at scale
Variety – data in many forms
Velocity – data at speed – streaming real time data
The ability to store, aggregate, align data and use the results
for deep analysis will become more accessible and important
The extraction of conclusions from data are also improving,
as increasingly sophisticated software and techniques combine with ever growing computing
horsepower, more cost effective hardware and data storage capability
Although the number of smart-phones has recently reached 1 Billion and currently growing at
more than 20% per year however, at the same time the number of industrial networked sensors
is increasing at more than 30% per year
This Big Data phenomenon is not just about smartphones, PCs
and social networking, this is also about industrial applications
Big Data” has been regarded as a term used by “softer” industries
to track people‟s behaviours, buying tendencies, sentiments, etc.
However, Big Data already has applications in the upstream oil &
gas sector...
So what about its potential for pipeline integrity and operational reliability?
5. Big Data Potential
The Structured Approach to Asset Management
Where does Big Data have potential to improve pipeline integrity for Pipeline Operators?
Data Mining - Operators with Integrity Strategy of Inspect & Repair
Improved analytical tools and skills to incorporate and align a variety of datasets for
enhanced defect characterisation & assessment of coincident anomalies for improved
decision making
Enables more sophisticated integrity assessments necessitating fewer repairs, more
accurate assessment of corrosion growth rates to extend remaining life and re-inspection
intervals
Improved accountability - operational and integrity KPI tracking and visualisation through
digital dashboards
On a recent project: Holistic integrity assessment including crack susceptibility
analysis, alignment of multiple data sets, fracture mechanics
and lab testing to determine immediate pipeline repairs
Immediate pipeline crack repairs reduced from 100 to 5, minimising
operational disruption and reducing costs whilst managing risk
6. Big Data Potential
The Structured Approach to Asset Management
Data Mining + Business Intelligence - Data Management & Risk Assessment
With an increasing culture, expectation and reliance on data driven decisions, more
focus will be required on data management
Big Data can drive more sophisticated algorithms and analysis to deliver robust and
more accurate risk assessments and improved data driven business decisions
On a recent project:• Alignment and analysis of network wide datasets used for the development of
pipeline condition & criticality algorithm to prioritise pipelines for replacement v
inspect and repair
Cost benefit analysis on a single pipeline demonstrated US$40M benefit and
replacement pipeline capital payback from production of <1 month
7. Big Data Potential
The Structured Approach to Asset Management
Data Mining + Business Intelligence + Visualisation - Integrated PIMS Programme
Data management and software becoming an increasingly essential pipeline integrity
management tool for data driven decision making
A shift from reactive pipeline integrity management and „inspect and repair„
philosophies to a structured and integrated approach for proactive pipeline integrity
management
Management Systems
Data Management
Pipeline IM
Programme
Design
Pipeline
Goals &
Objectives
Prioritise
Susceptible
Segments
Strategic
Planning
Data
Gathering &
Integration
Risk &
Integrity
Response
Communication
Plan
Risk
Assessment
Quality
Plan
Change
Management
Organisational
Effectiveness
Procedures
Development
Integrity
Management
Plans
Pipeline
Performance
Evaluation
Repair &
Rehabilitation
Repair &
Rehabilitation
Execution
Repair &
Rehabilitation
Planning
PIMS
Performance
Plan
Integrity
Evaluation
Direct
Assessment
Fitness-for
Purpose
In-Line
Inspection
Training &
Competency
Emergency
Response
Planning
Operating Effectiveness
8. Big Data Potential
The Structured Approach to Asset Management
On a recent project: PIMS system implemented with Business processes structured to drive dataflow from one step
of the process to the next
Proactive management of pipeline integrity, including all pipeline data, Risk
Assessment, Integrity Management Plans, budgets, workorders, Inspection results, reports and
repair execution
Store and alignment of above ground survey & ILI data, infield NDT & repair records, ROV/SSS
survey data and interfaced with GIS and SAP PM to update records for next Risk Assessment
Incorporated Risk Based Inspection management of static equipment assets within the same
software system and following consistent business processes to manage operational reliability
Implementation of Integrated PIMS and RBI system on a large network, driven
by business processes to ensure proactive Integrity management of onshore
and offshore pipelines and facilities transportation system
9. Big Data Potential
Summary
The Structured Approach to Asset Management
Big Data provides real potential to shift from periodic capture and analysis of lagging
data to real-time capture of leading data to monitor and control pipeline integrity
Pipeline threat modeling and risk assessment to identify critical parameters to
be visualised on digital dashboard to maintain pipeline condition and prevent
deterioration
Field sensors clustered into low cost Wireless Sensor Networks (WSN) gather
and send real-time or batch data using existing IT infrastructure, cellular
networks or via satellite.
Sensors and WSNs can operate in remote pipeline locations using batteries or
solar panels
Enables real time remote sensing of critical operating data such as fluid
composition, H2S, CO2 etc, dewpoint, chemical inhibitor residuals, bacterial
count, pipeline strike detection etc
Could also use satellite imagery to monitor pipeline ROW for 3rd party
encroachment, potential pilferage, leaks etc
10. Big Data Potential
Summary
The Structured Approach to Asset Management
Capitalising on the Volume, Variety and Velocity of Big Data has the ability to advance
integrity and operational reliability from Proactive to Predictive and Preventative
Rather than periodic ILI inspections for measurement of existing defects to determine
historical corrosion growth rates, real-time monitoring and analysis of critical pipeline
parameters allows implementation of threat mitigation plans before pipeline
deterioration
Traditional inspections used to confirm and verify pipeline condition and to tailor
and optimise integrity mitigation plans such as chemical injection programmes etc
This real-time data driven approach to condition monitoring and pipeline integrity
management has the potential to significantly extend asset life, optimise costs and
improve safety whilst managing risk
Big Data has great potential for real-time capture, visualisation and analysis
of critical pipeline threat and deterioration parameters to monitor pipeline
condition and proactively manage integrity & operational reliability
What is Big Data?Data is a torrent flowing in every area of the global economyCompanies churn out a huge volume of transactional data – trillions of bytes of information about customers, suppliers and operationsMillions of networked sensors embedded in the physical world – mobile phones, smart meters, industrial sensors and machines that sense, create and communicate dataSensors, PCs and smart-phones allow billions of people around the world to contribute to the amount of big data availableHowever, strong evidence is emerging that big data can play a significant economic role to the benefit of industry, national economies and citizens and can create significant value byenhancing productivity and competitiveness in the global economy Digital data is now everywhere – in every sector and every economy, every organisation and every digital userRecent research shows that we are on the leading edge of a tremendous wave of innovation, productivity and growth, all driven by big data & consumers and companies in every economic sectors are seeking to exploit its potentialBig data is now relevant for business leaders across every sector, as products, services and their competitiveness stand to benefit from its applicationMost organisations are currently in the early stages of big data development efforts – many piloting ways to mine existing data more effectively, improve visualisation and analystics in order to improve decision making and business outcomes
Why is Big Data capturing so much attention Well, it has been said that before long Human knowledge will be doubling every second!A Recent report from McKinsey Global Institute stated that Big Data could deliver the following:-$300 billionpotential annual value of Big Data to US health care—more than double the total health care spending in Spain€250 billionpotential annual value of Big Data to Europe’s public sector administration—more than GDP of Greece$600 billionpotential annual consumer surplus from using personal location data globally60%potential increase in retailers’ operating margins are possible with big data
So, Big Data is expanding on 3 fronts at an increasing rate:-Data Volume – Big Data means large scale volumes of dataData Variety – data in many different formsData Velocity – data at speed such as the streaming of real time dataTherefore, the ability to store, aggregate, combine & align data and use the results to perform deep analysis will become ever more accessible and importantThe extraction of conclusions from data are also improving - as increasingly sophisticated software and techniques combine with ever growing computing horsepower and more cost effective hardware and data storage capabilityAlthough the number of smartphones has recently reached 1 Billion and is currently growing at more than 20% per year however, at the same time the number of industrial networked sensors is increasing at more than 30% per yearThis Big Data phenomenon is not just about smartphones, PCs and social networking -this is also about industrial applications The term “big data” has historically been regarded by the oil and gas businessas a term used by “softer” industries to track people’s behaviors, buying tendencies, sentiments, etc.However, Big data has applications in the upstream oil and gas exploration sector and the total amount of oil and gas data is forecasted to double in the next 2 years.so what about the potential for Big Data in pipeline integrity and operational reliability??
So, where does Big Data have the potential to improve integrity and reliability for pipeline operators?Improvements in Data Mining provides benefits for Operators that manage integrity, primarily through a strategy of ‘Inspection and Repair’, by fully utilising the variety of pipeline data available Improved analytical tools and skills to incorporate and align a variety of datasets for enhanced defect characterisation & assessment of coincident anomalies for improved decision making – The variety of data can include ILI defect data, Above ground survey data, CP data, product data and operational parameters, GPS, Topography, Internal corrosion modelling,Flow modelling etcThis enables more sophisticated integrity assessments, necessitating fewer pipeline repairs, more accurate assessment of corrosion growth rates to extend pipeline remaining life and re-inspection intervalsIt provides Improved accountability - operational and integrity KPI tracking and visualisation through the increasing use of digital dashboardsAs an example - On a recent project:-A holistic integrity assessment including crack susceptibility analysis, alignment of multiple data sets, fracture mechanics and lab testing to determine immediate pipeline repairs required after inspectionData alignment and sophisticated Engineering Critical Assessment of high pH SCC were used to reduce the number of repairs from 100 to 5 whilst managing risk, reducing costs and minimising operational disruptionSpatial Data aligned:-Age, pressure, stress level, temperature, topographyIC & EC, crack defect depth, orientation as reported – multiple MFL and caliper runs and USCD – against age at time of inspectionDefect depth, orientation, WT – as measured in ditch after forensic digs including soil data, coating assessment, pipe to soil potential etcMaterial properties – confirmed through material testingConstruction and installation details etc
Improvements in Data Mining + Business Intelligence driven by the volume & variety of data available, provides improvements for Operators who also utiliseData Management & Risk Assessment to determine their pipeline Inspection and repair programmeWith an increasing culture, expectation and reliance of data driven decisions, more focus will be required on data management (capturing, structuring and management of data)Big Data can drive more sophisticated algorithms and analysis to deliver robust and more accurate risk assessments and improved data-driven business decisions As an example - On a recent project:- We carried out the alignment and analysis of network wide datasets used for the development of a pipeline condition & criticality algorithm to prioritise pipelines for replacement rather than continuing to inspect and repair Cost benefit analysis on a single pipeline demonstrated a US$40M benefit and the replacement pipeline capital payback from production was less than 1 month
For pipeline Operators with an Integrated Pipeline Integrity Management strategy they benefit from Data Mining, Business Intelligence and VisualisationIt is expected that Data management and software will become an increasingly essential pipeline integrity management tool for data driven decision makingA shift from reactive pipeline integrity management of ‘inspect and repair’ campaigns to a structured and integrated approach for proactive pipeline integrity managementClick to explain Pipeline Integrity Management system
As an example -On a recent project:-We implemented a PIMS system along with Business processes, structured to drive dataflow from one step of the process to the next This enabled Proactive management of pipeline integrity and Risk Based Inspection, including all pipeline and static equipment data, Risk Assessment, Integrity Management Plans, budgets, workorders, Inspection results, reports and repair executionThis systems is used to store and align above ground survey & ILI data, infield NDT & repair records, ROV/SSS survey data and interfaced with GIS and SAP PM to update records for next Risk AssessmentIt Incorporates Risk Based Inspection management of static equipment assets within the same software system and following consistent business processes to manage operational reliabilityImplementation of an Integrated PIMS and RBI system interfaced with GIS and SAP on a large network and driven by business processes ensures the proactive Integrity management of an onshore & offshore pipelines and facilities transportation system
However, Big Data has real potential to shift from periodic capture and analysis of lagging data to real-time capture of leading data, in order to monitor and control pipeline integrityThe PIMS system pipeline threat modeling and risk assessment would be used to identify critical parameters to be monitored, assessed and visualised on a digital dashboard in order to maintain pipeline condition and to prevent deterioration Low cost field sensors clustered into Wireless Sensor Networks (WSN) gather and send real-time data using existing IT infrastructure, cellular networks or via satellite.Standalone Sensors and WSNs can operate in hazardous areas and remote locations using batteries or photovoltaic solar panelsThis enables real time remote sensing of critical operating data such as fluid composition (H2S, CO2 etc), flow rate, dewpoint, chemical inhibitor residuals, bacterial count, pH, soil & CP data and even pipeline strike detection etcThis could also incorporate other relevantdata such as satellite imagery to monitor pipeline ROW for 3rd party encroachment, potential pilferage, leaks etc
Capitalising on the Volume, Variety and Velocity of Big Data, this concept has the ability to advance integrity and operational reliability from Proactive to PredictiveRather than periodic ILI inspections for measurement of existing defects to determine historical corrosion growth rates - real-time monitoring and analysis of critical pipeline parameters allows implementation of threat mitigation plans before pipeline deteriorationTraditional ILI inspections and above ground surveys can be used to confirm and verify pipeline condition and also to tailor integrity mitigation plans such as operational pigging procedures and optimisation of chemical injection programmes etcThis real-time data driven approach to condition monitoring and pipeline integrity management has the potential to significantly extend asset life, optimise costs and improve safety whilst managing riskSo, to conclude…Big Data has great potential for real-time capture, visualisation and analysis of critical pipeline threat and deterioration parameters in order to monitor pipeline condition and proactively and predictively manage integrity & operational reliability