This document discusses building data lakes and analytics on AWS. It covers challenges with big data like volume, velocity, and variety. An AWS data lake can quickly ingest and store any type of data. The data lake includes analytics, machine learning, real-time data movement, and traditional data movement. Metadata management is important for data lakes. AWS Glue crawlers can discover data in various formats and populate the data catalog. Different tools like Amazon Athena, Amazon EMR, and Amazon Redshift can be used for analytics depending on the user and use case. Machine learning benefits from big data, and a data lake supports agility in machine learning.
40. Federal Geospatial Platform
The Leader in Geospatial for the Government of Canada
Easy access to GC “AAA” Geospatial Data
Standards-based formats
RESTful web services
ISO Metadata
OGC
Simple workflow to assess, visualize and publish
Re-usable viewer on GitHub
Collaborative Mapping Environment on Esri’s ArcGIS Online
FGP Geo-Community Cloud Platform as a Service (PaaS) on AWS
A GC standards compliant Geospatial Platform On-Demand
41. …OBJECTIVES 2018-20
Make Government of Canada Earth Observation information more easily
available to Canadians
Access, Visualization and Analysis functionality for EO and Spatial
Information using the Federal Geospatial Platform (GC Tool)
Enhanced imagery visualization options (past/present time-series)
On-the-fly imagery processing (projection, class renderings, dynamic mosaics)
Geoanalytics against near real time GC imagery on-demand
46. FGP Geo-Community Cloud
2017-18 Proof of Concept on AWS (complete)
2018-19 Foundation Laid – SSC Brokered Cloud
FGP “Core Solution Stack”
2019-20
On Demand Processing Capabilities via API Gateway
Geospatial Managed Storage - host your own geospatial data
Support multiple “portals” from a common GC ecosystem
Concurrently…
Innovation Zone
Sandbox Enviros for broad-based Geospatial R&D P/T/A
AI and Machine Learning against Geospatial + EO integrated with FGP Platform as a
Service
47. Geo-Community Cloud – AWS Services
ca-canada-1a
Public Subnet
Private Subnet
Private Subnet
App Tier
Web Tier
DB Tier
Amazon Route 53
WAF Web Application
Firewall
Internet Gateway
Classic Load Balancer
EC2 Instances
Application Load Balancer
Elastic Block Storage
NAT Gateway
Database
S3 Bucket
Glacier Storage
NAT Gateway
NAT Gateway
Auto Scaling
Accessible
Authoritative
800 datasets and growing
Think enterprise, not silos
Build once, use many times, for the benefit of all, including common approaches and solutions
Think horizontal not vertical
Cloud First
Use existing GC standards and tools
Context
Climate Change
Cumulative Effects
Current launch window is Feb 18-24 2019
Qty 3 satellites all on same SpaceX Falcon 9 launch vehicle from Vandenberg Air Force base in California.
The 3 satellites will be released 3 minutes apart and then later once ‘fully woken up’ they will be moved into final position which is evenly spaced. (120 degrees apart)
1.4 TBytes/day, just under 1 PByte/year
RCM data policy is TBD but even free data for the public requires individual accounts due to Remote Sensing Space Systems Act (RSSSA). They will most likely be valuable added products that could be open (FGP/OpenMaps), but qty is unknown.
Question is this… How will it be made useful?
We decided to tackle the technical challenge of processing EO data our way – to create web services, which make it possible for everyone to get data in their favorite GIS applications using standard WMS and WCS services.
Also, processing on demand with Esri’s Image Server and GeoAnalytics Server.