3. Developer
Portal
Per app
report
App Tile App summary Adoption
Downloads
In-app
purchases
Usage
Ratings Quality Finance
Per Developer
report
Financial
summary
Store report
Explore Store
trends
Developer Analytics – Sitemap
This is an aggregation of downloads,
ratings, quality, finance, purchases and
in-app purchases report to give an
overview of all the metrics useful to the
developer.
This is an summary of downloads,
purchases, in-app purchases, usage
metrics
It also shows Discovery metrics – App
discovery/conversion funnel and app’s
web listing page’s web referrers
distribution
7. Architecture overview
• Component Grouping
• Copy, Validate & Publish SSAS Cube
• Extract and Load with basic processing
• Extract, Transform and Load
• Data Mining for Store Trends
• Workitem Framework Block Diagram
8. Architecture overview – Cube PAS
• Copy, Validate & Publish SSAS Cube
• Input – Cube backup
• Processing
• Restore cube backup on BE
• Validate Schema
• Publish
• Restore new cube to Passive instance
• PAS - Passive Active Sync
• Copy cube to passive instance of all FEs
• Flip Passive to Active instance.
Cube
(Backup)
DCS
Pipeline
DLS
Pipeline
Temp
Storage
Cube
SSAS
PAS
Pipeline
Telemetry
Cube
SSAS
Cube
SS
AS
FE Cube
SSAS
Telemetry
Cube
SSAS
Cube
SS
AS
FE Cube
SSAS
Passive
Active
FE
Portal
9. Architecture overview – Basic ETL
• Extract and Load with basic processing
• Input – Raw data
• Processing
• Validate Schema of raw data
• Filter unwanted data
• Remove duplicate entries
• Load
• Re-process SSAS Cube and Publish to FE using PAS
• Copy processed data to FE DB
FE
Portal
Staging DB DWHProcessing
Cube
SSAS
PAS
Pipeline
Store
Cube
SSAS
Store
Cube
SSAS
FE
Cube
SSAS
Feed Data Feed
Pipeline
App Store
Analytics
DB
App Store
Analytics
DB
FE DB
DPS
Pipeline
10. Architecture overview – ETL
• Extract, Transform and Load
• Input – Raw data
• Processing
• Data Fetch
• Validate Schema of raw data
• Filter unwanted data
• Remove duplicate entries
• Data normalization
• Data Processing – Data Aggregation
• Load
• Copy processed data to FE DB
FE
Portal
Staging
DB
DWH
Feed
Data Feed
Pipeline
App Store
Analytics
DB
App Store
Analytics
DB
FE DB
Data Move
Processing
Source
DB
Data
Fetcher
11. Architecture overview - Mining
• Data Mining
• Input – Acquisition data
• Processing
• Data Fetch
• Validate Schema of raw data
• Filter unwanted data
• Remove duplicate entries
• Data normalization
• Data Processing
• Data Preparation – Data Filtering, Data Sampling, Data Discretization
• Data Mining
• Output
• Mining Rules i.e. Observed data trends
FE
Portal
DWH
App Store
Analytics
DB
App Store
Analytics
DB
Rules DB
Data
Preparator
Source
DB
Data
Fetcher
Sampled
Data Store
Data
Mining
12. Architecture overview
• Workitem Framework
DB Based Workitem
Queue
Workitem
Generator1
WI Type 1 State 1
WI Type 1 State 2
WI Type 4 State 10
Workitem
Generator2
WI Processor
Type 1 State 1
WI Processor
Type 4 State 10
WI Processor
Type 1 State 2
WI Processor
Type 4 State 10
13. Architecture overview
Partner
Feed DB
RnR
Feed
ComTxn
Feed
DevAnalytics
Staging DB
MiningDWH
DB
MiningDWH
DBMiningDWH
DB
Sampled
Data Store
Sampled
Data Store
Data
Fetcher
WIFx
Data Feed
Pipeline
Data
Fetcher
WIFx
Mining Data
Preparator
WIFx
Mining Data
Miner – WIFx,
SSAS
Data Aggregator
WIFx
Move Data to FE
WIFx
Data Feed
Pipeline
Payout
Cube
SSAS
Telemetry
Cube
DCS
Pipeline
DLS
PipelineTemp
Storage
Telemetry
Cube
SSAS
PAS
Pipeline
Telemetry
Cube
SSAS
A. Te
le
m
Telemetry
Cube
SSAS
Manual
Cert DB
Data Feed
Pipeline
Certification
Reporting DB
Mining
Rules DB
Mining
Rules DB
Mining
Rules DB
DevAnalyticsFE
DB
DevAnalyticsFE
DBDevAnalyticsFE
DB
Analytics
Staging DB
Analytics
DWH
Load
DWH
SSIS
Store
Cube
SSAS
PAS
Pipeline
Store
Cube
SSAS
Store
Cube
SSAS
Store
Cube
SSAS
App
Catalog
Feed
Data Feed
Pipeline
Control DB
Data Fetch Workitem Generator
Data Mining Workitem Generator
Aggregation Workitem Generator
Settelment
Feed
CP
Feed
ForEx
Feed
App Store
Analytics DB
App Store
Analytics DB
App Store
Analytics DB
DPS
Pipeline
Tracking DB
Currency
Conversion
Tracking DB
Dev
Portal
Dev
Portal
Dev
Portal
Dev
Portal
Mining
Rules DB
Mining
Rules DB
Onboarding
DB
Stability Score
Pipeline
Stability
Score Feed
Data Feed
Pipeline
BI Cube
DCS
Pipeline DLS
Pipeline
BI Cube
SSAS
PAS
Pipeline
Telemet
ry Cube
SSAS
Tele
m
e
BI Cube
SSAS
14. Architecture overview – Cube PAS
Telemetry
Cube
DCS
Pipeline
DLS
Pipeline
Temp
Storage
Telemetry
Cube
SSAS
PAS
Pipeline
Telemetry
Cube
SSAS
Telem
etr
y
Cu
be
SS
AS
Telemetry
Cube
SSAS
Stability Score
Pipeline
Stability
Score Feed
BI Cube
DCS
Pipeline DLS
Pipeline
BI Cube
SSAS
PAS
Pipeline
Telemetr
y Cube
SSAS
Tele
m
e
t
r
y
C
u
b
e
S
S
A
S
BI Cube
SSAS
15. Architecture overview – Cube PAS
Fundamental Telemetry
5R – RnR
DSP Telemetry
Downloads
In-app purchases
Usage
Quality
Ratings and Reviews
Finance
Finance Summary
Adoption page
Explore Store trends
Page views + Conversion funnel
Primes + CP + CTP
PFDb -- PartnerFeedDb
BI Cube
Telemetry Cube
RnR BCP Feed
ComTransactionDb
16. Architecture overview
ComTxn
Feed
Data Feed
Pipeline
Payout
Cube
SSAS
Analytics
Staging DB
Analytics DWH
Load
DWH
SSIS
Store
Cube
SSAS
PAS
Pipeline
Store
Cube
SSAS
Store
Cube
SSAS
Store
Cube
SSASSettelment
Feed
CP
Feed
ForEx
Feed
App Store
Analytics DB
App Store
Analytics DB
App Store
Analytics DB
DPS
Pipeline
Currency
Conversion
Data Feed
Pipeline
18. Architecture overview – Aggregate DB and Mining
Partner
Feed DB
RnR Feed
DevAnalytics
Staging DB
MiningDWH
DB
MiningDWH
DBMiningDWH
DB
Sampled
Data Store
Sampled
Data Store
Data
Fetcher
WIFx
Data Feed
Pipeline
Data
Fetcher
WIFx
Mining Data
Preparator
WIFx
Mining Data
Miner – WIFx,
SSAS
Data Aggregator
WIFx
Move Data to FE
WIFx
Manual
Cert DB
Data Feed
Pipeline
Mining
Rules DB
Mining
Rules DB
Mining
Rules DB
DevAnalyticsFE
DB
DevAnalyticsFE
DB
DevAnalyticsFE
DB
App
Catalog
Feed
Data Feed
Pipeline
Control DB
Data Fetch Workitem Generator
Data Mining Workitem Generator
Aggregation Workitem Generator
Tracking DBTracking DB
19. Architecture overview – Aggregate DB and Mining
Fundamental Telemetry
5R – RnR
DSP Telemetry
Downloads
In-app purchases
Usage
Quality
Ratings and Reviews
Finance
Finance Summary
Adoption page
Explore Store trends
Page views + Conversion funnel
Primes
PFDb – PartnerFeedDb (Aggregation)
BI Cube (Copy to FEs)
Telemetry Cube (Copy + Stability feed generation)
RnR BCP Feed (Aggregation)
ComTransactionDb (Aggregation)
20. Architecture overview – Front End Block Diagram
ReportDefinitions.xml
CubeQueries.xml
DbQueries.xml
Report
Manager
Cube DAL
DB DAL
DataAccessService
DAL Interface
DeveloperAnalytics
Controller (MVC)
Highcharts
Agg DB
BI Cube
DevAnalyticsFE
DB
TransactionDB DAL
Mining Rules API
Agg DB
App Store
Analytics DB
Telemetry
Cube
Store
Cube
21. Architecture overview
Heading
Heading
Component + Physical Architecture for Dev Analytics for Windows BlueComponent + Physical Architecture for Dev Analytics for Windows Blue
Sources
Store(Ratings/
Review as bcp)
5/8 MB, 1/12 hrs
50 MB/Day 7MB/Day10 GB, 1/Day
Partner
Feed(Acquisitions,
Page Referrals,
Conversions)
2GB/Day
Manual Cert
DB
Telemetry(Quality,
Usage as a cube)
DSP Telemetry aka
BET (Acquisitions,
Conversion,
Referrers as a
cube)
Upgrade Model: FE is meant to be active even when BE is down. WebStore Layer is updated first. BE Layer
components next. FE components last. V (N) FE components must work with V(N+1) pipelines.
Availability : 3 way replication for FE stores . Replication Load prevents addition of a new DC. BE is Active/Passive.
Procedure to do failover is not tested or captured as a runbook.
Scalability : FE scale model is based on data replication. BE scale model is adhoc.
Portal
WebStore/SQL
Local Caching(for
metadata)
Internal Partners Only
Certification
Reporting DB
FE Layer(Active/Active))FE Layer(Active/Active))
DS2DS1
Backend Layer(Active/Passive)Backend Layer(Active/Passive)
Mining DWH
DB(partition 0)
Data Fetcher
WIFx(Partner
Feed DB)
Dev Analytics
Staging DB
Control DB
Mining DWH
DB (partition 1)
Mining Data
Preparator
WIFx
Data Feed
Pipeline(RNR,
Manual Cert,
App Catalogue)
Data
Aggregator
WIFx
PS1
Data Fetcher
WIFx(Partner
Feed DB)
Data Fetcher
WIFx(Dev
Analytics
Staging DB)
Mining Data
Preparator
WIFx
Sampled
Data
Store(partiti
on 0)
Sampled
Data Store
Data Miner
WIFx
PS2
Sampled
Data
Store(partiti
on 1)
Data Miner
WIFx
Tx Catalog (Finance,
as bcp)
300MB/Day
7MB/Day
SSAS Cube
Analytics BE CubeAnalytics Pipeline BE
Backend LayerBackend Layer
Analytics
Staging DB
Analytics DWH
DB
Store Cube
Telemetry
Cube
Store BI
Cube
BET
DCS
DLS
PAS
DCS
DLS
PAS
DCS
DLS
PAS
Mining WI
Monitoring WIFx
Mining Data
Preparator
WIFx
DC1
Payout Cube
Frontend Cube ServerFrontend Cube Server
Telemetry
Cube
Store Cube
Store BI
Cube
Frontend Cube ServerFrontend Cube Server
Telemetry
Cube
Store Cube
Store BI
Cube
Frontend Cube ServerFrontend Cube Server
Telemetry
Cube
Store Cube
Store BI
Cube
Frontend DB Server
App Store Analytics DB
Dev Analytics FE DB
Mining Rules DB
Onboarding DB
Frontend DB Server
App Store Analytics DB
Dev Analytics FE DB
Mining Rules DB
Onboarding DB
Stability Score Feed
All Dev. Analytics pages are instances of Reports and each report is rendered as per its report definition. A report definition includes all information to define the layout and look and feel of the report (ex. filters applicable, types of charts, chart layout, color scheme etc.). All report definitions are maintained in ReportDefinition.xml. ReportDefinition.xml contains other definitions too (ex. localizable dimensions, mining price tiers etc.) all of which get used during report generation or rendering. The Report Manager module loads and provides access to the report definitions.
Within each report, there could be multiple filters and charts present. Each chart and data driven filters (non-static filters) are rendered by firing a query against a data source (Cube / DB). Each such query is associated with a unique id (per data source) and is referred to in the ReportDefinition.xml to define the data source and query id for the chart / filter. For Aggregated DB, the queries are stored procedures and the query id to stored procedure mapping is maintained in DBQueries.xml. For cubes, both query definitions and the query id to query definitions mapping are maintained in CubeQueries.xml.
Since Dev. Analytics pages render information from multiple sources (Cube / DB), a uniform access mechanism is provided through the DataAccessService layer. DataAccessService (DAS) layer creates an appropriate DAL class instance which in turn queries the data source and returns the data. The respective DAL interfaces create the query (for cube only) and query the underlying data source to return data.
Analytics FE follows an MVC pattern. Typical request response flow is as follows:
Client makes a web request (which contains Product Guid and other information) for a full page load
Controller verifies product to developer association
It get requested report’s definition and asks DAS to return the data for rendering filters
DAS accesses DAL instance which queries the data source and returns data
The returned data is used to populate the view models and a view is returned to the client
The view upon loading makes AJAX calls to get and render chart data
Again, controller gets requested report’s definition and asks DAS to return the data for rendering for rendering charts
The returned data is used to populate the view models and a view is returned which is rendered on the client.
When user applies some filters, client makes an AJAX call to get filtered data
Controller receives filter request, gets requested report’s definition and asks DAS to return the data for rendering for rendering charts