How to leverage eazyBI for combined planning and finance reporting across the Tempo product suite.
- Kris Siwiec, Lead Consultant - New Verve Consulting, United Kingdom
11. Why eazyBI?
Freedom IntegrationLove
To structure,
query and
present data.
Bridging gaps
in functionality
and understanding.
For the tool,
the community,
the ecosystem.
13. “He that breaks a thing to find out
what it is has left the path of wisdom”
Tempo
“tantrums”
Rigidness
Built-in reports & gadgets not designed for easy
customization
• Pre-defined views of data
• Limited number of
- presets
- units
- scales
15. “A wizard is never late…nor is he
early”
Tempo
“tantrums”
Laziness
Built-in reports & gadgets computed on-
the-fly
16. “My preciousss…”Tempo
“tantrums”
Possessiveness
• Built-in reports not available through dashboard
gadgets
• Limited choice of built-in gadgets
• Gadgets respect user permissions
i.e. not suitable for unprivileged users (e.g. JSD
customers)
17. • Rigidness
• Obstructivenes
s
• Laziness
• Possessivenes
s
• Love
• Freedom
• Integration
= riches & peace
WHY EAZYBI WHY NOT TEMPO
22. select
portfolio_name,
expired,
actual,
account_id,
account_key,
account_name,
rate_table_id,
rate_table_name,
cost_name,
id,
category,
workload_scheme,
team_id,
team_name,
team_role_id,
role_id,
role_name,
start_date_p,
end_date_p,
amount
from
(
select
concat('ALOC', aloc.`ID`) id,
fo.`NAME` portfolio_name,
if(fo.`END_DATE`>=CURDATE(),'OPEN','EXPIRED') expired,
if(po.`ACTUAL`=1,'Actual','Planned') actual,
# filter.`VALUE` query, # only used with AO_2D3BEA_FILTER and AO_2D3BEA_USER_FILTER
# filter.`reqcontent` query, # returns the full JQL filter query
substring_index(substring_index(filter.`reqcontent`, ''')', 1), '(''', -1) account_key_jql, # extracts the account key from JQL query
# po.`ACCOUNT_KEY` account_key, # comes up NULL
#account.`ID` account_id,
fo.`WORKLOAD_SCHEME_ID` workload_scheme,
fo.`RATE_TABLE_ID` rate_table_id,
po.`TEAM_ROLE_ID` team_role_id, # WARN: does not line up with role_id below
user.`ROLE_ID` role_id,
user.`ROLE_NAME` role_name, # for comparison with ID
user.`TEAM_ID` team_id,
user.`TEAM_NAME` team_name, # for comparison with ID
po.`CATEGORY` category,
# 'Salaries' type, # optional constant
po.`NAME` cost_name,
wage.`WAGE`*aloc.`PERCENTAGE`/100*8*(5 * (DATEDIFF(aloc.`END_DATE`, aloc.`START_DATE`) DIV 7) +
MID('0123455501234445012333450122234501101234000123450', 7 * WEEKDAY(aloc.`START_DATE`) + WEEKDAY(aloc.`END_DATE`) + 1, 1)) amount, # needs reviewing
aloc.`START_DATE` start_date,
aloc.`START_DATE` start_date_p,
aloc.`END_DATE` end_date_p
from AO_2D3BEA_POSITION as po join AO_2D3BEA_FOLIO as fo on po.`FOLIO_ID`= fo.`ID`
join AO_2D3BEA_ALLOCATION as aloc on aloc.`POSITION_ID`=po.`ID`
join AO_2D3BEA_WAGE as wage on wage.`POSITION_ID`=po.`ID`
join AO_2D3BEA_USER_INDEX as user on user.`USER_KEY`=po.`MEMBER`
# join AO_2D3BEA_USER_FILTER as filter on filter.`ID`=fo.`SAVED_FILTER_ID` # both AO_2D3BEA_FILTER and AO_2D3BEA_USER_FILTER are useless
join searchrequest as filter on filter.`ID`=fo.`SAVED_FILTER_ID`
) t1
join
(
select *
from
(
select ID account_id, NAME account_name, `KEY` account_key
from AO_C3C6E8_ACCOUNT_V1
) t3
join
(
select NAME rate_table_name, SCOPE_ID scope_id, SCOPE_TYPE scope_type, PARENT_ID parent_id
from AO_C3C6E8_RATE_TABLE
where SCOPE_TYPE='ACCOUNT'
) t4
on t3.`account_id`=t4.`scope_id`
) t2
#on t1.`account_key_jql`=t2.`account_key` and t1.`rate_table_id`=t2.`rate_table_id`
on t1.`account_key_jql`=t2.`account_key` and t1.`rate_table_id`=t2.`parent_id` # cannot join on rate_table_id as we need the parent rate table
23. • indirect data access
• high access time
• low access cost
• balance of granularity
and control
• direct data access
• low access time
• high access cost
• maximum granularity
and control
DATABASE API
27. Request timeouts
WHEN MORE DATA = BIGGER DATA
Request proxying
• Response time
• Response size
• Page size
• Cache response
• Format/paginate
• Publish response
28. 1h → 1m → 1s / account
Data processing time reduction on a 500-account portfolio
29. API coverage
WHEN MORE DATA = BETTER DATA
API stability
• Not all data officially
exposed
• Sometimes in the response
you get more than you
bargained on
• Public endpoints
often experimental
• Private endpoints
here be dragons
35. Wrap-up
DO
• Proxy your data
• Cache your data
• Use public endpoints
DON'T
• Presume performance
• Use DB queries
• Use private endpoints
DOs & DON'Ts
36. Questions?
Kris Siwiec • Lead Consultant • New Verve Consulting • new_verve
kris@newverveconsulting.com
37. Cheers!
Kris Siwiec • Lead Consultant • New Verve Consulting • new_verve
kris@newverveconsulting.com
Notas do Editor
This is the boring slide.
Here’s New Verve picking up a recent business award nomination, with our glamorous founder to your left.
We’re an Atlassian solution partner and a micro-business, proudly embracing the start-up culture. We like a personal approach and like to think that small is powerful and agile. Being a start-up, in practice we don’t really have fixed roles: we work collectively and our roles emerge naturally from our experience.
We’re going through an exciting growth period, with a couple more team members already having joined since this picture was taken.
Now that that’s out of the way, let’s talk about why we’re here.
A brainy quote to set us off.
Rather than showing off swanky reports, this talk will be about the technical challenges of implementing eazyBI reports as part of a wider ecosystem. Integration carries unique logistical considerations that are worth exploring.
In this scenario, we’re integrating with the Tempo product suite, and we’re going to look at why we could or should make particular architectural decisions.
3 why use eazyBI reports?
* love <3 (we’re besties)
* freedom! (to structure & query data)
* bridging gaps
* in functionality
(migration)
* in understanding
(transition)
3 why use eazyBI reports?
* love <3 (we’re besties)
* freedom! (to structure & query data)
* bridging gaps
* in functionality
(migration)
* in understanding
(transition)
We think it’s a real gift and we want to give back!
3 why use eazyBI reports?
* love <3 (we’re besties)
* freedom! (to structure & query data)
* bridging gaps
* in functionality
(migration)
* in understanding
(transition)
3 why use eazyBI reports?
* love <3 (we’re besties)
* freedom! (to structure & query data)
* bridging gaps
* in functionality
(migration)
* in understanding
(transition)
We feel that integration is a very interesting use case for eazyBI and a healthy attitude to boot.
You don’t build a wall between you & Mexico, or try and leave the continent you’re part of: you build a bridge across the border.
Short description of Tempo:
Tempo is a cloud-first software company that helps teams at more than 10,000 companies—SMBs and large-scale enterprises—collaborate, plan and schedule resources, manage budgets, and track time directly from their daily workflow.
Longer description of Tempo:
Tempo products are designed to enhance the efficiency of Atlassian's JIRA helping teams and businesses collaborate, plan, budget, track, and work smarter seamlessly in one central software platform. More than 10,000 customers in over 100 countries trust Tempo to be successful. Our customers include small startups to large-scale distributed enterprises such as NASA, Amazon, Paypal, Toshiba, Disney, BMW, and Princeton University. Tempo is one of the largest, award-winning, and bestselling add-on vendors in the Atlassian Ecosystem, and has an established network of more than 100 partners worldwide.
Don’t get us wrong, we’re not knocking Tempo at all! We’re Tempo partners as well, after all.
Tempo produces an excellent product suite that we think is by itself best-suited for manager-level oversight, rather than customer-level reporting.
Here’s why…
4 why not use Tempo built-in reports?
* pre-defined view of data
* permission considerations
(try exposing
an Account Budget Burn-up Chart or
an Account Time and Expenses Report
to unprivileged customers)
* performance challenges with large amounts of data
(try running the Steering Committee Report over a portfolio of 450 projects)
* embedding options
We’ve called these “Tempo tantrums” – they’re not really tantrums, but it has a nice ring to it.
They are based on some quotes from everyone’s favorite wizard. Dumbledore applied to be on the slides, but he wasn’t Jira-certified, so.
4 why not use Tempo built-in reports?
* pre-defined view of data
* permission considerations
(try exposing
an Account Budget Burn-up Chart or
an Account Time and Expenses Report
to unprivileged customers)
* performance challenges with large amounts of data
(try running the Steering Committee Report over a portfolio of 450 projects)
* embedding options
As a Tempo user, you’re interacting with Jira as a Jira user.
4 why not use Tempo built-in reports?
* pre-defined view of data
* permission considerations
(try exposing
an Account Budget Burn-up Chart or
an Account Time and Expenses Report
to unprivileged customers)
* performance challenges with large amounts of data
(try running the Steering Committee Report over a portfolio of 450 projects)
* embedding options
4 why not use Tempo built-in reports?
* pre-defined view of data
* permission considerations
(try exposing
an Account Budget Burn-up Chart or
an Account Time and Expenses Report
to unprivileged customers)
* performance challenges with large amounts of data
(try running the Steering Committee Report over a portfolio of 450 projects)
* embedding options
So, to summarize, we have hopefully made a very strong case ;-) for using eazyBI to deliver planning/financial reports above & beyond what Tempo comes stock with…
Fortunately, Tempo does expose its data via Active Objects database storage as well as REST API calls: now we will discuss which of these options is in our opinion better-suited for everyday integration with eazyBI.
5 getting your hands on the data
* database query import approach
* pros & CONS
+ direct data access
+ maximum granularity & control
- need to cross-reference data from multiple database tables
- need to reverse-engineer all calculations done by Tempo
(e.g. costs accrued)
* REST API import approach
* PROS & cons
+ indirect data access
(various measures precomputed upfront by the API)
+ good balance of granularity & control
- indirect data access
(must obey API limitations)
- need to pre-process data prior to publishing
eazyBI allows us to use various data source connectors: among these are a SQL query connector and a REST call connector.
Let consider the pros & cons of each.
5 getting your hands on the data
* database query import approach
* pros & CONS
+ direct data access
+ maximum granularity & control
- need to cross-reference data from multiple database tables
- need to reverse-engineer all calculations done by Tempo
(e.g. costs accrued)
* REST API import approach
* PROS & cons
+ indirect data access
(various measures precomputed upfront by the API)
+ good balance of granularity & control
- indirect data access
(must obey API limitations)
- need to pre-process data prior to publishing
The DBA is naïve & idealistic: he looks up to the sky and hopes for the best.
The hacker looks down from the sky: she’s already there and she knows what she wants.
Theory – a simple query will suffice, right!?
It turns out that the more data you would like to pull in, the harder you have to look, especially if you want to cross-reference various data points.
Imagine you’re trying to cross-reference some Tempo folios with their parent portfolios, related accounts, team allocations, issue worklogs…
Reality – by the time we’re finished with the query, we need to be a SQL grand-master, we’re at least middle-aged already, and this is just the beginning…
There is a lot of SQL in that query: you can’t see it, because it was never meant to see the light of day. The SQL was crap, BTW: both the movie and the code.
There’s a key differentiator between DB and REST data access, and that is the technical debt/cost carried by the data access operation. Let’s consider this in more detail.
Database
Pros:
* direct data access
* maximum granularity & control
Cons:
* need to cross-reference data from multiple database tables
* need to reverse-engineer all calculations done by Tempo (e.g. costs accrued)
API
Pros:
* indirect data access (various measures precomputed upfront by the API)
* good balance of granularity & control
Cons
* indirect data access (must obey API limitations)
* need to pre-process data prior to publishing (to avoid timeouts)
5 getting your hands on the data
* database query import approach
* pros & CONS
+ direct data access
+ maximum granularity & control
- need to cross-reference data from multiple database tables
- need to reverse-engineer all calculations done by Tempo
(e.g. costs accrued)
* REST API import approach
* PROS & cons
+ indirect data access
(various measures precomputed upfront by the API)
+ good balance of granularity & control
- indirect data access
(must obey API limitations)
- need to pre-process data prior to publishing
6 getting our hands on MORE data
* public API endpoints
(experimental)
* private API endpoints
(enter at your own risk)
6 getting our hands on MORE data
* public API endpoints
(experimental)
* private API endpoints
(enter at your own risk)
Request timeouts – when the API call takes too long or doesn’t return
* Too much pre-processing – weak point of native APIs, linked to below
* Too much data in response – some calls return more than you’d expect!
* Response size too large
* Page size too large
Direct vs proxied calls
A naïve implementation importing data directly from public endpoints can take upwards of 1h/account, depending on the combination of data required
A slightly smarter implementation which pre-processes some data upfront can take 1m/account
An optimized implementation which pre-processes and caches all data at the proxy can be as efficient as 1s/account
6 getting our hands on MORE data
* public API endpoints
(experimental)
* private API endpoints
(enter at your own risk)
Emphasis on wee
6 getting our hands on MORE data
* public API endpoints
(experimental)
* private API endpoints
(enter at your own risk)
Here are the basic building blocks for our data processing workflow.
There are many options for choosing your data sources: it may be a combination of any number of data providers, from official endpoints to screen scrapes…
A convenient option for implementing a quick & dirty microservice is ScriptRunner for Jira.
Data can be cached anywhere convenient: from basic user properties to remote data stores in S3.
Here’s an example anonymized snapshot of a live eazyBI report pulling in a combined planning/financial dataset based on a cube of multiple Tempo sources.
We’ve put private endpoints under DON’Ts, as they’re a health & safety hazard, so don’t use private endpoints…but really, DO! :-) Just be aware of the risks…