The road to connected architecture

Today I’d like to share with you some of the things we learned in 20 years of ‘doing BI’ and
what we’ve learned through experience.
Most of the things I’m going to tell you probably sound familiar, but when I think about the
discussions I have with BI teams or managers, and when I read what people discuss about on
LinkedIn or at meetups, I feel we draw different conclusions and do our work differently.
Today is an attempt to explain a little why we got here and what that means for you, in your
every day challenges with people and information, in a professional environment.
[Note: this presentation takes you through the thinking in leaps and bounds, there is much more
to it than outlined here, of course.]
1© Free Frogs 2018
Free Frogs Klantendag 24 mei 2018

When we started our careers, which happened to be Business Intelligence, we were in the
fortunate circumstances, that we worked with people who really were invested in iterative
development methods.
From the first start on, we learned about a progressive and evolutionary way of working, that
step by step iterated from an idea to tangible IT products. In our case, dashboards and data
warehouses.
We were even more lucky that the focus was always on bringing together people from different
disciplines and what to do to make the exchange of ideas and knowledge as fluid as possible.
Collaboration was key.
2© Free Frogs 2018

A few years later, this way of working was published in a form which was called ‘the manifesto
for agile software development’.
We have spent many sessions on what the principles of the manifesto mean, and how to put
them into practice.
3© Free Frogs 2018

Flash forward to 2018 and “Agile” has become a thing of its own.
Many opinions can be found on what it means, how to work with it, and what it is doing, but in
the end it all boils down to a very pragmatic approach to get stuff done: you bring together
people with different skills and from different business contexts within your organisation, and
you start to develop stuff in small iterations. You iterate to validate everyone’s ideas about what
we are supposed to be creating, learn from putting what we’ve created to use, and adjust and
extend from what we’ve learned.
4© Free Frogs 2018

I’ve found this way of working is very apt to BI or analytics, because it reflects what we have
learned in BI: “people can tell you what they need, once they have seen it”.
Go to a user and ask her of him what information they require, and all you get back are blank
stares or conceptual ideas which are not enough to be able to create reports, dashboards or an
analysis.
5© Free Frogs 2018

Although results improve greatly with an agile approach to developing information, I found that
it isn’t easy to let the results stick.
Time and time again, and still today, I find myself in assignments where our help is asked to
improve the BI or analytics effort. Most of the time, someone before us has done a project or
created a new data warehouse. After some time, the information landscape looks chaotic.
Or the other way around, we’ve delivered a great project and a few years later the state of
things isn’t as great anymore.
I have been thinking about this a lot. It is my biggest source of frustration.
It seems we experience a “groundhog day” situation and we are not capable to secure the
results of the work we’ve done.
In the last years I’ve come to realize that a part of the explanation is that we, as a BI discipline,
have a perspective that is technology oriented. Our focus is very much from development
outwards.
Creating BI, data or analytics solutions is hard. The technology involved develops at an
incredible pace and the intellectual effort to create sound data models and deal with the
complications of working with large volumes of data is significant.
6© Free Frogs 2018

Part of that complexity stems from the ‘blank stares’ you get back when you request users what
their requirements are. I’m not saying that users of information are to blame, that would be too
easy.
But because we don’t know what we need to design our solutions for, one of two scenario’s
play out in reality:
• We create solutions that are specific to a specific question and over time the information
landscape is a chaos of partly overlapping solutions. Nobody knows what information is
trustworthy or not, because figures don’t align or match up.
• We create a ‘prepared for everything’ solution, that turn out to be very complex and hard to
maintain.
An illustration: can I have hands how many of you have been in a discussion where the
question is “Do we need a Data Vault or not”? And if you are, the discussion is always based
upon data issues and technicalities, never on ‘but what are we actually going to do with the
data’?
Data architecture has become a technique driven discipline and it causes a lot of the problems
we have.
But why are these discussion technical in the first place?
7© Free Frogs 2018

I think that one of the reasons is that we, as a BI or data discipline, haven’t got a clue how
people use information to arrive at a decision. We talk about “making data actionable”, but if I’m
honest we haven’t got a clue how that works. We trust our users do, but I’m pretty sure they
don’t.
In 2015 I read a book called “Business unIntelligence” by Barry Devlin. Turns out, he published
the first data warehouse architecture 30 years ago.
Barry Devlin’s book was a reaffirmation that we don’t focus enough on the way people use
information to make decisions, and support the way people do take decisions, in a business
context.
Decision-making it is a collaborative process. It is very hard to distil requirements from a
collaboration process as input for designing solutions. But once you make the switch in your
head, and start to think from the perspective how people do use information, the impact on how
to structure your information landscape is significant.
One lessons learned is that we need to turn the perspective around and start with the question
‘how do people use information, and what do we need to do to support the way people use
information’. That is quite a turn for most data architects who like to discuss the pros and cons
of data vault versus anchor modelling.
8© Free Frogs 2018

I like to keep the principles of the agile collaboration process, but apply them to how to organise
your BI or analytics effort and not just limit it to the development task. Your BI effort should start
with the question ‘what do we need the information for, how are we going to apply it?’
And the only way to get an answer, is if you collaborate on getting an answer through a
structured way of working, with all disciplines and voices necessary: users, analysts,
developers, IT operations.
9© Free Frogs 2018

To make it stick, you need to organise the responsibility for curating information with the people
who actually create value out of it, by using the information, by applying the insights derived
from it.
It doesn’t mean that users are left to their own devices, but that you drive the collaboration
process with developers from the responsibility of implementing and using information. This
sounds like an open door, but reality in most organizations it is quite different.
The IT wizardry involved in BI and Analytics is still intimidating enough to scare most
managers, and the skills needed to create information that is trustworthy, is and will remain a
hard to find skill. But that doesn’t mean you shouldn’t organise the responsibilities the right way,
because if you do the return on investment is much greater. This afternoon you will hear more
about this. [Note: different organisation presented their BI journey in the afternoon. They all
have organised their BI and analytics effort this way].
10© Free Frogs 2018

Putting human interaction first and organizing responsibility is just one step though, it isn’t
sufficient. How do we align the solutions to what information we need, and how we use that
information?

Remember that I said that often complex solutions, complex data warehouses or data lakes,
are created? Is it because BI people are just too caught up in technology, too nerdy or just blind
to the needs of users?
Not quite. One of the reasons the complex solutions emerge is because of context.

You cannot predict in which context information is needed. The same piece of data can be used
for example in an ad-hoc question, as input for a predictive model or in a regular dashboard. As
a developer or an architect, you try to prepare yourself for all possible contexts in use.
The real life challenge is, that each type of use imposes different requirements to the solutions,
because the way developers, data scientists and users interact and collaborate is different.
So, if we try to create a solution which can meet all possible requirements, you end up with
something complex. Often too complex for the specific use cases that do emerge in reality.
I’m not pointing fingers here, we have gone through these cycles more than once ourselves.
The question is more, how do you break the cycle?
A collaborative process, with ‘just enough’ progression in every iteration helps. ‘Just enough’
means that all involved in the iteration understands what the question is and what needs to be
created.
Turning your perspective, from perceiving data as a technology challenge, to people using
information, is a second step. It defines ‘just enough’ in terms of use, not of technology.
Making sure the people who use information are responsible for organizing the curation of
information, is a great leap forward to drive the process and manage your information
landscape.
But how do you align the solutions to the use cases?
If you develop use case by use case, you end up with an information landscape that is too big
and too fragmented to maintain. If you try to develop the one solution that fits all, you end up
with something that is too complex.
So, how do you connect your information requirements to the solutions? And how many
solutions do you need?

Luckily, the amount of archetypes of contexts in which information is used is quite limited. If you
concentrate on the use of information in decision-making processes, there are just five of them
that I can discern.
Monitoring is keeping a close eye on the execution of business processes. Once they are
outside predefined bounds, you will be alerted and corrective action is taken. The metal
detector at Airport Security is an example of monitoring a process. Another example is crowd
control during festivals.
Accountability reporting is the well-know BI use case. The dashboards that we love (and hate)
are an example. With accountability reporting the goals are set and we use information to see if
we are on target and if we have the means to reach to those goals at the end of the period.
Analysis is getting to a deeper understanding of the execution of business processes and see if
we can improve them. Another classic BI case, but with technological advances the array of
analytical tools we have at our disposal has increased significantly the last seven years.
The interaction between users, analysts and developers is quite different in these three
archetypes. In monitoring, the boundaries, the norms, the action to be taken is predefined and
automated. IT is dominant once users have agreed upon what to monitor and how the
measurements are defined.
Accountability reporting and analysis share the shame interaction pattern, but in accountability
reporting it is the user who has been trained to understand the dashboard and can take action,
once we have had a close collaboration between users, analysts and developers to define and
build a dashboard.
In analysis, the analysts explore, investigate and share their findings with users who discuss
about what it all means in terms of what to do next.

Technology has made prediction feasible. The statistical algorithms and neural networks have
been around for quite some time, but the current state of data availability, connectivity and
cheap processing power has made applying them economically feasible. It is where highly
skilled, very technical data experts (data scientists) are interacting (or should be interacting)
very close with the users. Despite the ‘making AI democratic and use it in a front-end tool’
technology push, you need to be very skilled to derive the right conclusions from predictive
models and techniques to decide what a next action could be.
Data exploration has been around for a long time, but has demanded centre stage when ‘Big
Data’ became fashionable. It is the realm of data experts who validate assumptions or
questions of business people. It is where human creativity, stubbornness, belief systems,
science and con artists play their role and sell their point of view. If there is an archetype of use
that showcases what the role of people in information is really about, it is to be seen in data
exploration. Blessed with wonderful discoveries and cursed with data governance headaches.
Those archetypes have quite different requirements to the provisioning of information. If you
cater, with your solutions, to the specifics of these archetypes, you are well covered.
Even better, the requirements on solutions are quite consistent and fairly simple to describe for
each archetype.

What we have learned is that there are five areas of requirements that determine what kind of
solution is needed. They push the solution into another direction. We call them the
schizophrenia in information, because users always demand that the information is real-time,
totally unambiguous, always available, incredible flexible and with zero time-to-market when
something new needs to be added. You can’t, but we try to build those solutions anyway.
As an example, for data exploration you need to be very flexible in using information, but you
don’t care about unambiguity. Exploring the level of ambiguity in the data is what exploration is
all about. You don’t care about availability that much and a lot of data is a one of, weeks old or
even years old.
For monitoring, the information must be as available as the process it monitors, often of low
latency - real-time - and must be very unambiguous. You don’t want flexibility, everything is set
in stone.
For every archetype, the requirements themselves, the combined set on these five areas, are
consistent enough to create a solution.
We want to be able to share and use data across these solutions, even when the requirements
to the solutions are different. Within the connected architecture framework we connect solutions
also on a technical level, without integrating them. It’s where its gets technical really quickly, so
don’t worry, I won’t try to explain it here. But it has impact on how to design the information
landscape.
We’ve found that with three to six solutions you can cover all use cases. That sounds like a lot,
but they are far simpler in design and easier to maintain because the purpose of each solution
is much clearer. They fit the context for use better. People understand what they can expect
and demand of a solution, which lowers the barrier to adoption. And that is the real importance
here, it fits the comprehension of human beings. Limited complexity, due to limited purpose, is
far easier to curate.

Wrapping it up. Once you turn your perspective and take a more humanistic approach to how
we, as people, as users of information, as analysts, as developers, are interacting in trying to
curate information, interpret it ,and apply the insights derived from it, you start to think of how to
define solutions differently.
Even though we use the same techniques and technologies in our data architecture, we
understand better what its purpose is, what fits a context of use.
That won’t organise itself, you need to actively align information use and solutions all of the
time. Every step forward, every change, every addition to the landscape, should be ‘just
enough’ to be within the comprehension of the people involved and driven by the people who
actually need the information.
Collaborating closely with the people who have the skills to gather, integrate and analyse
information, will keep your information landscape sustainable. More so than what we’ve been
able to in the last 20 years.

The question is still unanswered: ‘what goes on if people do use information to reach a
consensus on what to do next’. Barry Devlin has created a model for this, called ‘Modern
Meaning Model’ (or m3). [Note: Barry Devlin presented M3 afterwards].
For more in-depth background on connected architecture, and what is behind all aspects
touched upon, you can find resources on https://www.preachwhatyoupractice.nl

The road to connected architecture

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a The road to connected architecture

Semelhante a The road to connected architecture (20)

Último

Último (20)

The road to connected architecture