BizTalk Architects have always taken themselves a bit too seriously and have been able to dictate rules for integration based on things that is important to no one but them.
Only the BizTalk architect really care about port names and making the BizTalk environment look pretty. We are at a time in history were “integration is hard” is becoming less and less of an excuse for such behavior. Both technology and people have become smarter and integration has shifted to a more function based focus. Make it work is more important than make is pretty and reusable.
I will be presenting a solution to all these problems.
Strategies for Landing an Oracle DBA Job as a Fresher
The fall of the BizTalk Architect – From something abstract to something useful
1. enfo.fi / enfo.se
The fall of the BizTalk Architect
From something abstract to something useful
May 25th 2015 Mikael Sand
2. enfo.fi / enfo.seenfo.fi / enfo.se
Who am I?
ThefalloftheBizTalkArchitect
• Mikael Sand
• Live just south of Stockholm, Sweden
• 40(!)
• Senior integration architect and
Azure Associate Enfo Zystems
@mikaelsandmikaelsand.se
5. enfo.fi / enfo.seenfo.fi / enfo.se
What is important in an integration?
ThefalloftheBizTalkArchitect
6. enfo.fi / enfo.seenfo.fi / enfo.se
What is important in an integration?
ThefalloftheBizTalkArchitect
• It looks good
• It solves a problem
• It connects two or more systems
• It works
• It is reusable
8. enfo.fi / enfo.seenfo.fi / enfo.se
That gives us something
to think about
ThefalloftheBizTalkArchitect
9. enfo.fi / enfo.seenfo.fi / enfo.se
Perspectives
ThefalloftheBizTalkArchitect
Our usual world view
BizTalk
Salaries
Invoicing
That system
This system
www
Intranet
Shipping
partner
Other
partner
BizTalk
16. enfo.fi / enfo.seenfo.fi / enfo.se
Are we even the center
of integration?
ThefalloftheBizTalkArchitect
17. enfo.fi / enfo.seenfo.fi / enfo.se
Who are involved?
ThefalloftheBizTalkArchitect
PMCustomer
rep
Operations
owner
Enterprice
architect
System owner
Application
operations
System owner
More PMs
Technical
operations
The BizTalk
Architect
Developer
Developer End user
Tester
Security
20. enfo.fi / enfo.seenfo.fi / enfo.se
Perhaps we are not the center of
everything
ThefalloftheBizTalkArchitect
21. enfo.fi / enfo.seenfo.fi / enfo.se
ThefalloftheBizTalkArchitect
wwwIntranet
CRM Sales
Customer service
application
Business critical
Application
Printing Business data
22. enfo.fi / enfo.seenfo.fi / enfo.se
ThefalloftheBizTalkArchitect
www
Intranet CRM
Sales
Customer service
application
Business critical
Application
Printing
Business data
33. enfo.fi / enfo.seenfo.fi / enfo.se
Structure, architecture and
the right mindset
ThefalloftheBizTalkArchitect
34. enfo.fi / enfo.seenfo.fi / enfo.se
What do you get?
ThefalloftheBizTalkArchitect
• Quick answers to
complex problems
• Processes ->
• Predictability
• Measurability ->
• Feedback
• Again: Quick answers
• Processes ->
• Predictability
• Just a little more
concrete
Predictability Business centric
35. enfo.fi / enfo.seenfo.fi / enfo.se
Time for a definition
ThefalloftheBizTalkArchitect
What is an integration?
36. enfo.fi / enfo.seenfo.fi / enfo.se
Presentationname
Roughly the same information is moved from
roughly the same system to roughly another system
37. enfo.fi / enfo.seenfo.fi / enfo.se
ThefalloftheBizTalkArchitect
Information type: Beverages
System
A
System
B
Beers
38. enfo.fi / enfo.seenfo.fi / enfo.se
ThefalloftheBizTalkArchitect
Information type: Beverages
System
A
System
B
Beers
Wine
39. enfo.fi / enfo.seenfo.fi / enfo.se
ThefalloftheBizTalkArchitect
Information type: Beverages
System
A
System
B
Beers
Wine
System
C
Beers
40. enfo.fi / enfo.seenfo.fi / enfo.se
ThefalloftheBizTalkArchitect
Information type: Beverages
System
A
System
B
Beers
Wine
System
C
Beers
System
D
Empty bottles
44. enfo.fi / enfo.seenfo.fi / enfo.se
What is the atomic integration?
ThefalloftheBizTalkArchitect
• The integration has to contain all it’s parts and have as
few external references as possible.
• The integration will contain everything to be buildable
and deployable.
• But: ”Don’t be stupid!”
• Duplicate the artifacts and code reusability with copy-
paste
• But: ”Don’t be stupid”
45. enfo.fi / enfo.seenfo.fi / enfo.se
The atomic integration gives you
ThefalloftheBizTalkArchitect
• 1 integration
• 1 Visual Studio Solution
• 1 Application in BizTalk
• 1 msi for deployment
• or PS-script, or whatever.
• 1 set of documents
• 1 common name for the integration (business-IT)
• 1 entry in a repository
• A very simple solution to a very complex problem
46. enfo.fi / enfo.seenfo.fi / enfo.se
The atomic integration gives you
ThefalloftheBizTalkArchitect
• Easier follow-up
• A better fit with Sprints (the Agile method)
• A better fit with ITIL since you get a 1 to 1 relationship
• Easier add-on development
• Easier development over all
47. enfo.fi / enfo.seenfo.fi / enfo.se
The downsides
ThefalloftheBizTalkArchitect
• May cause large solutions, hard to cooperate within a
team.
• Double the info! Then again, gives a nice feeling of
control and isolation.
• ESB: Not a perfect fit, might become harder to do.
Then again, was ESB ever easy?
• A bit boring
48. enfo.fi / enfo.seenfo.fi / enfo.se
The structure in BizTalk
ThefalloftheBizTalkArchitect
• Integration name: Number + friendly name.
• Artefacts: Separated by type name using subfolders
within a project or
• One per type, per system.
• Orchestrations: Use hard bound ports as standard
and make them shared.
• Copy-paste for pipeline components and helper code.
• But: ”Don’t be stupid”.
58. enfo.fi / enfo.seenfo.fi / enfo.se
Mikaelism #6
Presentationname
Make sure the right error information is sent
to the right people in a way the person can
understand
Short preface:
I am, and have always being worlking as a consultant. Therefore I will use the term, client or customer. If you don’t have a client or customer just exchange these words for Clive or Susan or whoever orders new integration stuff from your.
I will also make an effort to present everything in black and white and simply say ”everyone” or ”all” where there is some room for discussion. I might also sound very cynical. I do not do this becaus I don’t know, but for dramatic effect and it is simply easier. I say this now once and for all so you can fill in this diclamer everytime you hear something that might need a bit more nuance. That way we all save time.
That might be a bit too overwhelming to answer. Rather >klick<
I would like to ask you but I have decided to not look at the chat window.
These are five things that is important.
Out of these five? Which one is the most important.
It is doing what it is supposed to be doing.
The important thing is not that it is cool looking or a nice technical solution.
This is our world. Systems are sending their data. They are depending on us.
I want you to look at something.
We always place ourselves in the center. HEY! It i seven called middleware.
We se ourselves as the center.
As a matter of fact, we might even see ourselves as…
This might even not be far for the truth.
Hoooooooweveeeer.
>klick<
Talk about Mikaelisms
If you are not working with biztalk to print money, you are an expense.
You are not making money for a client or an employer. Your reason for being there is to lower expenses. >klick<
If you havn’t realized this and also have that BIG BizTalk ego >klick<
This is an example of a small business meeting. The sums might not be internationally viable. If so just view them as numbers.
This is a real and not at all unusual business meeting I attended.
So perhaps you might not need to motivate your very existence.
But you are HAPPY if you have people abouve and around you that know exactly how important you are.
You might even have trouble getting love for other colleagues within IT.
So #2
This might seem very egomaniacal but think about it. You have special skills, or powers. You know this. You can do this because you did it before.
However I have often met other “integrators” that said they know this but when asked EXACTLY what is sent over the wire, because the soap header seems to off somehow, the respond. “we have no idea because we use a tool to generate java classes…” So you might be better but you need to >klick< Help them understand.
Noone has ever been motivated be being steped on. HELP not TELL people.
A technical solution might be clear and obvious to you but that in itself might not be a good reason for the person holding the money.
So even people within our business and even the same field have a hard time understanding us >klick<
You might even ask >klick<
I would like to pose this question. Are we really. Let’s see shall we?
So you have your basic receive file, transform it and send it somewhere else.
Who are involved?
Well the BizTalk architect ofcourse.
Everyone of these people have their own demands, wishes, wants and everyday worklife and when the integration is DONE it is THEIR reality that dictates what is ok and what is not.
EVERYONE of these people have ONE thing in common. ONE wish. ONE demand. >klick<
There are some exceptions of course that I might just point out in order to avoid a riot.
Security people does not want it to work. They want it to be secure. Not the same thing always.
The EA wants the integration platform to be a box so that it fits the EA diagrams but other then that people want it to work.
The user of you r integration could not care less if you used a deplyment framework and the project manager is utterly unintrested in that is a ”very cool ESB solution”.
So just to hammer it home.
Perhaps we might start to realize that we are not the center of the known universe.
We cost money and we are only a small part of an integration project.
How does a company or organization view the integration platform?
Here is an almost real customer systems landscape map. They have an integration platform.
The question is, can you find it?
If you are lucky, you might even be given an integration map.
This one is not far from the actual customer truth.
Missing from this is that they had actually put is the different kinds of that that was transported.
STILL there is no integration platform.
Let’s make up some statistics.
In about 90% of all cases, the client, user, the person who ordered the job, could not care less about what kind of framework you are using, naming conventions or anything like that. Ok, you might be allowed to give them one hour lesson on loosely coupled integrations but that is it! Then when you need to do something that cost money; motivate it,
The users really are the number 1 representatives for this. They are almost always interested in one single thing: That it works.
Only after it works, then you can start to focus on add-ons like BAM, monitoring and automatic error handling.
If the files uses one or two integrations, who cares?!
We are very funky and dynamic. Adaptive even compared to other platforms.
We can do a lot of specialized and funny stuff and use a lot of protocols.
This brings that we are often used as the ones going “you decide” when a customer want to connect systems and what to know how. That is usually just frustrating.
Remember the Mikaelism “Do not think that other people are as good at this as you are”
If a company wants to make a new integration >klick<
Webb is sending something to the backend system and the backend system is sending to the printing firm, this is the way the draw it.
Solid, well defined blocks and also well defined processes on how to do different things within the system. Not even we view this as something strange.
Well they realize that they need BizTalk to do the integration stuff. >klick< So they draw another box.
But if we act like >klick< we are not exactly going to be the flavor of the month.
We are perceived like this every time we do not have answers to difficult questions, quickly.
Connectors. We here might think about these as Api Apps but they are connectors and they are not a new thing.
Customers might be looking for a service to be able to “connect to some governmental agency” or “large corporation”. We (the BizTalk developers and architects) see this as quite an alien concept as we are so close to the wire, or the protocol. Were other platforms might say “yeah, we have a connector for that” we have to go “how do you want to connect”.
So another question.
That’s right. The preparations. The stage well before any development take place. We are quite good at developing integrations by now. The time from spec to tested code is short. The big part is NOT development and configuration but rather the preparations.
If we do not facilitate preparations but rather go “well you have to decide” we are problem bringers, rather then problem solvers.
Always present a solution together with a problem. Bosses LOVE that.
So, we are expenses. We are rather small in the big scheme of things. We are invisible and many view us as an unnecessary step in an integration. We flail about and we speak a language others do not understand.
What is the LEAST we can do?
YES! Say it with me now!!!>klick<
This is the turning point. After tearing you down I will now rebuild you.
Its here and now I will present the best BizTalk architecture you can get. The one that covers the most ground with the least effort.
It might not be a perfect fit, especially if you already have a framework in place, but then again you have not agreed with the problems I have presented so far. But in my view, this is the best possible solution if you have a bit of chaos or looking for your way of doing things.
This is the solution! It is not magic or the latest and greatest tech from your fav company.
It is as simple as that.
A predefined way of working and an, open, more customer centric way of thinking about a problem. Be part of the business, not the odd one out.
So what do you get?
Quick answers. Integration is complex but by using structure you can have predefined answers to questions that might be very complex within an enterprise, like ”what kind of way can we connect to an external partner?”
Structure means processes and processes gives you predictability. The answer to ”why do we have processes” is not ”because it tells people what to do”. The answer is that it gives you predictability. If I send an email to my IT-support, they will pick it up, create a ticket and come back to me.
It also gives you the ability to ”measure” your workload. How long do we think this will take and how long did it take.
On the business side
Again quick answers,
Processes that leads to predictability. You define the processes and make your clients jump thru hoops.
Lastly: We are very abstract in what we do. Not a lot of people understand why we do the thinks we do and what they get in the end. ”A way of working” just make it a little more concrete.
You might ask yourself: This is nothing new. No one disagrees with this, well then again why do so few work like this? And also I am just getting started.
There are more definitions of this then the world can handle, but I think this is the best one.
Before we move to these slides I thought I’d explain.
We always use the same kind of ”boring” financial data or Purchase order and it might be a bit hard to understand some points if you are new to BizTalk or accounting.
So I thought I would use something that everyone can relate to Alcoholic beverages!
So the information type is beverages.
This is definitely an integration. System A is sending Beers data to System B
Once again not the lack of a BizTalk box.
One more thing: The arrow “beers” is not an integration. It is a “flow” or a “flow of information”. The whole thing is an integration. The arrow is a flow.
This is the same integration. Same systems just another type of data.
This is still the same integration as we are still sending the same data to roughly the same system
This is where it starts to get tricky. It may depend of how the client ordered the integration and how they see the future of System D.
Use your head: Is it a good idea to keep these flows connected in one integration? How is the last flow related to the information type? Note that once again: The client does not care if it is two integrations or one.
Why do I define an integration like this?
Because that is how people see it! The people that ordered the integration and are paying for it.
An integration is not system A0002 is connecting to service SRV000023 using contract C00002.
If you start saying that people will fall asleep or get mad at you for not providing the solution but rather giving them a new problem: Trying to understand what the hell they ordered.
Try instead to have some humble pie, relax and ask them:
What do you need? We want to send invoices to clients. Deliver the batch file to my CRM system.
An integration and what people order are not the same but they tend to be just that. If that is how they see it, then perhaps you can see it the same way.
This is perhaps the place for a BizTalk architect: how do we split or merge what the client ordered into integrations and how does this new stuff fit with the old?
This is the big reveal. This is when I tell you how to solve 90% of all your BizTalk architectural problems.
And I am not even close to kidding.
Yes! The atomic integration.
Atomic in this case does not mean “indivisible” and the atom is splittable as we know.
Here it means contained or isolated. More like the atomic in the BizTalk transaction scope sense.
1. This also means that you clean up after yourself. This means duplication of schemas and other artefacts if needed yes.
2. This means you get to wave bye-bye to “shared” and large dll-libraries: However >klick<
3. Do not take this and run into the wall. (Large schemas that will never change and will be used throughout the integration platform like MS CRM, then place it in shared. Because not reusing the same schema over and over would be stupid. So don’t be stupid.
4. Simply talk about code reuse with copy paste and duplication
5. Once again: Don’t be stupid. This might be the most important thing. Be an adult about things. ALL architectural ideas can be abused as well as used. Just be an adult.
4. You can use a simple MSI. You do now need to develop a heap load of custom scripting do make your deploy work.
8. Lastly. I have seen a lot of solutions to the problems around BizTalk architecture and some of them are really good and useful they do however always contain some kind of technical caveat and codebase that needs to be maintained in order for it to work. This solution does not need that. It is ready to go out of the box.
A well contained integration is also a well contained definition and work load. So it is easier to follow up on.
The well contained and well defined integration is also a better fit for the agile sprint method.
A VERY good fit for the ITIL method of Incident, Problem, task and so on as it gives you a 1 to 1 relationship between your integration and ITIL artefacts.
If you have well defined integrations, independent of each other you can develop them as such as well. A single developer can take responsibility for a whole integrations, which gives you better quality code.
There are of course downsides to this. Nothing is perfect, not even I.
A large integration with a lot of different systems may cause large solutions and that makes it hard for different developers to cooperate as they all have to work within the same solution.
Double the info, yes this infers a bit of overhead as for example all the ports using the xml disassembler need to point to the assembly to resolve schemas.
If you are thing ESB if might not be a perfect fit a. You could set up your integrations as service points and endpoints but it is harder to do. Then again, was ESB ever easy to do?
It is a bit boring. Framed, well defined solution basically already done when you get the dev spec.
The name of the integration is very important as this is the common name for if thru every meeting with clients and users.
Artefacts might be separated using folders and the integration is essentially one project file. The different folders create different .net types and makes it easy to sort the names in lists. Personally not a total fan of this method as it creates monolithic dll:s (assemblies) that contains all the information. But HEY It Works and you know what I say about that.Another way is to have separate projects per artefact and system. So SystemA.Schemas, SystemB.Schemas, SystamB.Pipelines and so on.This is a bit more granular then the approach above. And make is possible to deploy part of an application should you need to patch a transformation for instance.
It is important to point out that Shared in this instance means shared within the integration. Orch are special as they usually don’t “belong to a certain system”. They belong to BizTalk and should be named after what they do.
DO NOT share component or helper code. I am not kidding. Use copy paste instead. Look at your component library as you do code you just binged… found on the internet. You copy that and make it your own. Then again. You might find cases were this is stupid, then don’t be stupid.
Some stupidities:
Using components to push code to all integrations using that particular component bacuase they wasn’t tested enough.
Duplicating large schema sets where you only use a subset of them.
Non expandable or versioned standard messages (one message to rule them all).
One pipeline to rule them all. Either a send or receive pipeline that was configured to on the fly load components EVERY time a message passed thru.
Gigantic solutions, I have been a victim and a creator of this. What should have been 6 integrations was put in one solution.
Special settings and configurations. I am a fan of using standard settings as much as possible. Don’t get me started on XSLT.
Time to get concrete.
This is what needs to be done.
This is what we got. I know it is in Swedish but I will translate.
This was the result of a meeting between two persons in a large company. They met over lunch as want’s this done.
You can see that they want to transfer Orders to the business data system and then they will send something to Printing.
Depending on your organization this might even be the actual order. Make this happen. I call this the “Integration initiative” or “napkin drawing”.
After some research we arrive at this visio drawing.
The website should send orders for new cards to the backend system. The backend system will send a batch file to printing once a day to order new cards.
We view this as one integration, that it is new and give it the name >klick< INT0001.CardTransactions
We see a future were the web site wants to send other types of card transactions more than NEW and give it a bit ”looser” name then ”OrderNewCards”.
After some more thinking we arrive at this.
Walk thru the solution.
Also note that this is about the time were we loose the interest of the client. This is just for us.
Who are the stakeholders
The website, everything web at this company is, as always, handled by young people who were Trillbys and scarfs inside in the middle of summer.
The salesforce, Thinks the cards will sell themselves and automatically. He seems to have missed the fact that he has not turned on his computer.
Backend: The business critical system. Unix based. Solid and probably some oracle databases in there for good measure.
Printing. In order to lower their costs they have strict processes and formats. One fail means a whole batch fails.
And somewhere there is also an IT-pro, the guys and gals that will take the blame for your mistakes for years to come. Note the ever present fleece.
Lets take a look at this.
This is important. Just make sure of it. Don’t think: How will I make this work, think what happens when it fails?
Moving on up
As you all know this is the new thing and it will change the world. Again.
Using API apps and Logical Apps we can connect all these capabilities without ever having to think about how.
So let us create a new app using >klick< >klick<
Now imagine how you keep all these services and API calls together. Yes! That is correct! You can use the Atomic model here to. Since you are basically doing the same thing here as you are in BizTalk. You need to keep things in order because as you all know. Democratizing integration as Microsoft puts it makes the next developer of integrations >klick< This guy!
This guy!
We are not the masters of the universe.
We are small compared to the rest of the enterprise.
We are a service and should function as such.
We have customers and they should be treated as such.
We are perceived as too complex and granular.
We have too many wants and needs.
There is a way out of this.