Driven by new business processes and regulation, the need for data exchange between organisations is heavily increasing. As 'Trading Partners', companies exchange data using electronic messages or 'events'. The automation of this B2B communication is very different compared to the use of Enterprise Service Buses within an organisation.
This presentation will show different insights in protocols and message formats which are being used in business, from file transfer and EDI until AS2 with different XML dialects. Also security and integration with 'endpoints' are explained.
3. Abstract Gedreven door nieuwe business processen en door regulering neemt de behoefte voor gegevensuitwisseling tussen organisaties hand over hand toe. Als 'trading partners' wisselen bedrijven massaal gegevens uit onder de vorm van elektronische berichten of 'events'. De automatisering van deze B2B-communicatie kent zeer belangrijke verschillen ten overstaan van de integratie door middel van 'enterprise service buses' binnen een organisatie. Tijdens deze avondconferentie verkrijgt U inzicht in de verschillende protocollen en berichtformaten die terzake worden toegepast, gaande van file transfer en EDI tot AS2 met allerlei XML-dialecten. Ook beveiliging en integratie met 'eindsystemen' wordt uitvoerig behandeld. De markt van B2B-integratie-oplossingen is overigens in volle evolutie. En met de komst van 'cloud computing' zal de wereld van B2B-communicatie nog drastisch wijzigen. Communicatie via de cloud, en met partners in de cloud, vormt een belangrijk toekomstperspectief ! De sprekers geven de nodige achtergrond en leiden U, in één avond, door de boeiende evoluerende wereld van B2B-integratie al dan niet via cloud computing.
4. Integration consultancy ESB, SOA, BPMS, B2B, EAI, Composite Apps Vendor independent 33 consultants Part of the Cronos group http://www.i8c.be Who are we?
5. The “waves” 1st: EDI and closed networks 2nd: From 2000 onwards: Internet & XML 3rd: 2010: Cloud & Integration-As-A-Service Picture by Mila Zinkova
11. The 1st wave Organization B Organization A Value Added Network LOC+147+0090305::5’ MEA+WT++KGM:22500’ LOC+9+NLRTM’ LOC+11+SGSIN’ RFF+BM+933’ EQD+CN+ABCU2334536’ NAD+CA+ABC:172:20’
12. Value Added Network Routing (but nor format translation) Each party has “mailbox” Different transmission methods Phone line, ISDN, Bisync, X.25, FTP, OFTP, X.400 mailbox, FTP, WebSphereMQ (IBM), .. Trusted Third Party, e.g. audit information Interconnect to other VANs No translation or transformations
19. VAN setup Back-end Application Back-end Application EAI File (CSV) Appl. adapter Translator Transform VAN #2 X400 EDI FTP Adapter VAN VAN connector FTP Mail Dial up VPN
30. 2nd Wave – Internet & XML Decline of the VANs Direct communication But also upcome of the TDN’s, ASP’s, ... XML as an alternative message format Standardization of XML message formats New protocols (on top of TCP/IP) HTTP: Web Services, EDIINT AS2, POX, RosettaNet RNIF, ebXML ebMS, ... File transfer: FTP, SFTP, FTPS Mail: SMTP, AS1
33. Too many B2B XML standards “Long tail of B2B standards”... EDI handles diversity with EDI Message Implementation Guidelines
34. “New” B2B Protocols Internet & TCP/IP de facto standard Decline of X.25, X.400, Bisync, ISDN, ... HTTP XML over HTTP (Plain Old XML) ebMS – ebXML Messaging, RosettaNet RNIF AS2 File Transfer: FTP, SFTP, FTPS, OFTP, OFTP2, AS3 Mail: SMTP, AS1 Very limited use of Web Services and SOAP!
35. EDIINT EDI over the INTernet Initiative within IETF Apply existing IETF Standards Adds retry, security (S/MIME), acknowledgements Any payload: EDI, XML, binary Applicability Statement EDIINT/AS1: over email EDIINT/AS2: over HTTP EDIINT/AS3: over FTP RFC 1123 Requirements for Internet Hosts RFC 1767 MIME Encapsulation of EDI Objects RFC 1847 Security Multiparts for MIME RFC 2045 MIME Format of Internet Message Bodies RFC 2046 MIME Media Types RFC 2049 MIME Conformance Criteria and Examples RFC 2298 An Extensible Message Format for Message Disposition Notifications RFC 2616 Hypertext Transfer Protocol -- HTTP/1.1 RFC 2821 Simple Mail Transfer Protocol (SMTP) RFC 2822 Standard for the Format of Internet Text Messages RFC 3370 Cryptographic Message Syntax (CMS) Algorithms RFC 3798 Message Disposition Notification RFC 3851 S/MIME Version 3.1 Message Specification RFC 3852 Cryptographic Message Syntax
41. Market consolidation Acquisitions Sterling Commerce, ViaCore IBM ADX & SoftShare Liaison Mergers Inovis/GXS merger Partnerships IBM partners with Hubspan Microsoft partners with GXS Oracle partners with E2Open and SPS Commerce SAP Partners with Crossgate
43. I nfrastructure Integration What is cloud computing? AAS As-A-Service Abstraction of the hardware Don’t care where those servers are
44. What is cloud computing? P latform AAS As-A-Service Abstraction of the software Don’t care: about OS, #servers, … Build or deploy my application on provided platform
45. What is cloud computing? S oftware AAS Storage As-A-Service SAAS: Abstraction of everything Just want to use the application
46. The Cloud stack Desktop Software Platform Infrastructure Storage
47. Some SAAS providers Billing: CRM: Collaboration: ERP: HR: Product & Price configuration: Business Intelligence:
48. 3rd wave: the Cloud Connecting to the cloud Connecting through the cloud Integration
49. Salesforce to elsewhere Outbound messaging Async Secure HTTPS Supports X509 Client Certificates Reliable, 24h retry Only notifications Info about objects that have changed api Earlier API version: only polling Enterprise customers: Salesforce connectors for SAP and Oracle
51. Google App Engine to on-premise Enterprise Google Google AppEngine Application Public Resource on Intranet Linux “Reverse Invoke” Secure Data Connector “SDC Server” Fetch Private Resource on Intranet SignedFetch SSL Username/password Resource Rules
52.
53. Cloud as DMZ BizTalk Server 2010 AppFabric Connect for Services WSO2 Cloud Services Gateway
57. 3rd wave: the Cloud On premise Cloud Company A Company B
58. Babelway Belgian company, founded in 2007 http://www.babelway.com B2B Integration SaaS [multi-tenant] GOAL:simplify B2B data exchange No system installation and maintenance No B2B software installation and maintenance Easy operations, easy conf. of connections 1 to many.
59. Babelway – Solution Overview 3 Main Functions Transport of electronic messages between 2 partners Transformation of messages from an input to an output format (Opt.) Storage of messages for a defined period of time. 2 Technical Parts Message Processing Engine Web GUI
60. Babelway – Configuration - Channel Channel = automatic data flow process between Your system Ext. System of B2B Partner Incoming messages can be routed to a particular channel in function of content or metadata
72. Babelway – System Architecture Amazon AWS EC2 and S3 Services (I-a-a-S)
73. Boomi – http://www.boomi.com Founded in 2000, acquired in 2010 by Dell "Integration is a key element in Dell's future transformation into software and services," says Ray Wang, principal Analyst and chief executive officer at Constellation Research. "Companies like Dell, HP, and Cisco will need to be in software in order to grow." Connects any combination of cloud, SaaS and on-premise applications with no appliances, no software, and no coding Software-as-a-Service platform No hw/sw to install or maintain, auto. upgrades Accessible via any internet browser Multi-tenant architecture
78. Main component in Boomi integration Represents an interface between 2+ systems on business proc. level Composed of shapes connected like a flow chart Executions can be event-based or schedule driven (complex) scheduler API to incl. event-driven integr. into your process Boomi AtomSphere – Process
79. End-points of a process Get data in or send data out Enable communication with applications and data sources On-premise applications - e.g. SAP Cloud/Web-based applications – e.g. Salesforce.com Data repositories – e.g. FTP, DB or an email server Accomplish integration by abstracting the technical details of APIs and transportation protocols. Boomi AtomSphere - Connectors
80. Boomi – The Boomi Atom (1) ... Lightweight dynamic runtime engine Can be deployed On a local machine in your network (on-premise) access to internal resources or applications (behind the company’s firewall) solves On-Premise-to-SaaS integration problem In the BoomiAtomSpherecloud connect with resources or applications accessible via the internet (e.g. web applications or FTP sites) all computing is performed in the Boomi data center > No software or hardware needs to be installed. for SaaS-to-SaaS integration
81. Contains all the components required to execute an integr. proc. from end to end (incl. connectors, transformation rules, …) Monitoring - Status and health of “all atoms” and integration processes Dashboard Provides secure access to on-premise application(s) when deployed behind firewall Boomi - ... The Boomi Atom (2) HTTPS -128 bit encr.
82. Build: ... Visual Integration Technology (point-and-click, drag-and-drop) Library of pre-built connectors and process maps Boomi – Integrate in 3 steps – 1. Build
83. 2. andDeploy integration process Boomi – Integrate in 3 steps – 2. Deploy Atom Name Atom
84. 3. Manage all integrations from the AtomSphere platform. Boomi – Integrate in 3 steps – 3. Manage Atom Name Atom Name
85. Version control offers rollback of integration processes Concept of environments is provided Molecules - (Ent. Ed.) Atom version which provides High-Availability - through Fault-Tolerance High Performance - through intelligent Load-Balancing Boomi – Extra Features
88. SL Server Execution engine (security, logging, runtime, ...) for complex data transfer and transformation tasks pipelines are published Can be installed Local Linux, Windows, Mac OS X In the cloud Rackspace, Amazon Web Service, Private cloud SnapLogic – Server
89. SL Designer Browser-based visual configuration tool (allowing drag-and-drop functionality) Create and configure workflow pipelines connecting any system (no need for custom code) SnapLogic - Designer
90. Orchestration of Snaps that solve an integration path from end to end Snaps are connected together in the Designer Can be scheduled, triggered on-event or on-demand Addressable by a simple REST URL SnapLogic - Pipeline
91. SL SnapStore - http://store.snaplogic.com/ Online store for snaps ( sw delivery model) Snap Certified connector plug-in for the SnapLogic Platform Interfaces with applications to easily perform data operations without any custom code Created by developers ( > SnapLogic API (SnAPI) and SDK) for private use to place for sale and download from the SnapStore Cost-effective alternative to custom hand-coding SnapLogic – SnapStore AND/ OR
92. SL SnapCenter Enterprise level clustering solution Allows easy conf. for 1+ SL servers to scale up parallel pipeline processing Users build, manage and execute pipelines as on a single SL server Cluster manages execution of pipeline behind the scenes in distributed env. Higher throughput, higher # of integration tasks Console’s tabs: overview, pipeline & server, events and history SnapLogic – Architecture - SnapCenter
105. - acquired by IBM WebSphere Cast Iron Cloud Integration Pre-configured integration solutions for hundreds of leading enterprise and cloud applications “Configuration, not coding" approach Cast Iron
106. Cast Iron: does 4 things Connectivity - Connect to ent. apps, on-demand apps, databases, flat-files etc. Transformation - graphically map data between source and target applications Business Logic (Workflow) - graphically define the flow of data between source and target applications Management-one console manages your integration, and have complete visibility to data flowing thru the appliance
107. 2 deployment models: in Cast Iron Cloud I-a-a-S on Cast Iron integr. appliances Integr.On-Premise WebSphere DataPower Cast Iron Appliance XH40: physical appliance to add to on-premise datacenter WebSphere Cast Iron Hypervisor Edition: virtual appliance that can be installed on own hardware Cast Iron – Product Solutions
117. Hubspan Example of Intergration Service Provider that pretends to be Integration As A Service Why not IAAS? No “self service”
118.
119.
120. Microsoft IAAS announced Also this week, we disclosed an early peek into our strategy of “Integration as a Service” which begins to shed light on how we will be taking the integration workload to the cloud. This is a transition we have already made with Windows Server and SQL Server (as we have released Azure flavors of these server products); and we are committed to following this same path with integration. Our plans to deliver a true Integration service – a multi-tenant, highly scalable cloud service built on AppFabric and running on Windows Azure – will be an important and game changing step for BizTalk Server, giving customers a way to consume integration easily without having to deploy extensive infrastructure and systems integration.
122. Summary Integration As AService Very promising Interesting new players + market turmoil Mix of in the cloud & on-premise Software and Hardware appliances Self-service (but someone’s got to do it) 1st wave: EDI & VAN Old technologies aren’t gone yet, e.g. EDI 2nd wave: XML over Internet AS2, SFTP, … With or without Integration Service Provider Many XML languages, too many XML dialects 3rd wave: Integration As AService
In the most basic form, a VAN (value-added network) acts as a regional post office. They receive transactions, examine the 'from' and the 'to' information, and route the transaction to the final recipient. VANs provide a number of additional services, e.g. retransmitting documents, providing third party audit information, acting as a gateway for different transmission methods, and handling telecommunications support.
TODO: animation(too fully understand, requires understanding of public/private key)Message signingUsually not HTTP Basic AuthenticationNon-repudiation through signed acknowledgements (MDN)Network security: discussed furtherMessage level encryption vs. HTTP/SUsually same keypair/certificate for signing/encryption
Interesting & relevant blog: http://blogs.gartner.com/benoit_lheureux/
Flexiscale is British/EuropeRackSpace doesn’t have a real offering yetEasy and quick to provision
Coghead runs on Amazon EC2; was acquired by SAPOverzicht van vele cloud offerings: http://peterlaird.blogspot.com/
Solve360=Norada37signals: http://highrisehq.com/
Contract is defined by of Web Service Endpoint is generated by Salesforce and needs to be implemented; Salesforce outbound messaging does not allow to call web services with contract defined elsewhere
Google - Secure Data ConnectorWorkday – acquired CapeClearSalesForce / Force.com
1 to many: e.g. Connected to X.400 => no seperate VAN supscription necessary
REST API: Tickets: search and listMessages: search, list and view detailsChannels: search and listCatalogue: search and listPush a message
Flow chart: to illustrate the steps through which data is transformed, routed and manupulated from source to destination.
Als metafoor: een connector is als een plug en een API als een socket
2: atoms can be deployed in the cloud to Boomi’s data center or a 3th party data center such as e.g. Amazon ???? How does this work?
Atoms track state to ensure that only unique data is processed (dupl. Detection)
Library: built by a community, the AtomSphere ecosystem expandsVisual Integration Technology (point-and-click, drag-and-drop)Library of pre-built connectors and process maps
Monitor health and activity of all AtomsReview detailed logs of processesSubscribe to alerts that broadcast via RSS for proactive notification of failuresVia Managament Tab users can follow up integration process(es) executionsAssociated log and status notificationsRetry capabilities for messages which had an error during transit
High-availability also ‘self-healing capabilities’Concept of environments: to have the same integration process pointed to different locations (e.g. Test, QA, Production)
TODO: put graphical example, eg For example, a simple pipeline could connect to a source system such as SugarCRM using a SugarCRM Snap, retrieve select data, transform it using a series of functional Snaps, and then write it to a target DB using a DBWrite Snap.
SL SnapCenter: central command center for enterprises to build, deploy, manage and visualize the complex data flow through their integrated applications
All end points are HTTP compatible data services, providing a REST interface to SaaS applications from any web technology.
DataIntegration: More about synchronization, replication Consider everything as data sources and pull data (as opposed to push based events)Informatica:- PowerCenter Cloud Edition: PowerCenter on Amazon EC2- Pervasive: still at v9.2, v10 will bring cloud solutionStill on-premise, not in the cloud?- KapowSoftware: “screen scraping” approach to cloud integration (http://www.kapowsoftware.com/)- Scribe: focus on integration with Microsoft Dynamics CRM (http://www.scribesoft.com)
Jitterbit: open source project, also version as-a-service available, deployed on AmazonAppirio Cloudworks; very vague