This document discusses various Enterprise Application Integration centric designs or approaches which may be considered as better practices, and have been implemented in the data acquisition platform at Nielsen. Targeted audience is tech leaders and architects working across the platform. There is a special focus on leveraging open source technologies where possible. Hoping that some of the material here can help fellow architects.

1. EAI Design Patterns/Best
Practices
- ​-Ajit Bhingarkar, ​October 30, 2020
Introduction 3
Design Patterns 3
Keeping 2 different data stores in sync 3
Asynchronous handling of the REST API 4
Zuul as an API Gateway and new ESB paradigm: 6
Synchronous Request-Reply Pattern using Kafka and Spring: 6
Kafka for http:// Based Requests: 7
Distributed Cache Pattern: 8
Mechanisms of Real-time Server Updates to the web browser to mimic
synchronicity for web operations: 9
Long-Polling vs WebSockets vs Server-Sent Events - Client Pull vs Server
Push 9
Ajax Polling (Client Pull) 9
HTTP Long-Polling (Client Pull) 10
WebSockets (Server Push) 11
Server-Sent Events (SSEs) (Server Push) 12
OAuth 2.0 for Data Acquisition Platform: Conceptual flow 13

2. Spring and Kafka to replace ESB for microservices: (Pls refer to #3) 14
API Design Best Practices: 14
Recommendations or factors to consider when using 3rd party API: 15
References 15

3. I​NTRODUCTION
This document discusses various Enterprise Application Integration centric designs or
approaches which may be considered as better practices, and have been implemented in the
data acquisition platform at Nielsen. Targeted audience is tech leaders and architects working
across the platform. There is a special focus on leveraging open source technologies where
possible. Hoping that some of the material here can help fellow architects.
D​ESIGN​ P​ATTERNS
1. Keeping 2 different data stores in sync
We often have requirements where we need to keep a target DB in sync with source,
particularly when we are moving from Oracle to PG and Mongo. This applies to reports
as well and in cases where APIs are still work in progress.
Using a DB link is not a preferred way, and views can become stale easily.
Pls note that fetching data through API is the preferred mechanism for inter module
communication.
This is a tough problem to solve using conventional methods. Often DB to DB data
refreshes happen as a batch job, but there is always the risk that the target data store
may not be in 100% sync with the source data store.
This causes revocation of further transactions, and redoing those again. This is even
more critical in the case of fast changing master data.
Daily, periodical refreshes or DB views can’t effectively address this issue.
The pattern below is a good solution which works on CDC (change data capture) and
then streams the updates to the target system via Kafka. It satisfies the requirement of
one-way sync of the data, and claims to have very low latency < 1 second.
Here is more on the features supported.
https://debezium.io/documentation/reference/1.1/features.html
It supports MongoDB, PG, Oracle as source systems, and MongoDB, JDBC, ElasticSearch
as sinks. Traffic is routed through Apache Kafka. All the connectors we need currently
are open source.

4. For a specific set of use cases, say sourcing from Oracle/PG, we can create a shared
solution so that every module is not required to build their own.
Pls refer to links pertaining to various sources and sinks.
Debezium connectors mapping to Kafka source.
https://debezium.io/documentation/reference/1.1/connectors/index.html
Kafka sink for target systems:
https://docs.mongodb.com/kafka-connector/master/
https://www.confluent.io/hub/mongodb/kafka-connect-mongodb
https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc
https://www.confluent.io/hub/confluentinc/kafka-connect-elasticsearch
https://www.confluent.io/hub/jcustenborder/kafka-connect-redis
2. Asynchronous handling of the REST API
This is another use case which we need to consider for all tasks requiring heavy
background processing or for batch jobs invoked using REST API.

5. The pattern uses a messaging layer to communicate between caller and server of the
API.
Following diagram illustrates the flow.

6. 3. Zuul as an API Gateway and new ESB paradigm:
This specific design just illustrates use of an API gateway for discovery, routing, and
governance of microservices.
This same design using Kafka as a middleware , can be used to replace any other proprietary
ESB e.g. TIBCO. This would allow:
- OOTB HA and redundancy on server side processing
- Can add server(s) on demand and can reduce # of servers as and when required. Scale
up/down will be really easy through Kubernetes infrastructure.
- Final state will always be captured in the data store.
4. Synchronous Request-Reply Pattern using Kafka and Spring:

7. 5. Kafka for ​http://​ Based Requests:

8. 6. Distributed Cache Pattern:
Reference data lookup from data stores should be preferably cached using an in
memory distributed cache like Apache Ignite or Hazelcast. Azure Cloud based Redis
solution can also work.
1. Eviction Policy: In most cache configurations LRU based eviction policy would be
configured.
2. Cache Replication Count: In most cases we will keep at least 1 redundant copy of
data, but we can also go with 2 copies.
3. Cache can be configured to overflow to disk, for a high volume of data
particularly if we are using cache over an RDBMS store. This will allow SQL92
queries on the cache itself.
4. Most suitable pattern we can follow is “cache aside pattern” in which we hit data
stores only when we face cache miss during a query, and when an update or
delete happens, we immediately sync the cache with that entry.

9. 7. Mechanisms of Real-time Server Updates to the web browser to mimic
synchronicity for web operations:
Long-Polling vs WebSockets vs Server-Sent Events - Client Pull vs Server
Push
Ajax Polling (Client Pull)
Polling is a standard technique used by the vast majority of AJAX applications. The basic
idea is that the client repeatedly polls (or requests) a server for data. The client makes a
request and waits for the server to respond with data. If no data is available, an empty
response is returned.
1. The client opens a connection and requests data from the server using regular
HTTP.
2. The requested web page sends requests to the server at regular intervals (e.g.,
0.5 seconds).
3. The server calculates the response and sends it back, just like regular HTTP
traffic.

10. 4. The client repeats the above three steps periodically to get updates from the
server.
The problem with Polling is that the client has to keep asking the server for any new
data. As a result, a lot of responses are empty, creating HTTP overhead.
HTTP Long-Polling (Client Pull)
This is a variation of the traditional polling technique that allows the server to push
information to a client whenever the data is available. With Long-Polling, the client
requests information from the server exactly as in normal polling, but with the
expectation that the server may not respond immediately. That’s why this technique is
sometimes referred to as a “Hanging GET”.
● If the server does not have any data available for the client, instead of sending
an empty response, the server holds the request and waits until some data
becomes available.
● Once the data becomes available, a full response is sent to the client. The client
then immediately re-request information from the server so that the server will

11. almost always have an available waiting request that it can use to deliver data in
response to an event.
The basic life cycle of an application using HTTP Long-Polling is as follows:
1. The client makes an initial request using regular HTTP and then waits for a
response.
2. The server delays its response until an update is available or a timeout has
occurred.
3. When an update is available, the server sends a full response to the client.
4. The client typically sends a new long-poll request, either immediately upon
receiving a response or after a pause to allow an acceptable latency period.
5. Each Long-Poll request has a timeout. The client has to reconnect periodically
after the connection is closed due to timeouts.
WebSockets (Server Push)
WebSocket provides ​Full duplex​ communication channels over a single TCP connection.
It provides a persistent connection between a client and a server that both parties can
use to start sending data at any time. The client establishes a WebSocket connection

12. through a process known as the WebSocket handshake. If the process succeeds, then
the server and client can exchange data in both directions at any time. The WebSocket
protocol enables communication between a client and a server with lower overheads,
facilitating real-time data transfer from and to the server. This is made possible by
providing a standardized way for the server to send content to the browser without
being asked by the client and allowing for messages to be passed back and forth while
keeping the connection open. In this way, a two-way (bi-directional) ongoing
conversation can take place between a client and a server.
WebSockets, though, may need stricter enforcement of security.
Server-Sent Events (SSEs) (Server Push)
Under SSEs the client establishes a persistent and long-term connection with the server.
The server uses this connection to send data to a client. If the client wants to send data
to the server, it would require the use of another technology/protocol to do so.
1. Client requests data from a server using regular HTTP.
2. The requested web page opens a connection to the server.

13. 3. The server sends the data to the client whenever there’s new information
available.
SSEs are best when we need real-time traffic from the server to the client or if the
server is generating data in a loop and will be sending multiple events to the client.
In summary, long-polling opens an HTTP request and remains open until an update is
received. Upon receiving an update, a new request is immediately opened awaiting
the next update. Server-sent events(SSE) rely on a long-lived HTTP connection, where
updates are continuously sent to the client.
8. OAuth 2.0 for Data Acquisition Platform: Conceptual flow

14. 9. Spring and Kafka to replace ESB for microservices: (Pls refer to #3)
10. API Design Best Practices:
As a guiding principle all reference data lookups from other modules should be accessed
via APIs.
1. Use an API Gateway like Zuul, pls reuse the features it supports OOTB. We can also
use the gateways provided by Azure as it reduces the overhead cost for us.
2. Security, throttling, scaling, redundancy, SLAs; all need to be considered.
3. Swagger documentation with examples needs to be built.
4. Most API calls are meant to be synchronous so if you have any long running backend
execution, pls consider a call back kind of implementation or the one using messaging.
5. If the API is querying something, pls make sure limits are in place. This will restrict the
payload size. Consider pagination if this is not possible.

15. 11. Recommendations or factors to consider when using 3rd party API:
1. Usually there are APIs with several signatures; some study of the API exposed via
Swagger is needed to identify the best suited one or the most recommended one. Pls
consider the pagination aspect if a large record set is being returned. E.g. For mobile
devices large payload processing could eat up all the memory and app may crash.
Similar situations have happened with BW in CPS in the past; pls do not make any open
ended queries; always have limits in place.
2. Security is utmost important. We have to use https:// along with other security like
API key/auth token etc.
3. API calls are usually synchronous but if you find that latency or blocking time is too
much; you will have to explore if the vendor can support the call back at the end of
processing e.g. if a large file processing is involved. This will allow async calls.
4. Scalability: We had issues with several third party APIs before. Pls understand the
limit on concurrent calls supported. Also pls make sure we can have multiple clients
connecting to the API.
R​EFERENCES
1. Async handling of API Requests :
https://docs.google.com/drawings/d/1WZTVBtnHfQvCqdh6XVOhS4SrnQ8Cv-bCG52BAft4d2o/edit
2. Zuul based request-reply architecture for REST calls:
https://docs.google.com/drawings/d/1hXEPF4O-To783TnSckh2YxMG6ldVmcw33eiZr9_5-2Y/edit
3. Kafka: Request-Reply pattern: ​https://dzone.com/articles/synchronous-kafka-using-spring-request-reply-1
4. Distributed Cache Pattern:
https://docs.google.com/drawings/d/1e1nLZHSQ2qBiHRWa1_bVPBppYoHPKg_wUYKVv-w2Phk/edit
5. OAth 2.0 Flow for Data Acquisition Platform:
https://drive.google.com/file/d/10b9Qpp-YuPjW3LHm4cqVa9thCY0BNbqE/view?usp=sharing
6. A comparison between WebSockets, server-sent events, and polling
https://aquil.io/articles/a-comparison-between-websockets-server-sent-events-and-polling
7. Microservices Guide ​https://martinfowler.com/microservices/