Denver MuleSoft Meetup: Greatest MuleSoft Hits of 2022

All contents © MuleSoft, LLC
DENVER MULESOFT MEETUP
Greatest MuleSoft Hits of 2022
Speakers:
Connor Fitzgerald: MuleSoft Lead Consultant
Dipak Patel: MuleSoft Architect
Jared Mosley: MuleSoft Lead Consultant

Speaker
2
Brian Statkevicus
MuleSoft Ambassador & MuleSoft Practice Manager
Big Compass
+ Based in the Denver metro area
+ Over 20 years of Integration experience
+ Hobbies - Skiing, Hiking, Participating in 5Ks
(I’m too slow to say I actually ‘run’ in them)
+ Ex-soccer ref, but I still know how to show a
yellow and red card

All contents © MuleSoft, LLC 3
Stephanie Lawrence
Denver MuleSoft Meetup Coordinator
Big Compass
Denver Meetup Introductions
Brian Statkevicus
MuleSoft Practice Manager
Big Compass
Aaron Lieberman
Director, Platform Engineering
Science 37

● First person to call out the correct answer, wins!
● One answer per person per question
○ If you answer more than 1x per question, you are ineligible for that question
● If you’ve already won during today’s Meetup, you cannot win again
● Big Compass makes the final decision on the trivia winners that will be submitted to MuleSoft
(see next slide)
Trivia Questions - “House Rules”
More important Trivia guidelines on next slide

● Big Compass emails trivia winners a link to a Google Form for the trivia winners to complete
○ Your email MUST match your training account email; otherwise, you won’t receive a voucher
● Once you complete this form, MuleSoft will verify:
○ You haven’t already won this month
○ You’re not trying other means to circumvent the 1x/month rule
● MuleSoft will send you the voucher
○ The voucher is a training voucher, not a standalone exam voucher. If you select an exam on the
Google Form as opposed to selecting a course, you will not receive a voucher.
○ MuleSoft retains the final decision on determining your eligibility to receive a training voucher
● If you have trouble redeeming a voucher or have not received a voucher, open a ticket at
training.mulesoft.com/question. DO NOT CONTACT US!
Trivia Questions - MuleSoft Guidelines

● You must complete the training request form by December 31, 2022
● You MUST schedule a class by January 20, 2023. The actual class can occur anytime, but
you must schedule by January 20, 2023
Otherwise, your voucher will be null and void
Final note about Vouchers

+ Connor Fitzgerald: RPA - Building Your First Automation
+ Dipak Patel: Large Payload Processing with MuleSoft
+ Jared Mosley: Handling Bursts of Out of Order Events
Agenda

Connor Fitzgerald
About Me
● MuleSoft Lead Consultant | Big Compass
● Live in Atlanta, Georgia
● Hobbies - DIY Projects and Exercise

Robotic Process
Automation
Building Your First Automation

Robotic Process Automation
Why? How?
Overview
What?

MuleSoft RPA
RPA Recorder
● Record / Learn
● Scaffold
● Model Process
RPA Builder
● Manage RPA Assets
● Build
● Test
Components Overview
RPA Manager
● Manage
● Evaluate
● Design
● Administer

Use Case Demo 1
Invoice to Legacy CRM
Use Case Requirements
● Input: Scanned PDF Invoices
● Output: CRM/ERP Database Entry
● Limitation & Hurdles
○ Handwritten values
○ Legacy System
■ Target is not reachable via API

API-Led Integration
Demo 1 - Automation Options

Use Case Demo 2
Emails to External CRM
Use Case Requirements
● Input: Emailed Service Tickets
● Output: 3rd Party CRM System
● Limitation & Hurdles
○ No Existing or Planned Integrations with Target
○ Timeline and Budget

API-Led Integration

Trivia Question

A. RPA can be used a stopgap while robust integrations are in development
B. RPA can work in tandem with other integration tools
C. RPA can be used to replace batch integrations
D. Process logic is defined using RPA Builder
Trivia Question #1 - Which of the following is false?

Questions?

Dipak Patel
About Me
● MuleSoft Architect | Big Compass
● Live in Denver, Colorado area
● Hobbies: sports (Tennis, Volleyball,
Table Tennis, Cricket, Badminton), Drawing
dipak.patel@bigcompass.com | www.linkedin.com/in/dipakppatel

Large Payload Processing
Using MuleSoft

Challenges
Large payloads pose many challenges, such as:
● HTTP Request timeout
● API call timeout during processing
● Process running out of memory
● Database connectivity errors
● MQ message size limitations

Use Case
● Two system APIs to read large amount of data from HTTP urls and two
system APIs to store that data into a database tables
● Number of records are undefined - could be thousands or even millions of
records
● Dynamic data - being updated at any arbitrary frequency
● Insert retrieved data into database of choice

Design Considerations
● Follow API-Led architecture
● The process API should be flexible enough to add additional system APIs
when the need arises
● Non-functional requirements for security, persistence, and performance
● Streaming, batching, throttling, and reliability must be used to deal with large
payload processing
● Mule’s batch processing vs MQ based solution

High Level Design
● Process API is invoked by a
scheduler or on-demand.
● Process API calls two system APIs
to bring data as well as two system
APIs to insert data using HTTP
Request.
● System APIs send the response
back to process API using MQ.
● DB bulk insert operation is used.
● Client-id and client-secret along
with secured properties are used.
● DLQs are used for failed batches.

Challenges and Solutions
Challenge Solution
HTTP timeout in system APIs
to retrieve due to large data
stream
Use HTTP response streaming.
Set Output MIME Type using
outputMimeType = "application/csv;
streaming=true"
Out of memory in system API
to Transform large data
stream
Use streaming functionality in Transform.
@StreamCapable() annotation
Out of memory in system API
to publish records in MQ
This happens because of MQ message limit
of 10 MB.
Use batch of 100 records using for-each.
HTTP timeout in process API
while calling system APIs
Make system APIs async behavior.
Wrap all processors in Try >> Async scope.
HTTP connectivity error in
process API while calling DB
system APIs
Use throttling in MQ subscribe to limit
number of message processing and
therefore number of open connections.
Use Subscribe type of ‘Polling’ with Fetch
Size of ‘10’ every second.

Implementation - Process API
Calls to long running
System APIs are wrapped
in Async to avoid HTTP
timeout error.

Implementation - System API
Streaming is enabled for the
HTTP URL Request and
Transformer to avoid Out of
Memory error.
Batching (using for-each) is used
while publishing to MQ to avoid 10
MB message limitation.

Implementation - Process and System API
Throttling is used in the
message subscriber to avoid
HTTP connectivity error in the
Database API Request.
Streaming is used
in the transformer
to avoid Out of
Memory error.
Bulk Insert is
used to improve
performace

Resources
● Streaming in Dataweave:
https://docs.mulesoft.com/dataweave/2.4/dataweave-streaming
● Streaming in HTTP Listener:
https://docs.mulesoft.com/http-connector/1.6/http-listener-ref
● MQ subscriber flow control: refer to Predictable Message Consumption in
https://docs.mulesoft.com/anypoint-mq-connector/3.x/anypoint-mq-listener

A. Batch processing may not allow you to have separate system APIs to retrieve
and insert data into a database, and therefore it would break the API-led
design principles.
B. Batch processing increases application complexity while limiting scalability.
C. Both
D. None
Trivia Question #2 - What is true about Mule’s batch
processing option for this use case?

Jared Mosley
About Me
● MuleSoft Lead Consultant | Big Compass
● Live in Seattle, Washington
● Hobbies - Music, Video Games, and Reading
● Passions - Curiosity and Connections

Handling Bursts of
Out of Order Events
Shopify to Salesforce

The Problem
Shopify Salesforce
eShopSync
Connector

Problems - Unordered Events
● Large objects with a top-level
change timestamp
● We only care about changes to
certain ﬁelds
● Custom triggers on Shopify cause a
series of updates to an object on
common actions
● Salesforce is receiving an update
anytime the modiﬁcation timestamp
changes

Problems - Time
● Events from Shopify are not guaranteed to be
ordered
● Events on an object come in bursts of 1-10
events in a second
● No way to tell if an event burst has ﬁnished
● No “rest” state for an object as it can be modiﬁed
at any time
● Only queues are VM queues available
● No intermediary storage is available

The Good Things!
● We only care about the latest state of the
object
● A 20 second delay is considered acceptable
● We know which ﬁelds we care about ahead
of time
● We don’t need to automatically store or
replay events

The Solution
Shopify Salesforce
MuleSoft

The Solution
Shopify Salesforce
MuleSoft
latestHash
latestUpdatedAt
updatedAt
eventId
latestHash
latestUpdatedAt

Wrap Up
● We can track every event from Shopify to
Salesforce
● We have greatly reduced the number of calls to
Salesforce
● Our events are always propagated in order
● We don’t make unnecessary update calls
● We can manually retrigger speciﬁc or latest
updates if needed

A. All events are sent to Salesforce
B. The event is newer than the one we have
C. The event’s hash doesn’t match the one we have
D. It is the latest event in the burst of events
E. B & C
F. C & D
Trivia Question #3 - What events do we send to Salesforce?

Next Steps

● Share:
○ Tweet using the hashtag #MuleSoftMeetups
○ Invite your network to join: https://meetups.mulesoft.com/denver
● Feedback:
○ Complete the Meetup survey, provide feedback, and suggest topics for upcoming events
○ Contact MuleSoft at meetups@mulesoft.com for ways to improve the program
What’s next?

● Have a great Holiday season! See you in 2023
● Future (tentative) Meetup schedule:
○ Mid February
○ Mid April
○ Mid June
○ Mid August
○ End of September
○ Mid November
○ Plus any special virtual roadshows
● We are looking for topic suggestions and speakers!
○ DataWeave is always popular
○ Local customer experience
○ Logging revisited
○ Deeper dive into RPA Builder
○ Tableau + MuleSoft
Denver Meetups 2023

Thank you!

Denver MuleSoft Meetup: Greatest MuleSoft Hits of 2022

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Denver MuleSoft Meetup: Greatest MuleSoft Hits of 2022

Semelhante a Denver MuleSoft Meetup: Greatest MuleSoft Hits of 2022 (20)

Mais de Big Compass

Mais de Big Compass (6)

Último

Último (20)

Denver MuleSoft Meetup: Greatest MuleSoft Hits of 2022