SpringIO 2015 conference slides about usage of Spring Batch in large enterprises.

1. Spring Batch for large
Enterprises operations
Ignasi Gonzalez
@ignasi15g

2. Let me do some initial questions

3. Which is the concept?

4. Batch Definition

5. Batch Usages
Correct Clean
GENERATE
Order

6. Agenda
• Batch at Java and Spring
• Usage of Spring Batch at a bank
• Usage of Spring Batch at an energy company

7. Batch Structure
JSR 352: Batch Applications
for the Java Platform
(May 24th, 2013)

8. Other Concepts
Batchlet
Chunk type Step
Batchlet type Step
Job Execution

9. JSR 352 & Spring Batch
JSR – 352
• Define an API.
• Provide an SDL (JSL)
• Provide a contract.
• Dependency injection is optional
Spring Batch
• Define an API.
• Provide an SDL. (JSL)
• Provide a contract.
• Provide implementation for Readers and
Writers.
• Provide specific batchlet
implementations.
• Provide JobRepository implementation.
• From Spring 3.0 support JSR-352.

10. Spring Batch Structure
http://docs.spring.io/spring-batch-old/1.1.x/spring-batch-docs/reference/html-single/index.html#d0e5652

11. Spring Batch Data Model
http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html

12. Spring Batch Data Model
http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html
BATCH_JOB_INSTANCE

13. Spring Batch Data Model
http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html
BATCH_JOB_EXECUTION

14. Spring Batch Data Model
http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html
BATCH_JOB_EXECUTION_PARAMS

15. Spring Batch Data Model
http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html
BATCH_STEP_EXECUTION

16. Spring Batch Data Model
http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html
BATCH_STEP_EXECUTION_CONTEXT

17. Spring Batch Data Model
http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html
BATCH_STEP_EXECUTION_CONTEXT
BATCH_JOB_INSTANCE
BATCH_JOB_EXECUTION
BATCH_JOB_EXECUTION_PARAMS
BATCH_STEP_EXECUTION

18. Main Benefits
• Integrated with full Spring framework supporting all Spring features such
as Dependency Injection, AOP & Testability.
• Well tested ItemReaders and ItemWriters covering vast options for I/O
(File, JDBC, NoSQL JMS).
• Management information of running and finished Batch processes,
including information about results.
• A collection of Tasklets (Spring Batchlets) for execution from shell scripts
to Hadoop processes.
• Framework flexibility to define complex steps configurations.
• Simplification of thread management and transactions management.
• Management of Batch lifecycle (Restartability, Run, Pause, Stop).

19. SPRING BATCH AT A BANK

20. BANK
• The bank is undergoing a process of migration of core applications from Host to a mid-level
system that uses a Spring-based architecture defined, developed and maintained by everis.
Time to
market
MIPS
reduction
Scalability
TestabilityCost
reduction
HOST MID-LEVEL SYSTEMS

21. Architecture
• Spring based architecture:
– Performs 7.000 transactions per second.
– Provides service to 30.000 users
– Architecture team composed by 30 software engineers.
INFR.
CORPORATE
CHANNEL
SERVICES
HOST
LOGIN
IDM
BATCH
BACKENDS
ONLINE
CHANNEL
CRYPTO
NEUTRAL
CHANNEL
OTHER
MEDIA
STORAGE
FINANCIAL
TERMINAL
ONLINE
BANKING
GROUP
COMPANIES
ATM
BATCH

22. Challenge
HOST
How to integrate Spring Batch in corporate tools?
SPRINGBATCH

23. Integration
Corporate scheduler

24. Integration
Corporate batch launcher
ClassPathXmlApplicationContext sbc = BatchApplicationContextUtils.createApplicationContext(properties);
JobLauncher launcher = (JobLauncher) sbc.getBean("jobLauncher");
Job job = (Job) sbc.getBean(this.jobId);
initBatchContext(jobParameters, appParams, jobParam, sbc);
public static ClassPathXmlApplicationContext createApplicationContext(Properties credentials)
throws IOException {
String[] contextFiles = new String[]{
"classpath*:batch/arch/internal/beans/-batch-arch*.xml",
"classpath*:batch/app/**/-batch-app*.xml"
};
ClassPathXmlApplicationContext sbc = new ClassPathXmlApplicationContext(contextFiles, false);
PropertyPlaceholderConfigurer ppc = new PropertyPlaceholderConfigurer();
ppc.setSystemPropertiesMode(PropertyPlaceholderConfigurer.SYSTEM_PROPERTIES_MODE_NEVER);
ppc.setIgnoreUnresolvablePlaceholders(true);
ppc.setProperties(credentials);
sbc.addBeanFactoryPostProcessor(ppc);
sbc.refresh();
return sbc;
}
protected void initBatchContext(JobParameters jobParams, String[] appParams, String[] jobParam,
ClassPathXmlApplicationContext xmlcontext) {
BatchContext.reset();
BatchContext bc = BatchContext.getInstance();
BatchParams bp = new BatchParams();
bp.setJobName(this.jobId);
bp.setInstanceId(jobParams.getString(BatchLauncher.ArchParamKey.INSTANCIA.toString()));
bp.setScheduleTimestamp(jobParams.getDate(BatchLauncher.ArchParamKey.SCHED_TS.toString()));
bp.setAppParams(Arrays.asList(appParams));
bp.setJobParam(Arrays.asList(jobParam));
bp.setJobentorno(jobParams.getString(BatchLauncher.ArchParamKey.ENTORNO.toString()));
bp.setJobappmvs(jobParams.getString(BatchLauncher.ArchParamKey.APPMVS.toString()));
bc.setBatchParameters(bp);
bc.setxmlContext(xmlcontext);
}

25. Results
Main areas with Batch processes:
• Markets
• IT
• Risk Management
• Digital Contents
• Banking services
• ATMs
• Online Banking
• Extranet
0
50000
100000
150000
200000
250000
ADCAEX
AGERIE
AMPAMP
ARTBKD
AUWATC
BCKARM
BCKPBC
BCKSDD
BKDAIR
BKDGBR
BKDPRP
CALBCK
COMCNF
ECCVAL
EPQEPQ
GESINS
IMPIMP
LVACUA
MUROGE
PERCAI
PUNBKD
SELCLI
SPCSPC
TRACEA
BSMTST
IDECLD
SISSEG
# executions per process
# executions per month

26. SPRING BATCH AT AN ENERGY COMPANY

27. ENERGY COMPANY
• Software factory composed by 4 different
providers.
• More than 300 developers.
• Migration from custom developed framework
to a Spring based architecture.

28. Architecture
Integración
JMS/MQ
WebServices
Documentum
Dazel/Papirus

29. Challenges
1. How to migrate current custom batch processes
on new Spring operational architecture?
1. How to ensure quality and correct Spring Batch
usage with a community of 300 developers
2. How to monitor and report batch activity
formed by more than 400 processes with 160
processes per day executions?

30. 1. MIGRATION

31. Migration
PoC Objectives:
• Evaluate the effort required by the
development teams to migrate.
• Evaluate the performance.
• Evaluate the total execution time.
• Identify improvements over
migrated process.

32. Migration
• Obtained results:
 30% Reduction of the Execution time
 15% Reduction of the Memory consumption
 10% Reduction of the CPU consumption
 Process migration, including tests, has been performed in 8 Working Days without
previous Spring Batch experience.
 Reduction of the proprietary code.
 Maintainability on batch processes (Query + DAO + Processor + Config writer).
• Where do we obtain performance benefits?
– Using JDBCPagingItemReader, we eliminate a reading lock at the beginning of the
original process.
– Using ApacheCommonsDBCP as a Datasource implementation for Reader, we optimize
the process as the old architecture component had performance issues.
– Using FlatFileItemWriter to generate CSV output file, the results are improved
compared to the ones previously obtained

33. 2. QUALITY

34. Quality

35. Quality
LOCAL TEST ACCEPTANCE
TRAINING PRODUCTION
PRE
PRODUCTION
PERFORMANCE
Productive environmentsTactic environments
Development
environments
Monitoring
QA environments
Dynamic Code AnalysisReporting & Bugtracking Static code analysis Performance testing
& profiling
Continuous Integration Static Data Model Analysis
OLTP

36. 3. MONITOR AND CONTROL

37. Monitor and control
01:00 AM
02:00 AM
03:00 AM
Planed
04:00 AM
05:00 AM
06:00 AM
01:00 AM
02:00 AM
03:00 AM
Happens
04:00 AM
05:00 AM
06:00 AM

38. Monitor and Control

39. Monitor and Control

40. Monitor and Control

41. Monitor and Control

42. Monitor and Control

43. Needs
Monitor and Control
Batch Executions
• No global and complete reporting tool around Batch processes.
• Difficulties on schedule modifications because huge number of processes (432) and high
number of executions (160 per day).
• No alert mechanism if planned Batch doesn’t start (Planned vs Executed).
Batch Plannifications

44. Monitor and Control
44
Batch Executions
Logs
Planificación Batch
• Daily global information.
• Visual control of executions vs planification
• Historical Data..

45. Conclusions
• Structure Batch following standards and best practices.
• Well tested and optimized components.
• Possibilities to exploit horizontal scalability framework offer.
• Integration with the rest of the Spring framework (Spring Integration / Spring XD).
• Reusability of business logic developed in online applications.
• Consider the complexity of integrating Spring Batch with existing enterprise tools.
• Migration process from current batches must be normalized to SpringBatch
characteristics.
• Ensure invariability of the process after migration using appropriate testing tools.
• Ensure correctness of framework usage using checkstyle tools.
• Design batch having in mind advanced framework capacities (chunking, parallelism).
• Measure processes execution ensuring correctness during all process life
Benefits
Advises

48. Spring Batch for large
Enterprises operations
Ignasi Gonzalez
@ignasi15g