Geant4 is the main simulation framework used for hight energy physics. It is used by the majority of the LHC experiments in CERN for detector simulation. Physics validation is a form of regression testing, that ensures that the produced simulation data do agree with previous versions of the framework and follow similar patterns with real data acquired from the experiments. I developed a new system that is able to formulate the process of running Geant4 applications/tests on the LHC world-wide computing grid (WLCG), gather the results, stores them in relational database for indexing and object storage and provide them to the user through an intuitive web user-interface that can run on-demand small analysis tasks.

1. George Lestaris
supervisors: W. Pokorski & A. Ribon
Geant4 physics validation
on the GRID
SFT meeting - 28/04/2014

2. Geant4 physics validation on the GRID
Introduction to the problem
• Geant4 is a simulation framework
• Physics validation is form of regression testing
but:
• simulation data can be compared with respect to
real (reference) data (from papers, test beams, etc)
• Monte Carlo simulation does contain by its nature
the concept of randomness
• need to run many simulation events to get data that
could be statistically close to previous versions
2

3. Geant4 physics validation on the GRID
Previous work
• Solution tailored for Simplified calorimeter
• a Geant4 application that simulates the showers that are
produced inside simplified geometries of LHC
calorimeters
• DIANE used to run jobs on GRID resources
• not very flexible:
• unable to easily restart single jobs
• scheduling among different GRID sites was round-
robin
• not maintained/supported anymore
3

4. Geant4 physics validation on the GRID
Previous work
• Process required manual intervention in different
stages..
• Error-prone actions
• Minor problems
• Slow access to stored data: mis-index MySQL
database
• Slow plotting: a new ROOT session per graph
4

5. Geant4 physics validation on the GRID
Previous work
5
Worker
Step 1
Build and store
binaries to CVMFS
Step 2
Setup DIANE Master
Step 3
Submit GANGA Jobs
Step 4
Run analysis of results
CVMFS
MySQL
DB
GANGA Iterface
WLCG
LxBatch
WorkerWorkerWorker...
DIANE Master
Web application
AFS

6. Geant4 physics validation on the GRID
Motivation
• Extensible system to
support multiple tests
• Use of a better
supported WMS
• Automations to
reduce the room for
user errors during jobs
submission and running
• Efficient storage and
plotting
6
Creation of
DIANE master
Preparation of
AFS directory
Submission of
pilot jobs
Results merging,
analysis and
storage
...
Old
Submission of
production
...
New
Results become
available in the
web application

7. Geant4 physics validation on the GRID
g4.Val
7

8. Geant4 physics validation on the GRID
Architecture
• Component-based architecture
• Core component contains essential classes and
modules, referenced by all other components
8
Production side:
• Production: Submission
of jobs.The job wrapper
that runs the tests,
collects stores and merges
results.
• Client: Communication
with the Web API.
Presentation side:
• DB: Abstract interface to
the database
• Analysis: Runs lightweight
analysis tasks (e.g.:
plotting) over the results.
• Web: Hosts the UI and
the results submission API.

9. Geant4 physics validation on the GRID
Architecture
9
Core
API Client
Production
Web
DB
Analysis
speaks API
DIRAC
server EOS MySQL
Redis
queue

10. Geant4 physics validation on the GRID
User roles (actors)
Test manager: person that imports a test in the
system
• not necessarily the developer of the test but
someone who understands how to run it
Production manager: submits the productions
(sets of jobs) for tests
End user: that looks into the final results (web UI)
by running on-demand analysis (plotting) tasks
10

11. Geant4 physics validation on the GRID
End user
11
http://g4-val.cern.ch/
Tests list Analysis plugins

12. Geant4 physics validation on the GRID12
Filters
Tabs
Results UI

13. Geant4 physics validation on the GRID
Results UI components
13
Version
selection
Filters can
be lists
of values

14. Geant4 physics validation on the GRID
Results UI components
14
Bookmarkable
pages
Different analysis (plotting) plugins

15. Geant4 physics validation on the GRID
Production manager
• Submission with a recipe
• predefined recipe that submits a production by
changing the Geant4 version
• Command line tool to interactively create
custom production
• Python API
• if need custom input sandbox, environment,
dependancies
15

16. Geant4 physics validation on the GRID
Recipe submission
$> source /cvmfs/geant4.cern.ch/grid/etc/login.sh
# prompt to type the certificate password
$> g4val-prod-recipe -t sc -v 10.0.ref03 -r test
# some settings
...
...
...
[g4val.production.submitter.dirac INFO] 2014-04-25 16:38:43 CEST -
Submitted job with reference: Id = 64849
[g4val.production.submitter.dirac INFO] 2014-04-25 16:38:43 CEST - 8 jobs
were successfully submitted!
16
Test code Version Recipe

17. Geant4 physics validation on the GRID
DIRAC
The DIRAC (Distributed Infrastructure with
Remote Agent Control) project is a complete Grid
solution for a community of users needing access to
distributed computing resources. DIRAC forms a
layer between a particular community and various
compute resources to allow optimized, transparent
and reliable usage.
17

18. Geant4 physics validation on the GRID
DIRAC (in English)
• Developed for the LHCb experiment initially
• Today a collaboration with many users around the world
• Geant4 uses a small subset of its functionality
• Pilot job approach as well
• Scheduling is done opportunistically, i.e.: the site that
runs the most pilot jobs runs the most DIRAC jobs
• Able to restart/reset/reschedule jobs
• Web UI to list, filter, monitor and manage jobs
18

19. Geant4 physics validation on the GRID
Test manager
Needs to define the test model
• what are the variables that can uniquely identify a
test run?
• what are the variables, extracted from the test
results, that are interesting to present?
19

20. Geant4 physics validation on the GRID
Test model
Attributes and parameters: variables that characterise
a test instance (execution)
Test run: assignment of values to all the test parameters
• A test run is unique
• Parameters are indexing list of runs
Observables: variables that describe results
• Can be tuples, list of numbers, or pointers to (ROOT) files
• A test observable is also having its attributes and
parameters
20

21. Geant4 physics validation on the GRID
Test model
21
str code
str name
str description
Test
str code
str name
int type
mixed default_value
bool allow_null
Attribute
bool use_reference
Parameterstr code
str name
str description
Observable
*
1
*
*
1
1

22. Geant4 physics validation on the GRID
Test running
A lot of extensions points available to override
system default behaviours for a test:
• Test running: production hooks
• add dependencies (libraries from CernVM-FS)
• environment (variables)
• macro file generation (template)
• macro file is used as an input to a Geant4
application
22

23. Geant4 physics validation on the GRID
Results storage
• Run splitting and merging:
• big (long) runs can be split to smaller jobs (e.g.:
reducing number of events)
• Results storage: production hooks
• raw results harvesting if necessary
• format of registered results (observables)
23

24. Geant4 physics validation on the GRID
Results presentation
On-demand analysis
• analysis plugins are independent of a test, i.e.: can
be reused
• mapping between what observables can be
plotted with which analysis plugin
24

25. Geant4 physics validation on the GRID
Technical details
(backup slides)
25

26. Geant4 physics validation on the GRID
Production: Job splitting and submission
• Large test runs can be split to smaller jobs
with reduced number of events
• Each job appends its results to the run results
• List of jobs is submitted with the submitter
interface
• Current implementation with DIRAC, but can be
any
• Job scheduling is done by the WMS
26

27. Geant4 physics validation on the GRID
Production: Job wrapper
g4.Val gets involved again when the job arrives to a
worker node (WN)
• Job wrapper is stored in CernVM-FS
• Job is a simple JSON file with its input sandbox
27

28. Geant4 physics validation on the GRID
Production: Job wrapper
1. Input sandbox checking
2. Macro file generation given the run parameters
3. Dependency resolution per test for WNs platform
4. Starts the Geant4 process
5. Transfers results to EOS when done
6. Appends job results to the run results if run was
split
7. Registers the results to the database with the API
client component
28

29. Geant4 physics validation on the GRID
Database component
Database schema is generated upon test import
• table for storing the runs
• one table for each observable
• attributes are fields and parameters make up the
table indexes
29

30. Geant4 physics validation on the GRID
Web component technologies
30

31. Geant4 physics validation on the GRID
Analysis component
• Web component submits
analysis task to redis queue
• Analysis workers are
processes that execute these
tasks
• Disposable in case of crashes
• Analysis load balancer starts
new workers if the queue has
many waiting jobs
31
AJAX
Server
Web
interface
User Browser
HTTP
Request
Request
Request
...
Workers
Database

32. Geant4 physics validation on the GRID
Examples
(backup slides)
32

33. Geant4 physics validation on the GRID
Simplified calorimeter
• Big runs, split to many jobs
• Raw results as ROOT files (merged with
TFileMerger)
• Registered results are value-error tuples for the
six observables
• Using many runs in one graph as beam energy is the
X-axis parameter
• Plotted with HighCharts JavaScript library (client
side)
• or ROOT TGraphError (server side)
33

34. Geant4 physics validation on the GRID
Simplified calorimeter
34
http://bit.ly/1lZQef0

35. Geant4 physics validation on the GRID
Geant4 test test30
• Quick runs, no need of splitting
• One pilot job can run tens of test30 jobs
• Raw results are ROOT files
• Registered results are these ROOT files URLs
(EOS supports natively XRootD)
• Each TObject is an observable
• Plotted with ROOT TObject superimpose plugin
35

36. Geant4 physics validation on the GRID
Geant4 test test30
36
http://bit.ly/1lXtSLc

37. Geant4 physics validation on the GRID
Example of test’s model
from g4val.model import Test, Attribute, Parameter
test = Test(
code="sc", name="Simplified calorimeter", description="..."
)
# Parameters
test.add_attribute(Parameter(
code="physics_list", type=Attribute.STRING, type_size=100, name="Physics list",
use_reference=False
))
test.add_attribute(Parameter(
code="calorimeter", type=Attribute.STRING, type_size=100, name="Calorimeter"
))
test.add_attribute(Parameter(
code="beam_particle", type=Attribute.STRING, type_size=5, name="Beam projectile",
default_value="pi-"
))
test.add_attribute(Parameter(
code="beam_energy", type=Attribute.FLOAT, name="Beam energy"
))
# Attributes
test.add_attribute(Attribute(
code="events_amt", type=Attribute.SHORT, name="Number of events",
default_value=5000
))
37

38. Geant4 physics validation on the GRID
Example of macro generation hook
def get_job_macro(job):
return """#verbose
1
#events
%(events_amt)d
#particle
%(beam_particle)s
#material
%(material)s
#energy(MeV)
%(beam_energy)s
#generator
%(generator)s
#run
#exit
""" % job
38
• From test test30
• Test parameters are:
• Number of events
• Beam particle
• Beam energy
• Material
• and Generator
• job is inheriting
Python dict

39. Geant4 physics validation on the GRID
Conclusion
g4.Val is a flexible, extensible system for Geant4
physics validation in our group
• Introduces a Geant4 test formalisation
• Uses DIRAC for job submission and running on
GRID and potentially other batch resources
• Efficient storage of results
• On-demand analysis tasks through an highly
interactive intuitive web interface
39