What are shared queues in IBM MQ, and diving into the techincal details of configuring your setup.

1. IBM WebSphere MQ for z/OS
Shared Queues
Alexander Ross (alexross@uk.ibm.com)
Software Developer

2. Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal
without notice at IBM’s sole discretion. Information regarding potential future products is
intended to outline our general product direction and it should not be relied on in making a
purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract. The development, release, and
timing of any future features or functionality described for our products remains at our sole
discretion
Performance is based on measurements and projections using standard IBM benchmarks in a
controlled environment. The actual throughput or performance that any user will experience
will vary depending upon many factors, including considerations such as the amount of
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and
the workload processed. Therefore, no assurance can be given that an individual user will
achieve results similar to those stated here.
© 2014 IBM Corporation
2

3. 3
Agenda
• What are Shared Queues?
• Large messages with DB2
• Shared Message Data Sets (SMDS)
• Structures – Persistence and recovery

4. Application Application
5
Shared queues
QMgr QMgr
Shared
Coupling Facility (CF) Queue
Same Sysplex

5. 7
Queue-sharing groups (QSGs)
Queue-Sharing
Group
Shared
Objects
Msg data
> 63KB
DB2 Data Sharing Group
CF
Shared Queues
Channel
Initiator
QMGR
Private
Queues
Private
Objects
Channel
Initiator
QMGR
Private
Queues
Private
Objects
DB2A DB2B DB2C

6. 11
Shared Message Data Sets (SMDS)
Queue-Sharing Group
DB2 Data Sharing Group
SMDS SMDS
Shared
Objects
CF
Msg data
> 63KB
Shared Queues
Channel
Initiator
QMGR
Private
Queues
Private
Objects
Channel
Initiator
QMGR
Private
Queues
Private
Objects
DB2A DB2B DB2C
V7.1

7. 13
CF structures for shared queues
{ Coupling Facility
Administration
structure
Application
structures
Information for unit-of-work recovery
{
Administration
Information for unit-of-work recovery structure
Application
Queue Queue Queue
structures
Structures
for QSG 1
Structures
for QSG 2
x 63
Queue Queue Queue x 512

8. Creating CF structures and shared queues
• Define a structure to z/OS by updating the CFRM policy:
– The structure is known to MQ by its 12-character str-name.
– The structure is known to z/OS by the 16-character name formed by:
• Application structures: qsg-name || str-name
• Administration structures: qsg-name || CSQ_ADMIN
• Application structures are defined to MQ using the DEFINE
CFSTRUCT command on any queue manager in the QSG
• Shared queues are defined using the DEFINE QLOCAL
command on any queue manager in the QSG:
– DEFINE QLOCAL(queue-name) QSGDISP(SHARED)
CFSTRUCT(str-name)
• z/OS allocates the structure in the CF when required (first use)
• MQ creates the queue when required (first use)
15

9. 17
Large shared queue messages (using SMDS)
QM1
Ptr to 100K
message
QM1
SMDS
QM2
SMDS
QM2
APP
MQPUT
APP
MQGET
1
2 3
4
Shared queue
V7.1
• One SMDS per queue manager per CF structure
• Each queue manager only writes large messages to its own SMDS
• Queue managers can get messages from any queue manager’s SMDS

10. 3 LPAR Test - SMDS
64KB Non-Persistent Messages In-Syncpoint - SMDS
1 2 3 4 5 6 7 8 9 10
19
SMDS performance improvement
3 LPAR Test - DB2
64KB Non-Persistent Messages In-Syncpoint - DB2
1 2 3 4 5 6 7 8 9 10
400
350
300
250
200
150
100
50
0
NP SIS Scaling –
3 qmgr
Queue Pairs
NP SIS Scaling –
6 qmgr
NP SIS Scaling –
9 qmgr
Transactions / Second
7000
6000
5000
4000
3000
2000
1000
0
NP SIS Scaling –
3 qmgr
Queue Pairs
NP SIS Scaling –
6 qmgr
NP SIS Scaling –
9 qmgr
Transactions / Second
• Tests show comparable CPU savings, making SMDS a more usable
feature for managing your CF storage
• SMDS per CF structure provides better scaling than DB2 BLOB storage

11. Creating a Shared Message Data Set
• SMDS is defined as a VSAM data set using DEFINE CLUSTER
– Requires LINEAR option
– Control interval size must be 4096, which is the default for linear data sets
– Requires SHAREOPTIONS(2 3), that allows one queue manager to write and
other queue managers to read at the same time
– If maximum size may need to exceed 4GB, requires SMS data class which
has VSAM extended addressability attribute
– If automatic expansion is to be supported, requires an appropriate secondary
space allocation, although a default of 20% will be used if an expansion
attempt fails because of no secondary allocation
• Can optionally be pre-formatted, for example using CSQJUFMT
– Otherwise formatted automatically when first opened
21

12. Access to Shared Message Data Sets
• Shared message data sets must be on shared direct access
storage accessible to all queue managers in the QSG.
• Normal running:
– Queue manager opens own data set read/write.
• Requires UPDATE access to own data set.
– Queue manager opens other data sets read-only.
• Requires READ access to all other data sets.
• Media recovery processing:
– Queue manager performing recovery opens own data set and all other
23
data sets for read/write access.
• Requires UPDATE access to all data sets.

13. Shared Message Data Set capacity considerations
• Each shared message data set only contains data for large messages
written by its owning queue manager
• Message size calculation:
– Each stored message includes standard headers (usually 352 bytes)
– Each message is stored as one or more message blocks
– Each message block is stored in a range of consecutive 4KB pages on the
data set, with a very small header (32 bytes)
– Approximate data set space required per large message, in bytes, is given by
size of message plus the header rounded up to the next 4KB
• Multiply by the maximum anticipated backlog of messages written by that
queue manager (plus some safety margin) to estimate the required data set
size
24

14. Shared Message Data Set expansion
• Data set can be automatically expanded when necessary
– Normally configured by the DSEXPAND(YES|NO) option on the CFSTRUCT
object, which specifies the default option for the data set group.
– Can also be overridden for individual data sets using ALTER SMDS with the
DSEXPAND option
• Expansion attempt is automatically triggered when 90% full
– If no secondary allocation was specified a VSAM error message will appear,
but the queue manager will retry using a default secondary allocation of 20%
of the existing size
– If expansion fails because there is not enough space available the queue
manager sets DSEXPAND(NO) to prevent further attempts. The operator can
use ALTER SMDS to restore DSEXPAND(YES) once the problem has been
resolved
– If the maximum number of extents are reached the data set cannot be
expanded further. However, it could be marked unavailable then copied to a
larger data set, which is subsequently renamed to the original name
25

15. 26
Selecting which messages to offload
• Messages too large for CF entry (> 63KB) are always offloaded
• Other messages may be selectively offloaded using offload rules
– On each CFSTRUCT definition, you can specify 3 off-load rules
– Each rule specifies:
• Message size in KB – OFFLDnSZ(size)
• Structure usage threshold – OFFLDnTH(percentage)
where n = 1, 2, 3.
• Data for new messages exceeding the specified size is offloaded (as for a
large message) when the structure usage exceeds the specified threshold
• Default rules are provided which should be useful in most cases
• Rules can be set to dummy values if not required
Offloading CF structure size Message size Number of messages
No 100GB 63KB 1.25M
Yes 100GB Any 140M

16. 0% 70% 80% 90%
~ 140,000
offloaded
messages
28
Storage benefits of offloading
• 1GB structure
• 20KB messages being put
~ 35,000 messages in CF
Thresholds for % of CF structure full
~ 5,000
messages
~ 140,000
offloaded
messages
> 63KB > 32KB > 4KB > 0KB
Message sizes (default rules)
• ~ 320,000 messages using offloading versus ~ 50,000 without offloading

17. SCM Storage (Flash) – Planned Emergency Storage
• z/OS 2.1 and zEC12 GA2 introduce Coupling Facility Flash storage
– Additional capacity for maintenance windows or consumer outages
• Consider CFSTRUCT OFFLOAD rules:
– OFFLD1TH(70), OFFLD1SZ(32K)
• Offload messages larger than 32KB if the structure is 70% full
– OFFLD2TH(80), OFFLD2SZ(4K)
• Offload messages larger than 4KB if the structure is 80% full
– OFFLD3TH(90), OFFLD3SZ(0K)
• Offload all message data if the structure is 90% full
• Progressively smaller messages written to SMDS as the structure fills
• At 90% full the queue manager stores the minimum data per message to
squeeze as many message references as possible into the remaining
storage
• CF Flash algorithm also starts moving the middle of the queue out to flash
storage, keeping the faster 'real' storage for messages most likely to be got
next
30

18. SCM Storage (Flash) – Maximum speed
• Quicker to access messages stored in the CF than in SMDS
– Keep high performance messages in the CF
•CFSTRUCT OFFLOAD rules disabled using OFFLD1SZ(64K)
– Maybe one rule to reduce large message threshold below 63KB
• At 90% full the CF Flash algorithm starts moving the middle of
the queue out to flash storage, keeping the faster 'real' storage
for messages most likely to be got next.
• As messages are got and deleted the CF Flash algorithm
attempts to pre-fetch the next messages from Flash storage
they are rapidly available for MQGET.
• In this scenario the Flash storage acts like an extension to 'real'
CF structure storage. However it is consumed more rapidly in
this scenario since all small message data is stored in it.
32

19. 34
SCM Storage (Flash) – Comparison
Scenario Message
Size
Total
Messages
# Messages
in 4GB 'real‘
SMDS
Space
# Messages in
200GB Flash
Augmented
(limit 30GB)
No SMDS
No Flash
1KB 3M 3M
4KB 900,000 900,000
16KB 250,000 250,000
SMDS
No Flash
1KB 3.2M 3.2M 800MB
4KB 1.8M 1.8M 5GB
16KB 1.3M 1.3M 20GB
Emergency
Scenario
1KB 190M 2M 270GB 190M 30GB
4KB 190M 600,000 850GB 190M 30GB
16KB 190M 150,000 3TB 190M 30GB
Speed
Scenario
1KB 150M 2M 150M 26GB
4KB 48M 600,000 48M 8GB
16KB 12M 150,000 12M 2GB

20. Queue
manager
Private
queues
36
Failure and persistence
Coupling Facility failure
Shared
queues
Queue manager failure
Queue
manager
Private
queues
Queue
manager
Private
queues
Nonpersistent
messages on
private queues
OK (kept)
Messages on
shared queues
OK (kept)
Persistent
messages on
shared queues
restored from log
Queue
manager
Private
queues
Private
queues
Queue
manager
Private
queues
Queue
manager
Private
queues
Shared
queues
Nonpersistent
messages on
private queues
lost (deleted)
Nonpersistent
messages on
shared queues
lost (deleted)

21. 38
Administration structure recovery
• In a disaster recovery situation (e.g. loss of CF)
– Prior to version 7.0.1
• Each queue manager rebuilds its own admin structure entries
• Therefore, you need to start all queue managers to rebuild admin structure
• Once recovered, application structures can be recovered
– From version 7.0.1
• Active queue managers notice if other queue managers don’t have entries
in admin structure and initiate rebuild of admin structure entries on their
behalf
• Application structures can be recovered before all queue managers are
restarted
• A version 7.0.1 or higher, queue managers can recover entries
on behalf of any queue manager at the same or lower version
of MQ
– Not all queue managers in the QSG need to be at V7.0.1, or higher

22. 40
CF loss of connectivity
CF
QM2
QM1
QM3
Pre-V7.1 Queue Managers
A failure of the Coupling
Facility is most likely
going to be presented
to connectors as a Loss
of Connectivity
Prior to V7.1 if a queue
manager receives a loss of
connectivity it will terminate.
If a Coupling Facility failure
occurs this results in an
outage for the entire QSG,
unless protected by CF
Duplexing

23. 41
CF loss of connectivity tolerance
V7.1 or later queue CF2
managers will not
terminate. They will
automatically attempt to
re-establish access to
the affected structures.
CF1
V7.1
QM2
QM1
QM3
V7.1+ Queue Managers
If a total loss of
connectivity occurs the
queue managers can
automatically recover
(RECOVER CFSTRUCT)
the structures that were
on the failed CF in to an
alternative CF if one is
available).

24. 43
CF loss of connectivity tolerance
CCFF12
V7.1
QM2
QM1
QM3
V7.1+ Queue Managers
If a partial loss of
connectivity occurs a
System Managed Rebuild
is automatically initiated
by the queue managers to
rebuild the structures in to
a more available CF. This
means both persistent
and non-persistent
messages are retained.

25. CF admin structure loss of connectivity tolerance
• Queue managers will tolerate loss of connectivity to the admin structure
without terminating if
– The QMGR CFCONLOS attribute is set to TOLERATE
– All the queue managers in the QSG are at version 7.1 or later
• All queue managers in the QSG will disconnect from the admin structure,
then attempt to reconnect and rebuild their own admin structure data
• If a queue manager cannot reconnect to the admin structure, for example
because there is no CF available with better connectivity, some shared
queue operations remain unavailable until the queue manager can
successfully reconnect to the admin structure and rebuild its admin structure
data
• The queue manager will automatically reconnect to the admin structure
when a suitable CF becomes available on the system
• Failure to connect to the admin structure during queue manager start-up is
not tolerated, regardless of the value of CFCONLOS
45

26. 47
CF structure definition considerations
MQ CFSTRUCT definition CFRM policy structure definition
CFSTRUCT(TEST1) STRUCTURE NAME(SQ27TEST1)
CFLEVEL(5) SIZE(50000)
CFCONLOS(TOLERATE) INITSIZE(20000)
RECAUTO(YES) DUPLEX(DISABLED)
OFFLOAD(SMDS) ALLOWAUTOALT(YES)
PREFLIST(P5CF01,P5CF02)
• If using CFCONLOS(TOLERATE) you should also consider:
– RECAUTO – automatic recovery of application structures
– PREFLIST – to allow the structure to be allocated in multiple Coupling Facilities
• ALLOWAUTOALT(YES)
– Enables CF to adjust entry/element ratio, and also automatically resize structure up to
a maximum SIZE value
– MINSIZE can be set to automatically adjust the structure size down
• CF duplexing can be used to duplex MQ structures
– Makes most types of failures transparent to MQ

27. For Additional Information
IBM Training
http://www.ibm.com/training
IBM WebSphere
http://www.ibm.com/software/websphere/
http://www.ibm.com/software/products/ibm-mq
IBM developerWorks
http://www.ibm.com/developerworks/websphere
https://www.ibm.com/developerworks/community/blogs/messaging

28. IBM MQ Sessions this week
10:30 - 12:00 13:15 - 14:15 14:30 - 15:30 16:00 - 17:00 17:15 - 18:15
Tuesday
Opening General Session- IBM Digital
Experience and WebSphere Technical
University
Session A31: IBM
MQ CHLAUTH rules
– with MQ V8
updates
Speaker: Morag
Hughson
Room 02
Session A4: WebSphere
MQ for z/OS:
Performance and
Accounting
Speaker: Alex Ross
Room 8
Session I26:
DataPower-MQ
Connectivity Deep Dive
(Theory)
Speaker: Robin Wiley
Room 27
Session Z1: WebSphere
MQ for z/OS V8: Latest
Features Deep Dive
Speaker: Damon Cross
Room 6
9:00 - 10:00 10:30 - 11:30 11:45 - 12:45 14:00 - 15:00 15:15 - 16:15 16:45 - 17:45
Wednesday
Session Z5:
WebSphere MQ for
z/OS: Security
Speaker: Damon
Cross
Room 02
Session A21: What's
New in IBM
Messaging
Speaker: Morag
Hughson
Room 8
Session C7:
Messaging in the
Cloud with IBM MQ
Light and IBM
Bluemix
Speaker: Rob
Nicholson
Room 27
Session A17: Managing
work-loads, scaling and
availability with IBM MQ
clusters
Speaker: David Ware
Room 6
Lab IL5: DataPower-MQ Connectivity Deep Dive
(Hands-On)
Speaker: Robin Wiley
Room 7b
Session A9: WebSphere
MQ for z/OS: The Inside
Story
Speaker: Damon Cross
Room 6
Thursday
Session A35: How to
Develop Responsive
Applications with IBM
MQ Light
Speaker: Rob
Nicholson
Room 27
Session A22: New
IBM MQ V8 Security
Features
Speaker: Morag
Hughson
Room 01
Session A3: WebSphere
MQ for z/OS: Shared
Queues
Speaker: Alex Ross
Room 6
Session A18: Using
Publish /Subscribe with
IBM MQ
Speaker: David Ware
Room 27
Friday
Lab AL6: Developing a First Application
with IBM WebSphere MQ Light
Speakers: Robert Nicholson, Alex Ross
Room 7b
Session A16: Using
IBM MQ Pub/Sub in
an MQ network
Speaker: David Ware
Room 6

29. Questions?
© 2014 IBM Corporation 50