Developer and Fusion Middleware 2 _Greg Kirkendall _ How Australia Post teach...
JDE & Peoplesoft 2 _ Sam Sampathnathan _ Best Practices for Managing Your JD Edwards Data.pdf
1. Best Practices for Managing JD Edwards Data:
Preparing for Upgrades, Migrations and more
Sambasivam Sampathnathan – World Wide Product Manager
IBM Software Group
08/16/2011
2. Today’s Discussion
• What
drives
upgrades
today?
• Data
management
strategies
for
your
upgrade
project
• Archiving
and
Upgrades
– Concepts
&
Benefits
– Key
Requirements
• Managing
test
data
• Client
Success
Story
• QuesEons
&
Answers
3. What Drives Upgrades Today
• Maximize
value
from
exisEng
investments
in
current
economic
condiEons:
Fusion
apps
sEll
under
development
• Changing
business
processes/requirements
• ApplicaEon
instance
consolidaEon
• Technology
–
Out
with
the
old,
in
with
the
new
– New
features;
industry
compliance
– Obsolete
hardware,
middleware
&
database
plaSorms
• Vendor-‐imposed
upgrade
deadlines
4. Common Drivers of Data Growth
• OrganizaEon
growth
• Reduce
IT
Cost
• Data
retenEon
compliance
• Too
much
of
everything
– Keeping
data
“always
available”
• Data
mulEplier
effect
5. How are Organizations Responding to Data Growth?
• Use database
Performance
partitioning
• Use database vendor
Database Size
compression
• Buy more Storage &
CPU hardware
Hardware Capacity
Forrester estimates that, on average, data repositories for large applications
grow by 50% annually (structured data).
*Source: Forrester Research, Database Archiving Remains An Important Part Of Enterprise DBMS Strategy, Q3 2007
6. What is Archiving?
• Segregate
historical
enterprise
applicaEon
data
by
age,
status,
event
or
other
criteria
• Copy
historical
records
to
a
secure
archive
• Delete
transacEon
details
from
producEon
system
• Retain
access
to
informaEon
for
query,
reporEng,
customer
service,
audit
and
discovery
requests
Production Archive
Reference Data
Archive
Historical
Historical Data
Current
7. What are the Benefits of Archiving?
• Streamline
the
upgrade
project
– Reduce
downEme
during
conversion
by
50%
or
more
– Improve
applicaEon
performance
from
old
version
to
new
version
by
reducing
OLTP
workload
• Maintain
a
superior
ownership
experience
–
ongoing
– Reduce
backup
and
disaster
recovery
Eme
by
hours
– Lower
database
maintenance
Eme
for
tasks
like
reorganizaEons,
refreshes,
clones
– Support
“compliance
readiness”
– Ensure
consistent
performance
to
Service
Level
objecEves
– Reduce
cost
of
storage
and
overall
TCO
8. Data Archive Strategy: Questions to Consider
• What
data
should
I
be
saving,
for
how
long
and
for
what
reasons?
• What
data
should
I
be
deleEng?
• How
am
I
going
to
find
the
data
when
I
need
it?
• What
do
I
do
with
the
data
when
I
no
longer
need
it?
• What
is
the
cost/benefit
analysis
to
support
an
archiving
soluEon
acquisiEon?
• What
is
the
most
appropriate
soluEon
to
meet
my
archiving
needs?
9. Archiving Solution: 4 Key Requirements
1
ApplicaEon
Intelligent
• JD
Edwards
EnterpriseOne
• Complete
Business
Object
2
Take
the
right
data
out
• Apply
Func>onal
Condi>on
Checks
• Accommodate
Unique
Business
Requirements
3
• Support
&
automate
data
reten>on
Store
it
where
you
policies
as
per
ILM
business
requirements
want
• Mul>ple
formats
–
DBMS,
File
4
Access
archived
data
when
&
how
you
want
• Na>ve
applica>on
access
• Applica>on
independent
access
10. 1 Application Intelligent
• Complete
Business
Object
– Includes
pre-‐defined
related
J.D.
Edwards
tables
– TransacEon
data
as
well
as
contextual
and
reference
data
• Integrity
Checks
• Custom
columns
are
recognized
automaEcally
• AddiEonal
custom
tables
can
be
added
visually
• SelecEon
criteria
or
integrity
checks
may
be
applied
to
any
column:
AutomaEcally
applied
to
all
related
rows.
11. 2 Archive the Right Set of Data
Extract
Production
Database
Archive Solution
1 - 2 Years
Current Data
• Access Definitions
• Complete Business Object
• Business Rules
• Validations
12. 2 Archiving a Complete Business Object
• Represents application data record – payment,
invoice, customer
– Referentially-intact subset of data across related
tables and applications; includes metadata
• Provides “historical reference snapshot” of business
activity
• Federated object support across enterprise data
stores
13. 2 Complete Business Object - Example
F0010
Company Master
F0008 F0101 F0006 F0011
Business Unit Batch Control
Fiscal Data Pattern Address Book Master (closed jobs)
F0012 F0018 F0911 F0911T
AAI’s Tax Table General Ledger GL Tag
F0025 F4008 F0902
Ledger Type Master Tax Area Account Balances
F0901 F0909
Account Master Chart of Accounts
Records in table removed from
database during archive process • Represents application data record – payment, invoice,
customer
Table captured as reference
table during archive process – Referentially-intact subset of data across related tables and
applications; includes metadata
All Records included • Provides “historical reference snapshot” of business activity
• Federated object support across enterprise data stores
14. 2 Business Rules & Validations Example
• Both
invoice
closed
date
and
receipt
GL
Date
must
fall
within
the
specified
period
• The
invoice
and
receipt
are
posted
and
all
corresponding
G/L
entries
are
posted
• All
the
invoices
must
be
paid
and
the
payment
status
must
be
equal
to
‘P’
• The
domesEc
and
foreign
open
amounts
both
must
be
equal
to
zero
• Recurrent
invoices
(doc.
type
RR)
are
only
archived
if
the
number
of
installments
is
blank
15. 3 Store It Where you Want
Compressed
Active/Historical Online
2 - 4 Years
Archives
XML
Extract
Production Archive
Database Database
On/Near-Line Archive
4 - 6 Years
Archive Solution
Non DBMS
1 - 2 Years Retention Platform
Current Data
• Access Definitions ATA File Server
• Complete Business SATA / FATA disk
Object
• Business Rules 6+ Years
Off-Line Archive
Off-line Retention Platform
• Validations CD,Tape,Optical, WORM
HP StorageWorks™, IBM TSM
NetApp NearStore® SnapLock™,
IBM Information Archive/TSM
EMC Centera™.
16. 3 Support ILM Business Requirement
FuncEonal
Usage
/
Access
Requirements
Over
Time
ExcepEon-‐based,
Infrequent,
ApplicaEon-‐
Frequent,
Ad-‐Hoc,
Independent
ApplicaEon-‐ Query-‐based
Access
Complete
DeleEon
Based
Access
(Self-‐
(24-‐hour
IT
(Dictates
storage
Business
Object
Access
Help)
response)
planning)
Journals
(GL)
Current
–
2Y
Years
3
–
5
Years
6
-‐
10
Year
11
Invoices
(AP)
Current
–
2Y
Years
3
–
5
Years
6
-‐
10
Year
11
TransacEons
(AR)
Current
–
2Y
Years
3
–
5
Years
6
-‐
10
Year
11
Purch.
Orders
(PO)
Current
Year
Year
2
Years
3
-‐
10
Year
11
Sales
Order
(SO)
Current
–
2Y
Years
3
–
5
Years
6
-‐
10
Year
11
Fixed
Assets
(FA)
Current
Year
Year
2
Years
3
-‐
10
Year
11
17. 4 Access Archived Data
Active/Historical Online
Compressed
2 - 4 Years
Archives
XML U
N
Extract I Native Application
Archive Access
V
Production Database
E
Database Restore
R
On/Near-Line Archive
S
4 - 6 Years
1-2 Archive Solution Non DBMS A IBM Mashups
Years Retention Platform L
Current Data
• Access Definitions ATA File Server
• Complete Business EMC Centera™, DR550, Etc.
A
Object C
• Business Rules C
6+ Years
Off-Line Archive
Off-line Retention Platform
• Validations CD,Tape,Optical, WORM E Additional Options
HP StorageWorks™, IBM TSM S ODBC / JDBC
NetApp NearStore® SnapLock™, S XML
IBM Total Storage® solutions SQL
(including the DR550)
EMC Centera™. Excel / Access
18. 4 Access Archived Data Thru Reporting Tools
Reporting Solution Options
• Report Writers:
• Cognos
• Business Objects
• Discoverer
• Mashup Center Archive
• Access
• Any SQL-based tool set
• Browser
• Java Application
• Open & independent
• No training of end users or audit staff
• Leverage existing tools and skills
• OLTP not required
19. Where Does Archiving Fit?
1.
Archive
before
an
Upgrade
– Reduces
amount
of
data
to
convert
during
producEon
cutover
– Reduces
downEme
during
upgrade
2.
Archive
in
Parallel
with
an
Upgrade
– Combines
common
technical
and
funcEonal
tasks
to
save
overall
elapsed
Eme
• Technical
Tasks:
Validate
individual
steps
(such
as
moving
query
tables
forward),
validaEng
new
technical
environment,
performance
tesEng
• FuncEonal/Business
Tasks:
Reconciling
“before”
&
“aner”
results,
regression
tesEng
on
key
business
processes
(such
as
payroll
processing
or
financial
close)
20. 1. Archive Before Upgrade
Older Version Upgrade Project Upgraded Version
Application Archive
Data
Current Data Current Data
Application Archives Application Archives
Universal Access to Application Data
Application ODBC / JDBC XML Report Writer
21. 2. Archive & Upgrade Projects in Parallel
Older Version Upgraded Version
Upgrade Project
Archive
Application
Data Archive Planning Current Data
Application Application Archives
Universal Access to Application Data
Application ODBC / JDBC XML Report Writer
22. Data Multiplier Effect
Dev
Test Back-up
1 TB
UA
Disaster
1 TB 1 TB Recovery
ERP/CRM
1 TB Application
Dev
Back-up Dev
Test Back- Test
up
UA UA
Master Data Data Disaster
Disaster Warehouses
Recovery Recovery
The Actual Data Burden = Size of all production database + all replicated clones
According to Forrester, on average, data repositories for large applications
grows by 50% annually (structured data)!
23. Creation of Test Environment: Current Practices
#1 - Clone Production #2 – Write SQL
Clone Production
Request for Copy Write SQL • Complex
Extract • Subject to Change
Wait
After
Changes
Production Production After
Extract
Database Database
Copy Copy
Changes • RI Accuracy? Expensive,
• Right Data? Dedicated Staff,
Manual examination: Ongoing
Right data? Responsibility
What Changed?
Correct results? Share test database
Unintended Result? with everyone else
Someone else modify?
24. Effective Test Data Management
Production or Production Clone
Relational subset
Extract Load / Mask
Insert / Update
2TB Compare
25 GB
25 GB Development
• Create targeted, right-sized test
environments
Unit Test
• Substitute sensitive data with fictionalized yet
contextually accurate data
• Easily refresh, reset and maintain test
environments 50 GB
100 GB
• Compare data to pinpoint and resolve
application defects faster Training
Integration
• Accelerate release schedules Test
25. Testing Best Practices - Oracle
Tip #27—Test with a Representative Subset of Production Data
“When performing the development upgrade, it is important to
leverage a representative subset of production data instead
of an exact copy; this is because the development environment
usually has less capacity in both memory and hard drive space
than the test and production environments. Limiting the size of
the conversion files during the development upgrade will better
ensure that the processes will complete in a timely manner.”
26. Data in Test & Development Environments
• 62%
of
companies
surveyed
use
actual
customer
data
instead
of
disguised
data
to
test
applicaEons
during
the
development
process
– 50%
of
respondents
have
no
way
of
knowing
if
the
data
used
in
tesEng
had
been
compromised.
• 52%
of
respondents
outsourced
applicaEon
tesEng
– 49%
shared
live
data
Source: The Ponemon Institute. The Insecurity of Test Data: The Unseen Crisis
27. Mask Data in Non-Production Environments
• Also
known
as:
data
de-‐idenEficaEon,
depersonalizaEon,
desensiEzaEon,
obfuscaEon,
data
scrubbing
• Technology
that
helps
conceal
real
data
• Scrambles
data
to
create
new,
legible
data
• Retains
the
data's
properEes,
such
as
its
width,
type
and
format
• Common
data
masking
algorithms
include
random,
substring,
concatenaEon,
date
aging
• Used
in
non-‐producEon
environments
as
a
Best
PracEce
to
protect
sensiEve
data
28. De-identify Data Without Impacting Dev & Test
• Mask
or
de-‐idenEfy
sensiEve
data
elements
that
could
be
used
to
idenEfy
an
individual
• Ensure
masked
data
is
contextually
appropriate
to
the
data
it
replaced,
so
as
not
to
impede
tesEng
• Data
is
realisEc
but
ficEonal
• Masked
data
is
within
permissible
range
of
values
• Support
referenEal
integrity
of
the
masked
data
elements
to
prevent
errors
in
tesEng
Personal identifiable
information is masked
with realistic but fictional
data for testing &
development purposes.
JASON MICHAELS ROBERT SMITH
29. Why Retire or Consolidate Applications
• Redundant
systems
acquired
via
mergers
and
acquisiEons
• Line
of
business
divested
• Legacy
technologies
not
compaEble
with
current
IT
direcEon
• Required
technical
skills
or
applicaEon
knowledge
no
longer
available
• Budget
pressures
–
do
more
with
less!
30. Retire Unused Legacy Applications
• Consolidate
mulEple
applicaEons
into
a
single
instance
and
reEre
unused
applicaEons
– Move
from
home
grown
to
packaged
system
– Consolidate
similar
systems
due
to
mergers
and
acquisiEons
• Consolidate
an
independent
business
process
with
others
– Move
automaEon
capabiliEes
into
a
single
system
and
reEre
independent
applicaEon
• Move
applicaEon
from
an
old
to
new
architecture
– Not
all
data
is
relevant
for
the
move,
but
it
must
be
retained
• Shut
down
legacy
system
without
a
replacement
In
almost
ALL
cases,
access
to
legacy
data
MUST
be
retained
while
the
applica<on
and
database
are
eliminated
31. Retire the Application, Not the Data
Archiving allows you to move the data needed while
maintaining access to the original data in its business-
object form without the original application.
Benefits
– Reduce
IT
infrastructure
costs
(hardware,
sonware,
labor
costs)
– Reduce
infrastructure
complexity
(eliminate
confusion)
– Reclaim
assets
32. About IBM Optim
• Proven
leader
in
Integrated
Data
Management
(IDM):
– Manage
and
Control
Data
Growth
– Data
RetenEon,
Compliance
&
Discovery
– Speed
ApplicaEon
Delivery
&
Quality
with
Test
Data
Management
– Speed
ApplicaEon
Upgrades
&
MigraEons
– ApplicaEon
ReErement
– Improve
Storage
Management
–
ILM
– Improve
ApplicaEon
Performance
and
SLAs
• Solving
complex
data
management
issues
since
1989
• Global
company:
2500
clients;
50%
of
Fortune
500
• Recognized
by
Gartner,
IDC,
META
as
EDM
industry
leader
with
46%
market
share.
33. IBM Optim™ Solutions
• Op>m
Data
Growth
Solu>on
(Archiving)
– Improve
performance
– Control
data
growth,
save
storage
– Support
retenEon
compliance
– Enable
applicaEon
reErement
– Streamline
upgrades
• Op>m
Test
Data
Management
Solu>on
– Create
targeted,
right
sized
test
environments
– Improve
applicaEon
quality
– Speed
iteraEve
tesEng
processes
• Op>m
Data
Privacy
Solu>on
– Mask
confidenEal
data
– Comply
with
privacy
policies
34. IBM Optim: Enterprise Architecture
Discovery
Test Data Management Data Privacy Data Growth Application Retirement
Organization environments are diverse, yet interrelated; therefore what you use to manage the data MUST support across
your environment
35. Success Story: Application Upgrade & Cost
Challenge Business Benefits
§ The upgrade and data conversion was
§ Leveraging their Oracle packaged- completed over a 3-day weekend, eliminating
application for manufacturing, shipping and the need for downtime during business hours.
finance. Since installing in 2003, the system § Archived data is still accessible to functional
has grown tremendously, from 300 GB to users via an archive database, so there was
more than 1 TB of information. minimal training needed.
§ Calculated it would take 5 to 7 business § Archived data was stored on “tier 2” storage.
days to perform the upgrade and data This provided an estimated annual savings of
conversion. That amount of downtime for a $75,000-$80,000 in storage costs.
business critical application would be § Archiving project was completed 3 weeks
unacceptable. ahead of schedule, allowing for additional
preparation time for the upgrade project itself.
“Without IBM Optim software, it would have taken
us nearly twice as long to upgrade our ERP
Solution application, forcing us to shut down
manufacturing. Instead, we completed the
• IBM InfoSphere Optim Data Growth Solution process in one three-day weekend without any
business interruption.”
36. Other Examples
Objective: Objective: Objective: Objective:
Minimize business Need to reduce Need to reduce the Minimize business
disruption during JD projected system database size of enterprise disruption during JD
Edwards upgrade and downtime for JD system, and manage the Edwards upgrade and
improve system Edwards upgrade from 8 ongoing data growth for control 25% annual data
performance to support days to 3 days to avoid financial and supply chain growth & corresponding
business operations business disruption information storage costs
Solution: Solution: Solution: Solution:
Optim Data Growth Optim Data Growth Optim Data Growth Optim Data Growth
Solution for JD Edwards Solution for JD Edwards Solution for JD Edwards Solution for JD Edwards
EnterpriseOne EnterpriseOne EnterpriseOne EnterpriseOne
Reduced the volume of Completed upgrade and Reduced the database size Attained a 50% reduction in
audit trail & workflow data conversion over a 3- & implemented data growth their needed future
activity by 63%, day weekend, eliminating strategy, creating a 50% purchases of data storage,
shortening data downtime during business reduction in system streamlined the data
conversion window and hours; stored archive data maintenance and 300% migration process during
improving system on “tier 2” storage, saving improvement in query times the upgrade project, and
performance. $80K annually in storage reduced the time needed
costs for ongoing maintenance
39. What’s the Take-Away?
• Define
your
InformaEon
Lifecycle
Management
(ILM)
requirements
– What
needs
to
be
archived
– How
long
you
need
to
hold
on
to
them
• “We’re
an
IT
shop
delivering
business
value”
• Today’s
IT
strategies
should
provide
an
immediate
payback
AND
set
the
stage
for
future
innovaEon
– Understand
your
data
– Manage
data
growth
via
archiving
– Use
subsets
for
test
data
creaEon
– Mask
data
in
non-‐producEon
environments
– ReEre/consolidate
legacy/unused
applicaEons
40. In Closing…..
• You
are
most
welcome
to
join
me
at
the
IBM
Booth
aner
this
session
to
discuss
this
presentaEon
or
your
specific
quesEons
or
requirements
• We’d
appreciate
if
you
can
complete
the
evaluaEon
form
on
your
seat
and
deposit
in
the
box
at
the
IBM
Booth
….
you’ll
also
have
a
chance
to
win
one
of
3
iPads!
• All
IBM
Summit
presentaEon
sessions
are
noted
in
the
flyer
on
your
seat
to
help
plan
your
parEcipaEon
…
we’d
love
to
see
you
at
some
more
of
our
sessions!
• Break
Free
at
our
next
IBM
event
…
see
the
invitaEon
envelope
on
your
seat
for
details.
42. Disclaimers
IBM
customers
are
responsible
for
ensuring
their
own
compliance
with
legal
requirements.
It
is
the
customer's
sole
responsibility
to
obtain
advice
of
competent
legal
counsel
as
to
the
iden>fica>on
and
interpreta>on
of
any
relevant
laws
and
regulatory
requirements
that
may
affect
the
customer's
business
and
any
ac>ons
the
customer
may
need
to
take
to
comply
with
such
laws.
IBM
does
not
provide
legal
advice
or
represent
or
warrant
that
its
services
or
products
will
ensure
that
the
customer
is
in
compliance
with
any
law.
The
informa>on
contained
in
this
documenta>on
is
provided
for
informa>onal
purposes
only.
While
efforts
were
made
to
verify
the
completeness
and
accuracy
of
the
informa>on
provided,
it
is
provided
“as
is”
without
warranty
of
any
kind,
express
or
implied.
IBM
shall
not
be
responsible
for
any
damages
arising
out
of
the
use
of,
or
otherwise
related
to,
this
documenta>on
or
any
other
documenta>on.
Nothing
contained
in
this
documenta>on
is
intended
to,
nor
shall
have
the
effect
of,
crea>ng
any
warran>es
or
representa>ons
from
IBM
(or
its
suppliers
or
licensors),
or
altering
the
terms
and
condi>ons
of
the
applicable
license
agreement
governing
the
use
of
IBM
so`ware.