1. 11/28/2017
1
TECHNOLOGY
DEVELOPMENT: WHAT IS
THE AUDITOR'S ROLE?
NOVEMBER 29, 2017
About Jim Kaplan, CIA, CFE
 President and Founder of AuditNet®,
the global resource for auditors (now
available on iOS, Android and Windows
devices)
 Auditor, Web Site Guru,
 Internet for Auditors Pioneer
 Recipient of the IIA’s 2007 Bradford
Cadmus Memorial Award.
 Author of “The Auditor’s Guide to
Internet Resources” 2nd Edition
Page 2

2. 11/28/2017
2
ABOUT AUDITNET® LLC
• AuditNet®, the global resource for auditors, serves the global audit
community as the primary resource for Web-based auditing content. As the first online
audit portal, AuditNet® has been at the forefront of websites dedicated to promoting the
use of audit technology.
• Available on the Web, iPad, iPhone, Windows and Android devices and
features:
• Over 2,700 Reusable Templates, Audit Programs, Questionnaires, and
Control Matrices
• Webinars focusing on fraud, data analytics, IT audit, and internal audit
with free CPE for subscribers and site license users.
• Audit guides, manuals, and books on audit basics and using audit
technology
• LinkedIn Networking Groups
• Monthly Newsletters with Expert Guest Columnists
• Surveys on timely topics for internal auditors
Introductions
Page 3
HOUSEKEEPING
This webinar and its material are the property of AuditNet® and its Webinar partners.
Unauthorized usage or recording of this webinar or any of its material is strictly forbidden.
• If you logged in with another individual’s confirmation email you will not receive CPE as the
confirmation login is linked to a specific individual
• This Webinar is not eligible for viewing in a group setting. You must be logged in with your
unique join link.
• We are recording the webinar and you will be provided access to that recording after the
webinar. Downloading or otherwise duplicating the webinar recording is expressly prohibited.
• If you have indicated you would like CPE you must answer the polling questions (all or
minimum required) to receive CPE per NASBA.
• If you meet the NASBA criteria for earning CPE you will receive a link via email to download
your certificate. The official email for CPE will be issued via NoReply@gensend.io and it is
important to white list this address. It is from this email that your CPE credit will be sent. There
is a processing fee to have your CPE credit regenerated post event.
• Submit questions via the chat box on your screen and we will answer them either during or at
the conclusion.
• Please complete the evaluation questionnaire to help us continuously improve our Webinars.

3. 11/28/2017
3
IMPORTANT INFORMATION
REGARDING CPE!
• SUBSCRIBERS/SITE LICENSE USERS - If you attend the Webinar and answer the polling
questions (all or minimum required) you will receive an email with the link to download your
CPE certificate. The official email for CPE will be issued via NoReply@gensend.io and it is
important to white list this address. It is from this email that your CPE credit will be sent. There
is a processing fee to have your CPE credit regenerated post event.
• NON-SUBSCRIBERS/NON-SITE LICENSE USERS - If you attend the Webinar and answer
the polling questions (all or minimum required) and requested CPE you must pay a fee to
receive your CPE. No exceptions!
• We cannot manually generate a CPE certificate as these are handled by our 3rd party provider.
We highly recommend that you work with your IT department to identify and correct any email
delivery issues prior to attending the Webinar. Issues would include blocks or spam filters in
your email system or a firewall that will redirect or not allow delivery of this email from
Gensend.io
• Anyone may register, attend and view the Webinar without fees if they opted out of receiving
CPE.
• We are not responsible for any connection, audio or other computer related issues. You must
have pop-ups enabled on you computer otherwise you will not be able to answer the polling
questions which occur approximately every 20 minutes. We suggest that if you have any
pressing issues to see to that you do so immediately after a polling question.
The views expressed by the presenters do not necessarily represent the views,
positions, or opinions of AuditNet® LLC. These materials, and the oral presentation
accompanying them, are for educational purposes only and do not constitute
accounting or legal advice or create an accountant-client relationship.
While AuditNet® makes every effort to ensure information is accurate and complete,
AuditNet® makes no representations, guarantees, or warranties as to the accuracy or
completeness of the information provided via this presentation. AuditNet® specifically
disclaims all liability for any claims or damages that may result from the information
contained in this presentation, including any websites maintained by third parties and
linked to the AuditNet® website.
Any mention of commercial products is for information only; it does not imply
recommendation or endorsement by AuditNet® LLC

4. 11/28/2017
4
TECHNOLOGY DEVELOPMENT: WHAT
IS THE AUDITOR'S ROLE?
NOVEMBER 29, 2017
Donald E Sparks
CIA, CISA, CRMA, ARM
Don@SmartCAATTs.com
407-756-0375
TODAY’S AGENDA
 What is a Technology Project
 Types of project audits
 Why projects fail
 Why projects succeed (GTAG 12)
 How the internal audit function can actively participate in
the review of projects while maintaining independence
 Outline a framework for assessing project related risks

5. 11/28/2017
5
SESSION OBJECTIVES
"IT systems development projects fail to meet expectations of management
and primary stakeholders at an alarming rate. Project challenges negatively
impact organizations, customers, financial position, and productivity.“ [The
IIA GTAG number 12]
Learn how even a non-IT auditor with a good data analysis skill set can add
value on any System Development project. This webinar will provide
auditors with key information to better facilitate performance on assurance
audits and non-audit/advisory services related to key IT systems
development projects.
These important issues will be discussed during the webinar:
•Project risks; •Participation, roles, and responsibilities •Project
management methodologies •Risk management •Phases
•Internal audit’s role •Status reporting
WHY ARE WE CONCERNED
• Technology projects are fraught with challenges
• Insufficient attention to these challenges will result in wasted
money and resources, loss of trust, and reputation damage — all
of which are huge risks and none of which is acceptable
• GTAG 12

6. 11/28/2017
6
RESOURCES
• GTAG 8
• GTAG 12
IN A 2002 INTERNAL AUDITOR ARTICLE,
RICHARD B. LANZA WROTE:
“To be successful, auditors must demonstrate to both senior
management and project managers the value that an independent
advisor can bring. Senior management can give auditors access to
projects, but auditors can be more effective when the project
managers buy into their involvement and give them greater access.”

7. 11/28/2017
7
WHAT IS A TECHNOLOGY PROJECT?
• Most system implementation or maintenance projects are
increasingly complex initiatives that involve or impact more than
just the IT department and, as such, should be considered as a
business project as well as an IT project.
• In the most general sense, a project is a unique set of activities
with a discreet beginning and end, undertaken to achieve a
particular purpose within defined constraints of schedule, scope,
and resources.
• It is important to note that this webinar is intended to focus on
projects that include a technology-related solution; however the
principles are very similar to other types of projects.
GTAG 12
THINGS HAVE CHANGED IN 20 YEARS
(IN 1995)
• 18 million American homes online, but only 3% of online users had ever signed on to the
World Wide Web.
• Amazon.com opens for business, hyping itself as “Earth’s Biggest Bookstore.”
• Craig Newmark starts craigslist, originally an email list of San Francisco events.
• Match.com, the first online dating site launches.
• Entrepreneur Pierre Omidyar launches ebay, originally named “AuctionWeb.” First item
sold: a broken laser pointer. A collector purchases it for $14.83.
• Netscape IPO starts the gold rush mentality for Web startups.
• Microsoft releases Windows 95 and the first version of Internet Explorer.
• Web hosting service GeoCities launches.
• The Dancing Baby, a 3D animation, becomes one of the first viral videos.

8. 11/28/2017
8
INFORMATION SYSTEM ACQUISITION
• In-house Development
• Acquired in merger/acquisitions
• Purchase & Customize
• Purchase Plain Vanilla
KEY TECHNOLOGY PROJECT PHASES
Top Management
Approval
Technology
Change Team
Process
Go Live Post
Development

9. 11/28/2017
9
RISK EXPOSURES – WHY PROJECTS FAIL
WHY TECHNOLOGY PROJECTS SUCCEED
1. User Involvement – Business users are involved with key
consensus-building, decision-making, and information-gathering
processes.
2. Executive Support – Executives provide alignment with business
strategy, as well as financial, emotional, and conflict resolution support.
3. Clear Business Objectives – Stakeholders understand the core
value of the project and how it aligns with business strategy.
4. Agile Optimization – Project uses iterative development and
optimization processes to avoid unnecessary features and ensure
critical features are included.
5. Emotional Maturity – Project manager directs the emotions and
actions of project stakeholders and avoids ambition, arrogance,
ignorance, abstinence, and fraudulence.

10. 11/28/2017
10
WHY TECHNOLOGY PROJECTS SUCCEED
6. Project Management Expertise – Organization uses project managers who
understand the basic skills and practices, such as certified Project Management
Professional from the Project Management Institute (PMI) or the like.
7. Financial Management – Project manager is able to manage financial
resources, account for project budget/costs, and demonstrate the value of the
project.
8. Skilled Resources – Skilled project personnel are acquired, managed,
retained, and controlled to move forward in the face of turnover and other
personnel hurdles.
9. Formal Methodology – There is a predefined set of process-based
techniques that provide a road map on when, how, and what events should
occur in what order.
10. Tools and Infrastructure – The project infrastructure is built and managed
with tools that enable management of tasks, resources, requirements, change,
risks, vendors, user acceptance, and quality management.
POLLING QUESTION #1

11. 11/28/2017
11
WHAT ROLE SHOULD INTERNAL AUDITING
PLAY?
 None
 From the outset of the planning phase – Cost Benefit, ROI
calculations
 Just the data conversion phase – test plans and data conversion
 When invited by the project development team – Seat on the
technology committee
 When trouble erupts
 Only on a post implementation basis (after go-live)
IPPF – ASSURANCES EXPECTATIONS
The role of internal audit in system development
Risk management processes are monitored through ongoing management
activities, separate evaluations, or both. (New)
2120.A1 – The internal audit activity must evaluate risk exposures:
• Achievement of the organization’s strategic objectives
• Reliability & integrity of financial & operational information
• Effectiveness & efficiency of operations & programs
• Safeguarding of assets
• Compliance with laws, regulations, policies, procedures, & contracts
2120.A2 – The internal audit activity must evaluate the potential for the
occurrence of fraud and how the organization manages fraud risk.

12. 11/28/2017
12
AUDITOR PARTICIPATION OBJECTIVE
The objective is for auditing to provide a written report along with
management response comments if available to assist management
in the go or no go decision. This meeting is before the decision to
move the newly developed system into production.
• Provide the project team with a status in each of the areas
discussed
• This will assist in the Go or No Go decision
• Audit does not have a vote (ad hoc) representation only to
preserve independence and objectivity
• Providing independent ongoing advice throughout the project.
• Identifying key risks or issues early, which enables project teams
to operate proactively to mitigate risks.
POLLING QUESTION #2

13. 11/28/2017
13
PROJECT FAILURES NOTHING TO LAUGH AT
REAL EXAMPLES OF AUDIT REPORTS
Things change. The first example was a purchased system with customization
Conducted by the company where I worked. The second example is for a client
engaged a third party vendor and is running in the cloud.
If you did not put your concerns in writing, then you did not say it

14. 11/28/2017
14
LESSONS LEARNED
• Always put in writing discussions and reporting
• Be alert that audit was included to “out gun a bigger gun”
• Audit role is that of a consultant
• Audit should recommend but not demand – escalate if necessary
• Audit should follow the audit work plan template – Cobit,
ISO21000, and other SDLC guides are for project leaders
• Be leery of others wanting to write your report
• Avoid accepting decision making roles or voting on issues
• Audit schedule time is about 10% of the total duration of the work,
i.e., 100 days to develop, plan for 10 days to review
• Be diligent when the project lead is a vendor protecting their rear
• Understand why sub-sub vendors were added
POLLING QUESTION #3

15. 11/28/2017
15
UPDATE YOUR SOX SMARTS
• The quarterly reporting requirement of material control changes
under SOX will have an impact on implementations and major
upgrades
• Control design, testing, and training must become a key part of the
implementation process
• Historically, implementation teams have not incorporated the new
skills that will be required
• Internal audit will need to become more proactive than ever before
• Involvement will differ in every phase of the implementation
lifecycle
• Common issues exist that will allow you to focus your attention on
a few key areas
• Independence can be maintained by empowering the process
teams with risk and control knowledge
Steve Biskie
RISKS NOT JUST IN CLASSIC PROJECT AREAS
Technology Data Process &
Solution
Governan
ce
Project
Mgmt
Organization
Infrastructure Data
Structure
Requirements Strategic
Align
Time
Schedule
Business
Impact
System
Architecture
Mapping Business
Processes
Top Mgmt
Buy-in
Budgets Training
Networking Cleansing SDLC Sponsor-
ship
Resources Communication
Availability Conversion Data Decision
Making
Staffing Company Align
Performance Validation Controls Issue
Tracking
Vendors Change Mgmt
Disaster
Recovery
Governance Bolt-on’s Knowledge
Share
Compliance/
Controls
Backup,
Restart
Interfaces Issue &
Risk Mgmt
Business
Continuity
BI & Reports Scope
Mgmt

16. 11/28/2017
16
WHERE SHOULD AUDIT START?
CONSISTENT & SUSTAINABLE!
Audit Program: Working
Papers:
Audit Report:
Planning Planning Planning
Testing Testing Testing
Processing Processing Processing
Training Training Training
Documentation Documentation Documentation
Security Security Security
VARIATIONS OK - ADJUST TO FIT COMPANY CULTURE
Project
Plans
Testing
Processes,
SOD, Files
Data
Conversion
Training &
Document
ation
Security Go-Live
Scoping
Control
Design
Control
Configuration
Policy/Proced.
Updates
Config.
Testing Control
TrainingRisk
Assessment
Monitoring
Walk-
throughs
Modified to Include Continuous Controls Monitoring
Traditional
Source: Sparks
Source: ISACA, IIA

17. 11/28/2017
17
POLLING QUESTION #4
PLANNING
Plans are sufficient to complete the project as approved by the
Technology Committee and Board of Directors (on time, in budget,
complete for all user expectations). There should be direct linkage
to the strategic objectives of the organization.
• Objective s formalized – what are the expectations of this
process change
• Feasibility complete and approved
• Personnel from all affected areas included on the team
• Problem reporting mechanism in place
• Project Proposal and Schedule Authorizations in place
Does the project plan include appropriate tests and criteria for
judging the completeness of the project?.

18. 11/28/2017
18
PROJECT PLAN PHASE – AUDITOR FOCUS
AREAS
• Ensure task list is complete
• Monitor status, budget, and issues
• All areas represented
• Consult on risk mitigation strategies
• Build audit & control activities into plan
• Identify (and sell) control requirements
• Assist project administration controls
WHAT CAN GO WRONG IN PLAN
• Controls building not part of the plan
• Lack of understanding where controls are needed
• Lack of communication/change management plan
• Lack of QA function
• Auditor loses independence: ok to coach, facilitate, and educate

19. 11/28/2017
19
COST BENEFIT ANALYSIS
TANGIBLE INTANGIBLE
Increase Revenue: Increased Customer Satisfaction
 Sales in existing markets Improved Employee Satisfaction
 New markets More current information
Cost Reduction: Improve decision making
 Labor Faster response to competitors
 Operating expenses More effective and efficient
 Maintenance cost Improve communications &
Improve control environment
TESTING
A test plan was maintained and agreed to by the end users before Go
Live.
• Determine a test methodology has been designed and approved by
management. Ideally three months parallel (old/new)
• Specific control totals have been identified and agreed upon for
comparison purposes.
• A procedure for dealing with differences or system problems has been
established.
• A process for handling data rejected and errors to be sure corrected and
re-input.
• An ongoing testing (monitoring) process developed to be used prior to
implementing future enhancements and other significant changes.
• A Service Level Agreement (SLA) should be starting to formulate. what
metrics will be required on a regular basis after Go-Live to provide
assurances everything is running as designed.

20. 11/28/2017
20
GATHER FACTS:
• Data Sources
• Users
• Data Stores
• Processes (HR, Payroll, etc)
• Data Flows
• Controls
• Transaction Volumes
• Error Rates
• Auditor Role – special audit ability features or embedded modules QA test process
and results
• Validate test completeness
• Design test scripts for controls
• Provide audit specific functionality
• Independently test key controls
PROCESSING
System programs will process data accurately, completely and in a timely
manner and there is clear lines of separation of conflicting duties between
input, processing and output.
Data input for processing should be validated and edited as close to the
point of origination as possible. Error handling procedures should be in
place to facilitate the timely and accurate re-submission of all corrected
data.
Separation of duties between origination of data; input of data; processing
of data; and, distribution of data should exist.
If an override authorization exists, be sure there is an automatic log is
provided and a manual review of the log is in place.

21. 11/28/2017
21
HEAVY “AUDIT” SKILLS
Processes,
Risks
Locations,
Set the Priorities
Expected
Controls
Automated
Manual or
System-Dependent
Observe &
Walkthru
Tests
Confirm
working
Monitoring
Test/advice on controls Sustainable
Not enough
or not complete
Too much or
not efficient
Controls don’t address
identified risks
Anticipated controls not set-up or
appropriately communicated
Controls not set-up correctly
Inconsistent execution
Effectiveness loss over time
Not Key
Control
----------------------------------------------------------------Looking for-----------------------------------------------------------------------
POLLING QUESTION #5

22. 11/28/2017
22
NIFTY TOOL FOR AUDITING:
{KEY PROCESS STEP}
Resources and Master Data files
Usage. Examples: Vendor Master,
Taxes, etc
Participation discussions, observations &
brainstorming session:
•Cashiers handle the same number of sales
•Cashiers receive about the same number of refunds or
returns
•Employees purchases should not receive “refunds”
•All invoices should be sequential numbered without gaps
•Refunds are computed dollars, distribution must follow
Benford’s
•An automated system should not allow duplicate refunds
•Customers not refunded more than what they paid
originally
•Supervisor override during work hours when customers
present
•Correlation link sales and refunds = refunds follow sales
trend
Input
Operati
ons
Output
Operati
ons
PROCESS - DETAIL DESIGN
• Designs for input screens and source documents - mapping
• Designs for screen outputs, reports and operational documents
• Normalizing the databases
• Data Flow Diagrams (DFD’s)
• Data Dictionary
• Processing logic including formulas

23. 11/28/2017
23
PROCESS - CONVERSION
• Converting the Databases
• Validation
• Reconciliation
• Backup
• Cold turn-key cutover
• Phased cutover
• Parallel operation cutover
• Auditors provide operational
expertise, review documentation,
verify control adequacy and
knowledge share with external
audit and regulators.
PROCESS - DATA VALIDATION AND EDITING
PROCEDURES PERFORM:
• Individual and supervisor
authorization or approval code
• Check digits on all identification
keys
• Check digits at the end of a string
of numeric data that is not
subjected to balancing
• Validity of codes
• Alphanumeric or numeric values
• Field sizes
• Combination of fields
• Limit or reasonableness of values
• Signs
• Record matches or mismatches
(records not dropped or overright)
• Sequences
• Cross-footing checks

24. 11/28/2017
24
TRAINING
All roles and duties identified and training conducted prior to Go
Live.
Training is a critical element of any system implementation. If the
users (and their backup) and help desk staff are not properly trained
the day to day operation of the system may not be performed timely,
accurately or completely. The project plan should itemize: Type of
Training; Persons to be Trained; Date of Training; and Was training
reinforced by user testing (parallel testing).
Does management of the departments above consider the training
to be sufficient?
DOCUMENTATION
System documentation is adequate and complete for end user,
operations and help desk needs. Procedures should be reviewed on a
regular basis to identify efficiencies that could be gained and controls
strengthened prior to the go-live decision.
Documentation Should include:
• User and may include desktop reminder tips
• Operations
• Programming
• Help Desk

25. 11/28/2017
25
SECURITY
Data files can be recovered and/or reconstructed in the event of a mishap.
Areas to address by project team:
• Physical access to processing equipment
• Access to data at rest or on a backup device
• Contingency Plans (Disaster Recovery)
• On-line Recovery and restart
• Production Program Source access controls
• Auditors verify original system is free from material errors and fraud
• Written backup and recovery restart processes
POLLING QUESTION #6

26. 11/28/2017
26
AUDIT OBJECTIVES IN SECURITY
• Detect any unauthorized source program changes
• Verify maintenance procedures protect applications from
unauthorized changes
• Verify applications are free from material errors
• Verify maintenance special commands are controlled
• SOD between application developers and maintenance activities
after go live.
HOW CAN DATA ANALYTICS HELP?
• Parallel Testing – A period of time where the end users maintain
the old system and the new system. Compare the results of data
input between the systems.
• Look at the completeness of data in the new system. Test data
files to determine if gaps exist such as incorrect formulas,
incorrect results or blanks in key data cells.
• Key business task overlooked – Determine that log files are
available and accessible to support the business. For example a
popular ERP for education did not provide a history log of parent
and student access to student grades. Teachers and
administrators of the schools needed access to prepare for parent
meetings.

27. 11/28/2017
27
POLLING QUESTION #7
QUESTIONS?
• Any Questions?
Don’t be Shy!
Page 54

28. 11/28/2017
28
IN THE QUEUE -
Page 55
AUDITNET® AND CRISK
ACADEMY
• If you would like forever
access to this webinar
recording
• If you are watching the
recording, and would like
to obtain CPE credit for
this webinar
• Previous AuditNet®
webinars are also
available on-demand for
CPE credit
http://criskacademy.com
http://ondemand.criskacademy.com
Use coupon code: 50OFF for a
discount on this webinar for one week

29. 11/28/2017
29
THANK YOU!
Page 57
Jim Kaplan
AuditNet® LLC
1-800-385-1625
Email:info@auditnet.org
www.auditnet.org
Don Sparks - CIA, CISA, CRMA
SmartCAATTs LLC
don@SmartCAATTs.com