1. Effective 2/20/2015
CEDAR Proposal
Last printed 2/20/2015 8:02:00 PM
CIP Compliance Proposal
CEDAR Technology Strategy and Roadmap
Prepared By:
Michael Yu
Mike McWethy
Stephen Corbett
Joseph Perry
Version # 2.1 Updated on 5/23/2013
2. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 2
Version 2.1
Acknowledgments
The contribution of the following individuals in preparing this document is gratefully
acknowledged:
Matt Laullen, CEO CEDAR
Role Name Phone # E-Mail Address
Owner Michael Yu
Author Mike McWethy
Contributor
Michael Yu
Mike McWethy
Stephen Corbett
Joseph Perry
Reviewer Joseph Perry
Approval
Stephen Corbett
Document Number 2.1
Document Name CEDAR Proposal
Date Created (Draft) 4/13/2013
Date Approved 5/23/2013
Location Chicago, IL
Medium of Distribution Electronic
Security Classification Confidential
Retention 1 year after the completion of the project
Archive Location somewhereimportant
3. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 3
Version 2.1
MMJS – CEDAR Proposal:
Version Control
Version Date Author Change Description
1.0 4/13/2013 Michael yu Document created
1.2 4/17/2013 Mike McWethy
Stephen Corbett
Joseph Perry
Peer review
1.3 5/04/2013 Michael Yu
Stephen Corbett
Joseph Perry
Update of CIP 1->10
1.4 5/11/2013 Michael Yu
Stephen Corbett
Joseph Perry
Update CIP2, 3, 7,10
Include CIP11
2.0 5/18/2013 Michael Yu
Mike McWethy
Stephen Corbett
Joseph Perry
Draft proposal
2.1 5/23/2013 Michael Yu
Mike McWethy
Stephen Corbett
Joseph Perry
Final version for proposal
4. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 4
Version 2.1
DOCUMENT ACCEPTANCE and RELEASE NOTICE
This is version 2.1 [0.0] of the MMJS – CEDAR Proposal.
The MMJS – CEDAR Proposal is a managed document. For identification of amendments, each
page contains a release number and a page number. Changes will be issued only as a complete
replacement document. Recipients should remove superseded versions from circulation. This
document is authorized for release after all signatures have been obtained.
Please submit all requests for changes to the owner/author of this document.
PREPARED: DATE:___/___/___
(Michael Yu, Document Owner)
ACCEPTED: DATE:___/___/___
(CEDAR, CEO)
6. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 6
Version 2.1
4.7.5 Visitor logging ....................................................................................................... 24
4.8 Cyber Security Systems Management............................................................................ 24
4.8.1 CIP007– System Management............................................................................. 24
4.8.2 Section 1............................................................................................................... 25
4.8.3 Section 2............................................................................................................... 25
4.8.4 Section 3............................................................................................................... 26
4.8.5 Section 4............................................................................................................... 27
4.8.6 Section 5............................................................................................................... 27
4.9 Incident Reporting and Response Planning.................................................................... 28
4.9.1 CIP008– Regulatory Requirements...................................................................... 28
4.9.2 Plan Specification ................................................................................................. 28
4.9.3 Plan Testing.......................................................................................................... 29
4.9.4 Plan Communication ............................................................................................ 29
4.10 Recovery Plan BES Systems Compliance.................................................................... 29
4.10.1 CIP009- Regulatory requirements........................................................................ 29
4.10.2 CEDAR Disaster Recovery Process .................................................................... 32
4.10.3 Disaster Recovery Plan – Roles and Responsibilities ......................................... 33
4.10.4 CEDAR Disaster Recovery Tier ........................................................................... 33
4.10.5 Live system recovery............................................................................................ 35
4.10.6 Data Backup ......................................................................................................... 36
4.10.7 Data De-duplication.............................................................................................. 37
4.10.8 Alerting.................................................................................................................. 38
4.10.9 Monitoring and Backup Reports ........................................................................... 38
4.10.10 Resilience Management ProgramDisaster Recover...................................... 40
4.10.11 Implementation cost analysis.......................................................................... 42
4.11 Change Management.................................................................................................... 42
4.11.1 CIP010– Regulatory Requirements...................................................................... 42
4.11.2 Change Tracking Software................................................................................... 42
4.11.3 Change Management Process............................................................................. 42
4.12 Information Protection ................................................................................................... 43
4.12.1 CIP011– Regulatory Requirements...................................................................... 43
4.12.2 Information Protection .......................................................................................... 43
4.12.3 Media Reuse and Disposal................................................................................... 43
5 APPENDICES........................................................................................................................ 44
7. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 7
Version 2.1
1 EXECUTIVE SUMMARY
[Provide a high-level overview of channel strategy to executives.]
1.1 Background
[Provide information about why this channel strategy is required.]
1.2 Objectives
[Provide objectives that need to be achieved.]
1.3 Overview
[Provide a brief overview of strategy and plan.]
2 BUSINESS OBJECTIVES
[Define business objectives and alignment with strategic objectives.]
2.1 Objective A
[Insert objective here.]
2.2 Objective B
[Insert objective here.]
3 CEDAR OVERVIEW
3.1 Overview
[Describe channel schema.]
[Insert channel schema here.]
8. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 8
Version 2.1
4 PROPOSAL
4.1 Analysis
[Insert list of all available channels here.]
[Example]
4.2 Sabotage Reporting
This report details the requirements that CEDAR must implement in order to be in full
compliance with CIP-001-2a, Sabotage Reporting. Given that CEDAR has no formal
policies in place and that previous sabotage events have occurred, it is highly
recommended that CEDAR implement these changes immediately. A cost structure for the
requirements is located herein. The aforementioned structure details estimated time
requirements, organizational impact as well as the potential monetary policies should the
Executive Committee choose to forego / ignore these requirements.
When assembling the formal policy for Critical Infrastructure Protection -001, management
as well as legal counsel must have an active role. In addition, all personnel should
participate in an annual meeting whereby they are to acknowledge and sign a document
indicating that they have read and understand the sabotage awareness policy.
4.2.1 CIP001– Financial Concerns
Given the current heightened state of awareness to acts of terrorism, the penalties for
failing to comply with any areas of sabotage reporting are costly. Further, the fines accrue
on a daily basis and at a minimum are considered moderate. In order to put that into
perspective, a violation severity level that is classified as moderate, with a violation risk
factor classified as medium will cost $100,000 per day as long as an organization remains
out of compliance. However, the majority of penalties that exist in CIP-001 are considered
“high to severe” and carry far more aggressive fines. Failure to comply carries significant
penalties and the time as well as the organizational impact in order to achieve compliance
is minimal.
There is no capital outlay for the purchase of equipment or other materials in order to
achieve compliance with CIP-001. Further, the organizational impact will be minimal. The
only requirements are drafting policies as well as informing and educating all of the
employees at CEDAR. Drafting the policies should be done by management and legal
counsel in conjunction. Once the policies have been finalized, the time required to train /
inform CEDAR personnel should be minimal.
4.2.2 Policy Requirements
There are a total of four requirements that CEDAR will be audited / measured upon
in order to determine compliance.
1. CEDAR must have a written, well documented policy in place that includes a
detailed procedure designed to train all of its employees in the recognition of a sabotage
event which affects CEDAR as well as other areas of the interconnection. In laymen’s
terms: “If you see something, say something.”
The policy must include:
9. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 9
Version 2.1
a. Procedure for the recognition of a sabotage event.
b. Procedure for the recognition of a sabotage event that will affect other areas of
the interconnection, i.e., facilities that are not owned and operated by CEDAR.
c. Procedures that detail the steps for educating personnel on what constitutes a
sabotage event on CEDAR owned facilities as well as other areas of the
interconnection.
i. The compliance auditor will require a written narrative that compliance
has been accomplished.
ii. The compliance auditor will require the name of the file, file extension,
revisions, dates, sections, policy authors and their titles, section titles as
well as a description.
2. CEDAR must have a written, well documented policy in place that includes
procedures for the communication of information with regard to a sabotage event to
appropriate parties in the interconnection.
The policy must include:
a. A documented procedure for the communication of information with regard to a
sabotage event to the appropriate parties in the interconnection.
b. Current contact information for the “appropriate parties” of the interconnection.
i. The compliance auditor will require a written narrative of how this
requirement is met. The auditor wants evidence that this has been
accomplished.
ii. The compliance auditor will require the name of the file, file extension,
revisions, dates, sections, policy authors and their positions, section titles
as well as a description.
c. The term “appropriate parties” is defined as: “entities with whom the reporting
party has responsibilities and/or obligations for the communication of physical or
cyber security event information.”
3. CEDAR must provide operating personnel with sabotage response guidelines.
This policy must include:
a. Sabotage response procedures and guidelines are distributed to operating
personnel. Operating personnel include, but are not limited to, field personnel.
b. Guidelines may be distributed during safety meetings, training sessions, e-mail or
a combination of the above. It is recommended that more than one method is
used to distribute guidelines to all personnel. Safety meetings are an ideal
distribution opportunity as all personnel are required to attend. E-mail also
affords the luxury of a read / received receipt.
c. Response guidelines must include personnel to contact for reporting an event.
i. The compliance auditor will require a written narrative of how this
requirement is met. The auditor wants evidence that this has been
accomplished.
ii. The compliance auditor will require the name of the file, file extension,
revisions, dates, sections, policy authors and their positions, section titles
as well as a description.
iii. The compliance auditor will also utilize an operator interview to
determine how versed the operating personnel are with regard to
sabotage response reporting.
10. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 10
Version 2.1
iv. The compliance auditor will also check that response guidelines are
posted in the control room of CEDAR facilities.
1. The guidelines may be available in either a posted hard copy or
electronically in the control room.
4. CEDAR shall establish communications / contact information with local Federal
Bureau of Investigations officials in order to develop reporting procedures with regard to a
sabotage event.
This policy must include:
a. Current contact information to the local FBI office including address, phone
number(s) and e-mail address(es).
b. Procedures for reporting sabotage to the FBI.
i. The compliance auditor will require a written narrative of how this
requirement is met. The auditor wants evidence that this has been
accomplished.
ii. The compliance auditor will require the name of the file, file extension,
revisions, dates, sections, policy authors and their positions, section titles
as well as a description.
4.3 BES Cyber System Categorization
Identify and categorize Bulk Electric Systems (BES) Cyber Systems and their associated
BES Cyber Assets for the application of cyber security requirements corresponding with
the adverse impact that loss, compromise, or misuse of those systems could have on the
reliable operation of the BES. Systems are categorized based on their impact on the BES
systems and are classified as High Impact, Medium Impact, or Low Impact.
4.3.1 CIP002– Regulatory Requirements
A. Control Centers and backup Control Centers, Transmission stations and substations,
Generation resources, Systems and facilities critical to system restoration, including
Blackstart Resources and Cranking Paths and initial switching requirements, Special
Protection Systems that support the reliable operation of the Bulk Electric System, and
for Distribution Providers/Protection Systems must be identified as either High, Medium,
or Low Impact BES Cyber Systems
B. At least once every 15 calendar months, the identifications of the assets as described
above must be reviewed and/or updated and must be approved by the CIP Senior
Manager or delegate
4.3.2 Implementation
Dated electronic records or physical lists that exist within a Document Management
System (DMS) contain the asset inventory and BES Cyber System Categorization. It is
proposed to CEDAR to use PowerDMS as their Document Management System.
PowerDMS provides document authoring, review and approval workflows, document
lifecycle management, document versioning, employee testing capabilities, proof of
compliance, change management notifications, and report building.
Materials List:
http://www.powerdms.com/compliance-management-software-solutions/policy-and-
procedure-management-software.aspx
11. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 11
Version 2.1
4.4 Security Management Controls
4.4.1 CIP003– Regulatory Requirements
Establish responsibility and accountability to protect BES Cyber Systems against
compromise that could lead to misoperation or instability in the BES through consistent and
sustainable security management controls
4.4.2 Requirements
A. One or more documented cyber security policies that collectively address
Personnel & training, Electronic Security Perimeters including Interactive Remote
Access, Physical security of BES Cyber Systems, System security management,
Incident reporting and response planning, Recovery plans for BES Cyber Systems,
Configuration change management and vulnerability assessments, Information
protection, and Declaring and responding to CIP Exceptional Circumstances for each
High Impact and Medium Impact Asset and they must be reviewed and approved by CIP
Senior Manager once every 15 months
B. Document cyber security policies that collectively address Cyber security awareness,
Physical security controls; Electronic access controls for external routable protocol
connections and Dial-up Connectivity, and Incident response to a Cyber Security
Incident.
C. Identify a CIP Senior Manager by name and document any change within 30 calendar
days of change
D. Documented process to delegate authority unless no delegations are used. Where
allowed by the CIP Standards, the CIP Senior Manager may delegate authority for
specific actions to a delegate or delegates. These delegations shall be documented,
including the name or title of the delegate, the specific actions delegated, and the date of
the delegation; approved by the CIP Senior Manager; and updated within 30 days of any
change to the delegation. Delegation changes do not need to be reinstated with a
change to the delegator.
4.4.3 Implementation
Materials List:
http://www.assetpoint.com/industries-cmms-electrical-generation.htm
4.5 Personnel and Training
Training personnel knowledgeable in BES Cyber security is critical for compliance,
operational efficiency, security and risk standpoint. Lack of training can have immense
impact to the brand of CEDAR in the power generation and distribution market. Today’s
electrical energy distribution consists of highly complex interdependent systems.
There are many treats to BES Cyber Security Systems. There are potential insider crime
from disgruntled worker (including contractors) to carless or poorly trained employee that
may introduce malware or accidently change systems without proper training. CEDAR
must have established process for documenting personnel training. This section will
explore different CIP compliance requirements and recommend tools and processes to
mitigate risk and cost to CEDAR
4.5.1 CIP004– Regulatory Requirements
12. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 12
Version 2.1
Personnel and Training
Parts Physical Access
Control Systems
(PACS)
Requirement Measurement
1.1 R1 M1 H, M Quarterly personnel training for employees and
Third Party contractors who has access to
electronic or physical access to BES system
Recorded action of requirements
2.1 R2 M2 H, M 2.1.1. Cyber security policies
2.1.2. Physical access controls
2.1.3. Electronic access controls
2.1.4. The visitor control program
2.1.5. Handling of BES Cyber System Information
and its storage
2.1.6. Identification of a Cyber Security Incident
and initial notifications in accordance with the
entity’s incident response plan
2.1.7. Recovery plans for BES Cyber Systems
2.1.8. Response to Cyber Security Incidents
2.1.9. Cyber security risks associated with a BES
Cyber System’s electronic interconnectivity and
interoperability with other Cyber Assets.
Evidence may include but are not limited
to, training material such as power point
presentations, instructor notes, student
notes, handouts, or other
training materials.
2.2 H, M Require completion of the training specified in
Part 2.1 prior to granting authorized electronic
access and
authorized unescorted physical access to
applicable Cyber Assets, except during CIP
Exceptional Circumstances.
Examples of evidence may include, but
are not limited to, training records and
documentation of when CIP Exceptional
Circumstances were invoked.
2.3 H, M Require completion of the training specified in
Part 2.1 at least once every 15 calendar months.
Examples of evidence may include, but
are not limited to, training records and
documentation of when CIP Exceptional
Circumstances were invoked.
3.1 R3 M3 H, M Process to confirm identity. An example of evidence may include, but
is not limited to, documentation of the
Responsible Entity’s process to confirm
identity
3.2 H, M Process to perform a seven year criminal history
records check as part of each personnel risk
assessment that includes:
3.2.1. Current residence, regardless of duration;
and
3.2.2. other locations where, during the seven
years immediately prior to the date of the
criminal history
records check, the subject has resided for six
consecutive months or more
An example of evidence may include, but
is not limited to, documentation of the
Responsible Entity’s process to perform a
seven year criminal history records check
3.3 H, M Criteria or process to evaluate criminal history
records checks for authorizing access
An example of evidence may include, but
is not limited to, documentation of the
Responsible Entity’s process to evaluate
criminal history records checks.
3.4 H, M Criteria or process for verifying that personnel
risk assessments performed for contractors or
An example of evidence may include, but
is not limited to, documentation of the
13. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 13
Version 2.1
service vendors are conducted according to Parts
3.1 through
3.3.
Responsible Entity’s criteria or process for
verifying contractors or service vendors
personnel risk assessments.
3.5 H, M Process to ensure that individuals with
authorized electronic or authorized unescorted
physical access have had a personnel risk
assessment completed according to Parts 3.1 to
3.4 within the last seven years.
An example of evidence may include, but
is not limited to, documentation of the
Responsible Entity’s process for ensuring
that individuals with authorized electronic
or authorized unescorted physical access
have had a personnel risk assessment
completed within the last seven years.
4.1 R4 M4 H, M Process to authorize based on need, as
determined by the Responsible Entity, except for
CIP Exceptional
Circumstances:
4.1.1. Electronic access;
4.1.2. Unescorted physical access into a Physical
Security Perimeter; and
4.1.3. Access to designated storage locations,
whether physical or electronic, for BES Cyber
System Information.
An example of evidence may include, but
is not limited to, dated documentation of
the process t authorize electronic access,
unescorted physical accessing a Physical
Security Perimeter, and access to
designated storage locations, whether
physical or electronic, for BES Cyber
System Information.
4.2 H, M Verify at least once each calendar quarter that
individuals with active electronic access or
unescorted physical access have authorization
records.
Examples of evidence may include, but
are not limited to:
∙ Dated documentation of the verification
between the system generated list of
individuals who have been authorized for
access (i.e., workflow database) and a
system generated list of personnel who
have access(i.e., user account listing), or
∙ Dated documentation of the verification
between a list of individuals who have
been authorized for access(i.e.,
authorization forms) and a list of
individuals provisioned for access(i.e.,
provisioning forms or shared account
listing).
4.3 H, M For electronic access, verify at least once every
15 calendar months that all user accounts, user
account groups, or user role categories, and their
specific,
associated privileges are correct and are those
that the Responsible Entity determines are
necessary
An example of evidence may include, but
is not limited to, documentation of the
review that includes all of the following:
1. A dated listing of all accounts/account
groups or roles within the system;
2. A summary description of privileges
associated with each group or role;
3. Accounts assigned to the group or role;
and
4. Dated evidence showing verification of
the privileges forth group are authorized
and appropriate to the work function
performed by people assigned to each
account
4.4 H, M Verify at least once every 15 calendar months
that access to the designated storage locations
for BES Cyber System Information, whether
An example of evidence may include, but
is not limited to, the documentation of
the review that includes all of the
14. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 14
Version 2.1
physical or electronic, are correct and are those
that the Responsible Entity determines are
necessary for performing assigned work
functions.
following:
1. A dated listing of authorizations for BES
Cyber System information;
2. Any privileges associated with the
authorizations; and
3. Dated evidence showing a verification
of the authorizations and any privileges
were confirmed correct and the minimum
necessary for performing assigned work
functions.
5.1 H, M A process to initiate removal of an individual’s
ability for unescorted physical access and
Interactive Remote Access upon a termination
action, and complete the removals within 24
hours of the termination action (Removal of the
ability for access may be different
than deletion, disabling, revocation, or removal
of all access rights).
An example of evidence may include, but
is not limited to, documentation of all of
the following:
1. Dated workflow or sign‐off form
verifying access removal associated with
the termination action; and
2. Logs or other demonstration showing
such pe
5.2 R5 M5 H, M For reassignments or transfers, revoke the
individual’s authorized electronic access to
individual accounts and authorized unescorted
physical access that the Responsible Entity
determines are not necessary by the end of the
next calendar day following the date
that the Responsible Entity determines that the
individual no longer requires retention of that
access.
An example of evidence may include, but
is not limited to, documentation of all of
the following:
1. Dated workflow or sign‐off form
showing a review of logical and physical
access; and
2. Logs or other demonstration showing
such persons no longer have access that
the Responsible Entity determines is not
necessary.
5.3 H, M For termination actions, revoke the
individual’s access to the designated
storage locations for BES Cyber System
Information, whether physical or
electronic (unless already revoked
according to Requirement R5.1), by the
end of the next calendar day following
the effective date of the termination
action.
An example of evidence may include,
but is not limited to, workflow or sign‐
off form verifying access removal to
designated physical areas or cyber
systems containing BES Cyber System
Information associated with the
terminations and dated within the next
calendar day of the termination action.
5.4 H For termination actions, revoke the
individual’s non‐shared user accounts
(unless already revoked according to
Parts 5.1 or 5.3) within 30 calendar
days of the effective date of the
termination action.
An example of evidence may include,
but is not limited to, workflow or sign‐
off form showing access removal for
any individual BES Cyber Assets and
software applications as determined
necessary to completing the revocation
of access and dated within thirty
calendar days of the termination
actions.
15. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 15
Version 2.1
5.5 H For termination actions, change
passwords for shared account(s) known
to the user within 30 calendar days of
the termination action. For
reassignments or transfers, change
passwords for shared account(s) known
to the user within 30 calendar days
following the date that the Responsible
Entity determines that the individual no
longer requires retention of that
access. If the Responsible Entity determines
and documents that extenuating
operating circumstances require a
longer time period, change the
password(s) within 10 calendar days
following the end of the operating
circumstances.
Examples of evidence may include, but
are not limited to:
∙ Workflow or sign‐off form
showing password reset within
30 calendar days of the
termination;
∙ Workflow or sign‐off form
showing password reset within
30 calendar days of the
reassignments or transfers; or
∙ Documentation of the
extenuating operating
circumstance and workflow or
sign‐off form showing password
reset within 10 calendar days
following the end of the
operating circumstance.
4.5.2 Employee Background Check
During the employee screening process, the selected candidate will have their last seven
years of criminal background check. CEDAR has selected the services of Intellicorp
(http://www.intellicorp.net/marketing/home.aspx) to screen potential employees following
CIP-004 Part 3.2->3.5 guideline. Any employee of CEDAR must be able to pass criminal
background check regardless to level of cyber asset category.
All contractors must have their criminal background check validated by their respective
companies. They must show certificate of background check indicating their employees
have gone through similar background check and show no risk to CEDAR.
4.5.3 Training
The CEDAR Learning and Development (L&D) methodology will consist of online or
classroom training. New hire employees are required to conduct a through training for
systems which they are responsible for. The employees will be trained on part 2.1 of the
CIP-004 guidelines using CEDAR new hire onboarding process. All new employees who
require access to high and medium cyber assets, as part of onboarding process, will be
trained on two day CIP Compliance Foundations Training. CEDAR has partnered with
EnergySec (http://www.energysec.org/) to provide in house training. Hiring manager will
be responsible for scheduling the new employee in the monthly in-house training.
Employees must be trained as part of orientation program on this foundations training.
CEDAR will be receiving a discount at $200/employee. Below is the agenda that will be
covered within the training. Testing will be conducted and each employee must pass the
final exam before they are allowed to work on high and medium cyber security assets at
CEDAR.
Topics
Unit 1: Terminology 101
Unit 2: What Are We Trying to Protect? (CIP-002)
Unit 3: Security Perimeters - Logical and Physical (CIP-005 and CIP-006)
Unit 4: Consolodating Efforts to Save Time and Money (CIP-008 and CIP-009; CIP-007
R1, R1 and CIP-003; CIP-007 R2, R8 and CIP-005 R4)
Unit 5: Inventory for Success; Hardware, Software, People (CIP-002, CIP-004, CIP-005,
CIP-007)
16. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 16
Version 2.1
Unit 6: Policies, Procedures and Processes (CIP-002 through CIP-009)
Unit 7: Technical Feasibility Exceptions
Unit 8: Useful Open Source Security Tools (CIP-005/ CIP-007)
Unit 9: Compliance and Security Crystal Ball
Contractors and any third party providers, who must access high and medium cyber
security assets, must show baseline understanding of the CIP requirements before they
are given access. If the contractors and third party providers do not require access to
Cyber Security System, then they can obtain escort badge to access non critical asset
areas. Upon completion of the required training either through a instructor-led or the online
training system, their records will update. The in house developed access security system
called cACCESS will automatically allow employees access to roles to either electronic
control or physically access critical cyber system. Each major cyber system access will be
managed by a supervisor who will be alerted of the training and access requirements.
They will validate the training and approve access to those systems. Employees will be
given 30 day reminder of the training through Cedar L&D. If the employees and
contractors do not completed the required training, a reminder will be sent to the employee
and the group manager within 5 days of expiration. Any employees who do not complete
the required training will automatically be removed from electronic group and physical
access until training is completed. Exception override can be made through senior HR
lead due to extended vacation or personal circumstances.
Continued L&D
Each employee and third party contractors will log on to CEDAR L&D to identify
themselves and their training progress. Certification of completion will be tracked as part
of employee records. No employees or contactors will have access to any cyber assets
with High or Medium category unless they’ve been properly certified. For some critical
systems the employee may be required to demonstrate their skills through either simulation
system or tested by senior trainer. Employees and third party contractors must perform
quarterly training before there are allowed access to electronic or physical access to BES
system. Subcontract companies must provide certificate of training before their employees
are allowed access. The system supervisor will grant access upon the validation by the
contract company of certified trainee. Statement of work must include that all
subcontractors will be CIP compliant and trained.
4.5.4 Physical Access Software
Physical access to all CEDAR will be managed by Lenel security products onGuard and
goEntry 3.0 (http://www.lenel.com). Lenel has open architecture for security access
decoupling the physical access hardware from the software access controls. Each major
office will have security desk for guest and employee access control. Security guard will
also be posted in any shipping and receiving areas. Access control to non security guard
access points will have ID card access security with random digital pin pad. Each
employee who requires access to these entry points will be given personal secret unique
pin. Employees by policy are not allowed to load out their ID cards or give out their pins.
17. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 17
Version 2.1
4.5.5 ElectronicPhysical Authentication and Access
CEDAR directory services for user authentication is Microsoft Active Directory (AD).
Access will be granted based on role groups. Each group will have a supervisor owner
who is given ownership. The group must be reviewed every 15 months to audit and
validate the users in the group. Any new employee or employees transferred out will be
reviewed and removed if access is no longer required. Supervisor will also determined if
the employee conducted proper training to keep access to the role. The security groups in
AD are synced with cAccess that automatically sync with Lenel onGuard system. Any
employees who are removed from the security group will automatically be removed for
security physical access. The access rules will have special rules when fire is detected in
the facility for fast exit of employees. Special case will also be enabled for fire and weather
related drills.
4.5.6 Employee Termination
Any employees or contractor termination will be entered in cACCESS. Employee manager
or supervisor will notify HR. HR will request a termination of employee via cACCESS.
Employee’s AD account will be disabled and access to physical access will also be
terminated. Employee accounts will be removed automatically after 30 days.
4.6 Electronic Security Perimeter
This report details the tools and recommendations that CEDAR must implement in order to
be in full compliance with CIP-005-5, the secure electronic perimeter. The electronic
security perimeter is a significant portion of any defense in depth strategy. It is also one of
the first areas to come under attack. The tools recommended to secure the perimeter are
18. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 18
Version 2.1
discussed in the following sections. An overview of additional considerations is included as
well. It is highly recommended that CEDAR undertake the necessary measures to
implement a secure electronic perimeter immediately.
4.6.1 CIP005– Perimeter Concerns
The devices specified in this section have been selected for their ability to perform their
intended tasks very well. However, that was not the only criteria taken into consideration.
Ease of network integration, reliability, how familiar network security and network
administrators are with the underlying systems has also been factored in. A projected
breakdown follows.
Estimated total outlay for devices specified in the this section $616,000
Estimated setup hours / network integration time. However, each device must undergo
testing before it can be placed in the production environment. The device testing time can
exceed 35 days with software updates to the machines and attempts to minimize
configuration conflicts. The time required to test the devices alone dictates that these
solutions be implemented immediately.
4.6.2 Protecting the Perimeter
In order to protect the BES cyber assets classified as “high” and “medium” as well as their
associated protected cyber assets that are specified in section 2, the purchase of several
security appliances are necessary. “Netwitness” is a highly regarded tool with a trusted
track record used for monitoring and investigating network activities. “Netwitness” is
capable of analyzing, detecting and monitoring every packet that travels across the
network. “Netwitness” will monitor every individual traffic flow on the network. Further, it
includes report generating and alert capabilities. These abilities allow for detecting and
tracking insider threats as well as an external network breach should one occur. The
downside to implementing “Netwitness” is the involved installation time and the cost
associated with each unit. Given the network segregation detailed in section 2, multiple
units need to be purchased for each network located in CEDAR and the backup network.
The “Netwitness” machines have to be stacked and run in a serial fashion. This is
necessary in order to have a near instant recovery time should one of the units cease to
function properly.
1. Price per unit $49,999 (This unit price quoted is from 2012.)
2. Estimated number of units required 8.
3. Total cost outlay $400,000
The Cisco ASA 5585 – X firewall and the SSP20 Intrusion Prevention System have been
chosen to secure the connection between CEDAR and any external entity. The 5585 – X
will form the outer and inner perimeter of the DMZ. These devices have been chosen for
several reasons. The first is the consistently high reviews they receive. Also, writing and
integrating firewall rules is a relatively easy process for cisco devices. In addition, most
security professionals are familiar with Cisco IOS, which can lead to a faster integration
time. The ASA 5585 – X is also capable of supporting a 10Gb link with the appropriate I/O
module. The ASA 5585 -X firewall provides room to support an expanding network without
needing to be replaced and an integrated intrusion prevention system. As with the
“Netwitness” devices, the ASA 5585 firewalls will have to be stacked and configured to run
in serial in order to maintain a secure perimeter. Should one device fail, the other can take
over immediately.
1. Price per unit $48,600
2. Two are required per external connection, an additional 2 are required to close off the
DMZ.
19. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 19
Version 2.1
3. Estimated cost outlay $200,000
The Cisco 5515-X has been chosen as the firewall to further segregate the internal
networks. The aforementioned reasons for choosing the 5585 –X apply to the 5515 as
well: familiarity with the operating systems, the potential ease of integration, creation of
new rules and so forth. However, the maximum stateful inspection throughput that the
ASA 5515-X is capable of supporting is 1.2Gbps. Given that these firewalls are being used
to further secure internal operations, this is not an issue. The placement of these firewalls
should further isolate the separate internal networks containing devices classified as “high
and medium” BES cyber assets as described in section 2.
Another matter warrants consideration as the electronic security perimeter is being
discussed. The majority of individuals carry some form of smart phone with them. Further,
there are organizations that have adopted “bring your own device” policies. No outside
devices should be allowed in the CEDAR environment. Further, smartphones that have
open physical ports and cameras should not be allowed.
In order to allow the secure connectivity of the field technician’s laptops, the Barracuda 480
SSL VPN appliance was selected. This device is to be placed in the DMZ to add another
layer of security. This device serves as an intermediate system so that the technicians
avoid directly accessing an applicable cyber asset. A username and password are
required when the technicians access the device to gain intermediate network access to
the DMZ. In addition, the username and password that the technicians use to access the
device must include a random unique identifier localized to the technician trying to gain
access. Minimum password guidelines must be incorporated into the CEDAR username /
password policy as specified in Section 7. The Barracuda 480 device was selected for
multiple reasons. We wanted to avoid relying too heavily on one organizations technology
(Cisco). The 480 SSL VPN device supports multiple forms of encryption as well as
hardware token authentication. There is also an integrated audit log feature. With regard to
the VPN device, split tunneling is not to be allowed, remote desktop connections are not to
be allowed, nor telnet. The device is to be configured to allow only the absolute minimum
access needed by the technicians. In addition, technicians that have logged into CEDARs
network and remained inactive for a period of 15 minutes shall be disconnected. The
Barracuda 480 VPN device should be configured to prevent any forms of synthetic
connection “keep alive” efforts. Given the devices ability to work with active directory,
maintaining strict access permissions should be easily accomplished.
Price per unit is $4,000.
Required units = 1 per DMZ where the field technicians dial in.
Estimated cost outlay $8,000.
Estimated daily penalty $100,000.
Given that there are many new security related threats that are discovered on a daily basis
and that it is virtually impossible to maintain a static environment, an annual penetration
test should take place. This test should be conducted by responsible individuals from a
reputable firm that have experience working with sensitive assets. While CEDAR needs to
be aware of any security vulnerabilities that exist, it should be made clear to the
penetration testers that the utmost care is to be used when testing the environment. The
firm that has been recommended to conduct the test is KPMG. KPMG is recommended
due to preexisting relationships with individuals employed at KPMG and the strong
reputation of the firm. However, given that the penetration testing field has become
commodity oriented, any reputable firm should suffice. Another point that requires
attention, regardless of the chosen firm, the individuals that perform the test must all sign
20. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 20
Version 2.1
non-disclosure agreements stating that they will not discuss CEDAR’s environment. The
cost associated with the penetration test exists on a sliding scale.
Materials List:
http://www.emc.com/security/rsa-netwitness.htm
http://www.cdw.com/shop/products/Cisco-ASA-5585-X-Integrated-Edition-SSP-20-and-
IPS-SSP-20-Bundle-security/2912607.aspx
http://www.ctistore.com/catalog/cat/prod,541751.html?gclid=CL3fqbywoLcCFYFhMgodOU
UAWg
http://www.barracudastore.com/barracuda-ssl-vpn-
380.html?gclid=CIial7i9oLcCFexcMgoddFoAwQ
4.7 Physical Security of BES Cyber System
Physical security is critical in a large engineered electrical grid system. A wide variety of
motives exists to attack against power grid from economic, to pranks, all the way to
terrorism. A smart grid system has capability of reaching every single home. It is vital that
systems from the power generation to networks distribution must be protected. This
section discusses the physical security of BES cyber system. Physical security must deter
potential intruders, distinguish authorized and unauthorized personnel, delay physical
attack, detect intrusion and trigger a response. Various proposals below will provide
CEDAR with roadmap to secure access to its facilities and protect cyber assets.
4.7.1 CIP006– Regulatory Requirements
Part Physical Access Control
Systems(PACS)
Requirement Measurement
1.1 R1 M1 M, H Define operational or procedural controls to
restrict physical access.
An example of evidence may include, but is not limited
to, documentation
that operational or procedural controls exist.
1.2 M Utilize at least one physical access control to
allow unescorted physical access into each
applicable Physical Security Perimeter to
only those individuals who have authorized
unescorted physical access.
An example of evidence may include, but is not limited
to, language in the
physical security plan that describes each Physical
Security Perimeter and
how unescorted physical access is controlled by one or
more different
methods and proof that unescorted physical access is
restricted to only
authorized individuals, such as a list of authorized
individuals accompanied by access logs.
1.3 H Where technically feasible, utilize two or
more different physical access controls(this
does not require two completely
independent physical access control
systems)to collectively allow unescorted
physical access into Physical Security
Perimeters to only those individuals who
have authorized unescorted physical access.
An example of evidence may include, but is not limited
to, language in the
physical security plan that describes the Physical
Security Perimeters and
how unescorted physical access is controlled by two or
more different
methods and proof that unescorted physical access is
restricted to only
authorized individuals, such as a list of authorized
individuals accompanied by access logs.
21. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 21
Version 2.1
1.4 H Monitor for unauthorized access through a
physical access point into a Physical Security
Perimeter.
An example of evidence may include, but is not limited
to, documentation of controls that monitor for
unauthorized access through a physical access point
into a Physical Security Perimeter.
1.5 M, H Issue an alarm or alert in response to
detected unauthorized access through a
physical access point into a Physical Security
Perimeter to the personnel identified in the
BES Cyber Security Incident response plan
within 15 minutes of detection.
alarm or alert in response to unauthorized access
through a physical access control into a Physical
Security Perimeter and additional evidence that the
alarm or alert was issued and communicated as
identified in the BES Cyber Security Incident Response
Plan, such as manual or electronic alarm or alert logs,
cell phone or pager logs, or other evidence that
documents that the alarm or alert was generated and
communicated.
1.6 M, H Monitor each Physical Access Control
System for unauthorized physical access to a
Physical Access Control System.
An example of evidence may include, but is not limited
to, documentation of controls that monitor for
unauthorized physical access to a PACS.
1.7 M, H Issue an alarm or alert in response to
detected unauthorized physical access to a
Physical Access Control System to the
personnel identified in the BES Cyber
Security Incident response plan within
15minutes of the detection.
alarm or alert in response to unauthorized physical
access to Physical Access Control Systems and
additional evidence that the alarm or alerts was issued
and communicated as identified in the BES Cyber
Security Incident Response Plan, such as alarm or alert
logs, cell phone or pager logs, or other evidence that
the alarm or alert was generated and communicated.
1.8 M, H Log (through automated means or by
personnel who control entry) entry of each
individual with authorized unescorted
physical access into each Physical Security
Perimeter, with information to identify the
individual and date and time of entry.
logging and recording of physical entry into each
Physical Security Perimeter and additional evidence to
demonstrate that this logging has
been implemented, such as logs of physical access into
Physical Security
Perimeters that show the individual and the date and
time of entry into
Physical Security Perimeter
1.9 M, H Retain physical access logs of entry of
individuals with authorized unescorted
physical access into each Physical Security
Perimeter for at least ninety calendar days.
Dated documentation such as logs of physical access
into Physical Security
Perimeters that show the date and time of entry into
Physical Security
Perimeter.
2.1 R2 M2 M, H Require continuous escorted access of
visitors(individuals who are provided access
but are not authorized for unescorted
physical access) within each Physical
Security Perimeter, except during CIP
Exceptional Circumstances.
Language in a visitor control program that requires
continuous escorted access of visitors within Physical
Security Perimeters and additional evidence to
demonstrate that the process was implemented, such
as visitor logs.
2.2 M, H Require manual or automated logging of
visitor entry into and exit from the Physical
Security Perimeter that includes date and
time of the initial entry and last exit, the
visitor’s name, and the name of an individual
point of contact responsible for the visitor,
except during CIP Exceptional
Circumstances.
Language in a visitor control program that requires
continuous escorted access of visitors within Physical
Security Perimeters and additional evidence to
demonstrate that the process was implemented, such
as dated visitor logs that include the required
information.
22. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 22
Version 2.1
2.3 M, H Retain visitor logs for at least ninety
calendar days.
An example of evidence may include, but is not limited
to, documentation showing logs have been retained for
at least ninety calendar days.
3.1 R3 M3 M, H Maintenance and testing of each Physical
Access Control System and
locally mounted hardware or devices at the
Physical Security perimeter at least once
every 24 calendar months to ensure they
function properly.
Maintenance and testing program that provides for
testing each Physical Access Control System and locally
mounted hardware or devices associated with each
applicable Physical Security Perimeter at least once
every 24 calendar months and additional evidence to
demonstrate that this testing was done, such as dated
maintenance records, or other documentation showing
testing and maintenance has been performed on each
applicable device or system at least once every 24
calendar months.
4.7.2 Physical access policy
Access to CEDAR controlled facilities are stated in the policies. Any unauthorized
personal who have not been properly trained will not be allowed access to cyber systems.
As discussed on the Personnel and Training section, each employee and contractors must
be authorized access to cyber system, either physically or electronically. The supervisors
and manager owners of the various technology must grant access via cACCESS.
4.7.3 Physical Security and Monitoring
Each of the CEDAR facilities will have security desk as an entry point. The security guard
will be subcontracted from Sonitrol (http://sonitrolwc.com/company-info/) a Chicago based
company specializing in security system, policies, process, and technology implementation
company. Each security desk will have at least two security guards during the business
day to handle guest access. The security guard will have access to web based tool for
those entering the facility along with closed circuit monitor system for key entry points.
Sample view of the screen is below. Employees and contractors will access via kiosks.
Employee will access through their ID card with imbedded chip that identifies the individual.
The ID card will be utilized in non security manned access points but will require pin
access through a key pad. All entry access will be kept in electronic logs that will be
backup for a year. Sonitrol will utilize Lenel software to monitor and track employee
access. Any forced entry will be invested by the security guard within 15 minutes of the
event. The violation will be investigated and logged in the security log. As part of the
contract agreement, Sonitrol will perform physical security test and access control every
quarter to determine any maintenance requirement or potential gaps in the security.
23. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 23
Version 2.1
Access to physical access will be authorized through the employee cAccess (CEDAR in
house developed application). The web-based tool is used to manage the user access
integrating physical and electronic access. Upon the completion of new hire orientation
training, the employee is given access to the general office areas. Employees upon
training through CEDAR L&D system will be given additional access based on area
supervisor leads. The supervisors will request access through the cAccess system.
cAccess is integrated with onGuard and goEntry to automatically allow access to defined
secure access areas. Supervisors will be allowed to grant access to only the areas their
control. Any employees who require access can request access through cAccess system.
24. Effecti
Versio
4.7.4
4.7.5
4.8
4.8.1
ive 05/23/2013
on 2.1
Automate
All access p
Closed circu
generate ran
failed attemp
attempts, aft
employee or
access failu
Visitor log
Security gua
the visitor na
visit. All visi
the visitor.
desk to sign
the visit will
Cyber Sec
This focus o
many aspec
security pat
The focus o
compliance
CIP007–
The devices
intended tas
Ease of netw
administrato
breakdown f
Estimated to
Estimated se
testing befor
exceed 35 d
ed Alert Syst
point without a
uit monitoring
ndom numbe
pts will send a
ter investigati
r contractor w
re.
gging
ards and ship
ame, data of e
itors will be g
Each visitor b
out. The sec
be kept by th
curity Syst
of CIP-007 is
cts to it. Som
tch managem
of this report d
e.
System Man
s specified in t
sks very well.
work integrati
ors are with th
follows.
otal outlay for
etup hours / n
re it can be p
days with softw
MMJS –
tem
a security des
system will a
r which the pe
a silent alert f
ion will be log
will be contact
pingreceiving
entry, entry ti
iven “Escort r
badge numbe
curity guard w
e security co
tems Mana
Cyber Secur
me of the more
ment, audit tra
describes the
nagement
this section h
However, th
on, reliability,
he underlying
r devices spec
network integ
laced in the p
ware updates
– CEDAR Propo
sk will have a
also monitor th
erson with on
for the securit
gged and sent
ted for follow
g will log any
me, exit time
required” bad
er is logged a
will allow man
mpany for 90
agement
rity Systems M
e prominent a
il and malicio
recommenda
ave been sel
at was not th
, how familiar
systems has
cified in the th
ration time. H
production env
s to the mach
osal
keypad syste
hese entry po
nly the ID card
ty guard to in
t to security o
up to determ
visitors. The
, employee sp
ge and must
nd they must
nual exist from
0 days.
Management.
areas include
ous software p
ations and too
ected for thei
e only criteria
r network secu
also been fa
his section $6
However, eac
vironment. T
ines and atte
em with ID Ca
oints. The ke
d holder will k
vestigate. An
officer for follo
ine the root c
e security pers
ponsor, and r
have an emp
t exit through
m the facility.
. This is a wid
but are not li
prevention ma
ols required to
ir ability to pe
a taken into c
urity and netw
actored in. A
616,000.00
ch device mus
The device tes
empts to minim
Page 2
ard scanner.
ypad will
know. Three
ny failed
ow up. The
cause of the
sonnel will log
reason for
ployee guiding
the security
The logs of
de area with
mited to:
anagement.
o achieve
erform their
onsideration.
work
projected
st undergo
sting time can
mize
24
g
g
n
25. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 25
Version 2.1
configuration conflicts. The time required to test the devices alone dictates that these
solutions be implemented immediately.
4.8.2 Section 1
1. “Nmap” port scanner and network identification tool shall be used to identify all logical
open ports located on any asset classified as a “high” or “medium” impact BES cyber
system as detailed in section 2. In addition, “EACMS, PACS and PCA” as classified in
section 2 that are associated with the aforementioned BES cyber systems shall undergo
the same logical port scan. Any ports that are found to be open that are not necessary
for normal business operations are to be closed immediately. Windows firewall on
workstations allows administrators to close ports that are not deemed necessary. Cisco
IOS also contains the ability to shutdown ports that are located on a network. Should the
device not contain the ability to close the ports, they are deemed “necessary” for
operation under CIP requirements. An added benefit of choosing “Nmap” is the ability to
perform an entire network scan. This will further assist in asset inventory allowing the
discovery of “overlooked” workstations in the network topology.
a. Cost in dollars: 0.00, Nmap is distributed free of charge.
b. Evidence that this has been completed as required by the auditor. Configuration
of host based firewalls can be used to satisfy this requirement. In addition, output
from “netstat” can be shown to auditors.
c. It is recommended that network / port scans take place outside of normal
business hours. It is further recommended that the IT staff be on hand should a
potential issue arise.
2. Physical port lock and blocks are to disable access to devices classified as “high impact”
BES cyber systems. “Medium impact” BES cyber systems located at control centers are
also subject to this. Given that there are approximately 300 physical devices, each
containing an average of 3 ports, a total of 900 devices is needed. Each device costs
$15.00. Total cost: $4,500.00.
a. CEDAR can display the above devices in order to demonstrate compliance to an
auditor. A purchase invoice may also be used.
4.8.3 Section 2
A patch management process is required for tracking, evaluating and installing cyber
security patches on systems classified as “high” or “medium BES Cyber Systems as
detailed in section 2 as well as the associated EACMS, PACS and PCS” devices. It is
expected that the security analysts as well as network engineers / administrators
maintain a current knowledge base on newly discovered vulnerabilities that effects
software deployed in CEDAR. The newly discovered vulnerabilities are to be tested on a
virtual environment that directly mimics the production environment. The virtual
environment shall be located on CEDAR’s backup network in the Waukegan facility.
Multiple “VMware Sphere,” instances (price available upon request) as well as 230
licenses of “VMware Workstation” (250.00 US) are required for the virtualization
environment. Once the virtual environment has been configured, “Solarwinds, Patch
Manager” will be used to deploy and track patches in an orderly fashion.
a. Documentation of the patch management process may be provided to the auditor
in order to satisfy this requirement. The “Patch Manager” application also
contains a module that supports detailed logging that will aid in this aspect.
26. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 26
Version 2.1
The patches that have been deployed in the virtual testing environment are to be
monitored regularly by the security analysts as well as the network administrators.
However, every 35 days the security analysts and network administrators are to perform
a detailed evaluation and determine the suitability of the patches for deployment in the
production environment.
a. Previous evaluations may be provided to the auditor in order to satisfy this
requirement.
After the 35 day testing and evaluation procedure concludes, the applicable patches are
to be applied or a plan to mitigate the vulnerabilities shall be implemented. It is highly
recommended that “Patch Manager” be used to distribute the patches to the applicable
systems in a staggered format. It is not wise or recommended that all of the systems are
patched at the same time.
a. Compliance records and deployment information from “Patch Manager” can be
shown to the auditor to satisfy this area.
“Patch Manager” also contains an area that will allow the network administrators and the
security analysts to choose a future date to address mitigation plans that may have been
deemed necessary in section 2.3. “Patch Manager” provides easy to use scheduling
software to deploy future mitigation solutions and issues reminders in order to ensure
these solutions are met. This feature will ensure that the requirements under section 2.4
are met. Further, the dates that patches are scheduled to be deployed can also be
adjusted if a “CIP delegate / Senior Manager” approves. The logs and records of
implemented mitigation plans from “Patch Manager” can be used to satisfy auditor
inquiries.
a. “Patch Manager” pricing begins at 3000.00 US and can escalate based on
additional modules / options that are included.
4.8.4 Section 3
This section pertains to those systems that are classified as “High and Medium” BES
cyber systems as well as the associated EACMS, PACS and PCA. While there are
many different choices available for malware detection / prevention, the primary concern
that continued to arise was ease of system integration and overall performance
degradation. However, timely updates also played a factor in determining which solution
to undertake. Due the above concerns, it is recommended that “Microsoft Security
Essentials” be used to deter, detect and prevent the propagation of malicious code on
the workstations running Windows. It should be made clear that updates are not to be
installed on any workstations until they have undergone the testing in the virtual
environment. The new detection signatures should then be deployed with the “Patch
Manager” program. “CLamAV “has been selected as the most appropriate program to
protect against malware in the UNIX environments (SCADA control systems). The
“ClamAV” updates are subject to the same testing procedures on the virtual
environment. However, a network administrator that is familiar with the Unix
environment on the SCADA systems may only complete the update process.
i. ClamAV is an open source virus detection application. The program has no
cost associated with it.
ii. Microsoft Security Essentials is free to use as well.
27. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 27
Version 2.1
iii.
Documentation of deployment can serve as evidence of compliance. In addition,
written records of the malicious code response process will also serve as evidence. The
Logs generated through “Patch Manager” as well as a written log for updates to the
SCADA systems may be used.
4.8.5 Section 4
“Solar Winds, Log and Event Manager” has been selected for its ability to monitor large
numbers of versatile machines on a network as well as generate alerts. The “Log and
Event Manager” program easily satisfies the requirements of maintaining and generating
logs for successful logins, unsuccessful login attempts and malicious code detection. In
addition, the “Log and Event Manager” can generate alerts that inform security personnel
and network administrators when malicious code has been detected or a device is in a
“failure” state. Further, the data in the logs is easily displayed in report form that can be
customized based on user input, such as a summarization of logged events over the last
30 days. The administrator may specify how long data is to be retained or removed.
a. Pricing for “Log and Event Manager” starts at $4500.00.
b. In order to show evidence that the above security procedures are in place,
system generated listings of security events may be provided. Documentation of
the event log process may also be displayed showing the amount of time that
logs are to be retained. Displaying log data is an area where “Log and Event
Manager” excels. One of the prominent advertising points on the web page
directly states “the ability to quickly generate reports for NERC CIP compliance.”
4.8.6 Section 5
Active directory with Kerberos is the preferred solution to enforce authentication and
control user access for systems running Windows. The Unix systems are to follow the
same recommendations as the Windows systems; the only difference is that the Unix
environment will not be managed through active directory. The implementation of active
directory shall also be used to identify and manage shared account access. For every
“High and Medium BES Cyber asset as well as the associated EACMS, PACS and PCA
all of the default accounts associated with the devices / workstations must be disabled.
No generic / default accounts of any type are to remain on a BES cyber asset. Generic
accounts may include, but are not limited to: default accounts from the equipment
manufacturer, system name, group of system names and location. The security analysts
shall eliminate all of the aforementioned accounts. In addition, all of the default
passwords must be changed as well. Individual users must have unique user names that
contain letters as well as numbers. The passwords are required to be “complex.” That
is, they must contain letters, numbers and symbols. Further, the passwords must not be
derived from user information, must be longer than 8 characters and changed every thirty
days. Once a password has been changed, it may not be changed again for a period of
24 hours. User passwords also have to be significantly different than their last 2
passwords. In order to monitor the number of unsuccessful login attempts as well as
generate alerts for the threshold exceeding unsuccessful login attempts, active directory
will interact with “Log and Event Manager” from SolarWinds. While it is preferred that
password only devices are not acquired, the password on the devices that do offer
password only authentication must be changed every 15 months.
28. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 28
Version 2.1
a. The “Log and Event” manager can assist with providing the majority of
documentation in order to demonstrate compliance with an audit. However, system
manuals and records of password change procedures can also be used to demonstrate
compliance.
Materials List:
http://www.kensington.com/kensington/us/us/p/1645/K67718US/usb-port-lock-with-
blockers.aspx
http://www.solarwinds.com/patch-manager/patch-
management.aspx#Patch%20Compliance%20Reporting
http://www.solarwinds.com/log-event-manager/log-analysis-event-management.aspx
http://www.clamav.net/lang/en/
http://nmap.org/
http://windows.microsoft.com/en-us/windows/security-essentials-download
4.9 Incident Reporting and Response Planning
4.9.1 CIP008– Regulatory Requirements
CIP-008 outlines proper procedures for incident reporting and response. This document
outlines the minimum requirements for CEDAR and is adapted from CIP-008-5 from
NERC. This policy also outlines tools used for incident reporting.
4.9.2 Plan Specification
1. Implementation of an intrusion detection system for monitoring computer network
traffic for potential threats to the infrastructure.
2. The IDS system will have rules in place to detect abnormal traffic.
a. Rules will be properly documented in order to determine any suspicious
traffic outside of normal system operation.
b. Rules will be based on SNORT and will be customized for CEDAR traffic
patterns.
3. The IDS system will be monitored by authorized and trained personnel.
4. All potential threats will be documented with date and time following established
procedures.
5. Documentation shall be made through MS System Center so it can be linked to any
needed change controls.
6. Thresholds will be set in place to determine what incidents will need to be reported to
the Electricity Sector Information Sharing and Analysis Center (ES-ISAC).
7. A Cyber Security Incident response group will be created to address all incidents and
testing.
a. Group will be made up of the CSO, CIO and network security personnel.
b. The group shall meet on a monthly basis and in the event a threat is
detected.
8. Incident response procedures shall be created describing proper procedures for
response processes and incident handling.
29. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 29
Version 2.1
4.9.3 Plan Testing
1. In the event no incidents are reported, CEDAR shall conduct a test of the incident
response plans at least once every twelve (12) months.
2. Approved testing methods are:
a. Response to actual reportable cyber-security incident.
b. Paper drill or tabletop exercise of a cyber-security incident.
c. Operational exercise of cyber security incident.
3. Proper evidence of testing must be documented containing date of incident test, lessons
learned, test summary, logs and communications from the test.
4. Records of testing or incident must be maintained for a period of three (3) years.
4.9.4 Plan Communication
1. Communication of test or actual incident response plan must be completed within 90
calendar days after response.
2. If changes are made to roles or responsibilities these must be documented within 60
calendar days of change being made.
3. Communication must include at a minimum dated documentation of lessons learned,
detailed meeting notes, incident response plans.
4. Communication shall be made via email, mail service or electronic distribution system.
a. Proper logs must be kept showing distribution of results.
4.10 Recovery Plan BES Systems Compliance
System recovery is critical task for CEDAR to recovery from hardware and software
failures. Critical electric controls system to back end office application servers for both
physical and virtual systems must have backup and recovery plan to address CIP009
regulatory compliance requirements. Network configuration, file, SharePoint, database, or
plant system configuration all must be backed up for recovery in case of DR or failure.
Real-time backup and recovery platform of local and remote systems must be cost
effective that performs seamlessly with little administrative overhead and training to
recover critical system. Backup and recovery must be executed within minutes to reduce
downtime and effects of a disaster. Some critical client workstations must also be backed
up. The backup recovery system must be able to scale out by adding incrementally low
cost hardware without the need to upgrade to more expensive and newer hardware (scale
up). The backup platform must be able to backup heterogeneous operating environment
backup and increase operational efficiency of administering backup failures with robust
alerting features that integrates with existing incident management system. The backup
and recovery platform must be hardware agnostic for plug and play scalability for changing
technologies while reducing TCO for lifecycle of the equipment.
Speed of recoverability is one of the key components of a recovery plan. System backup
and restore must be performed quickly in the event of failure to bring them back to
operations. Backup job scheduling must be managed within central console and alerted to
appropriate system administrator for root cause analysis and job rescheduling.
Below are the regulatory requirements of NERC’s CIP compliance for recovery plan.
4.10.1 CIP009- Regulatory requirements
30. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 30
Version 2.1
Recovery Plans for BES Cyber Systems
Part Physical Access
Control Systems
(PACS)
Requirement Measurement
1.1 R1 M1 M, H Conditions for activation of the
recovery plan(s)
An example of evidence may include, but is
not limited to, one or more plans that
include language identifying conditions for
activation of the recovery plan(s).
1.2 M, H Roles and responsibilities of
responders.
An example of evidence may include, but is
not limited to, one or more recovery plans
that include language identifying the roles
and
responsibilities of responders
1.3 M, H One or more processes for the backup and
storage of information required to recover
BES Cyber System functionality.
An example of evidence may include, but is
not limited to, documentation of specific
processes for the backup and storage of
information required to recover BES Cyber
System functionality
1.4 M, H One or more processes to verify the
successful completion of the backup
processes in Part 1.3 and to address any
backup failures.
An example of evidence may include, but is
not limited to, logs, workflow or other
documentation confirming that the backup
process completed successfully and backup
failures, if any, were address
1.5 M, H One or more processes to preserve
data, per Cyber Asset capability, for
determining the cause of a Cyber
Security Incident that triggers
activation of the recovery plan(s).
Data preservation should not impede or
restrict recovery.
An example of evidence may include, but is
not limited to, procedures to preserve data,
such as preserving a corrupted drive or
making a data mirror of the system before
proceeding with recovery.
2.1 R2 M2 M, H Test each of the recovery plans
referenced in Requirement R1 at least once
every 15 calendar months:
‐ By recovering from an actual
incident;
‐ With a paper drill or tabletop
exercise; or
‐ With an operational exercise.
An example of evidence may include, but is
not limited to, dated evidence of a test(by
recovering from an actual incident, with a
paper drill or tabletop exercise, or with an
operational exercise) of the recovery plan at
least once every 15 calendar months. For
the paper drill or full operational exercise,
evidence may include meeting notices,
minutes, or other records of exercise
findings.
2.2 M, H Test a representative sample of
information used to recover BES Cyber
System functionality at least once every 15
calendar months to ensure that the
information is useable and is compatible
with current configurations. An actual
recovery that incorporates the information
used to recover BES
Cyber System functionality substitutes for
this test.
An example of evidence may include,
but is not limited to, operational logs
or test results with criteria for testing
the usability (e.g. sample tape load,
browsing tape contents) and
compatibility with current system
configurations(e.g. manual or
automated comparison checkpoints
between backup media contents and
current configuration).
31. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 31
Version 2.1
2.3 M, H Test each of the recovery plans
referenced in Requirement R1 at least once
every 36 calendar months through an
operational exercise of the recovery plans
in an environment representative of the
production environment.
An actual recovery response may
substitute for an operational exercise.
Examples of evidence may include, but
are not limited to, dated documentation of:
‐ An operational exercise at least once every
36 calendar months between exercises, that
demonstrates recovery in a representative
environment; or
‐ An actual recovery response that occurred
within the 36 calendar month timeframe
that exercised the recovery plans.
3.1 M, H No later than 90 calendar days after
completion of a recovery plan test or
actual recovery: 3.1.1. Document any
lessons learned associated with a recovery
plan test or actual recovery or
document the absence of any
lessons learned;
3.1.2. Update the recovery plan based
on any documented lessons
learned associated with the
plan; and
3.1.3. Notify each person or group
with a defined role in the
recovery plan of the updates to
the recovery plan based on any
documented lessons learned.
An example of evidence may include, but is
not limited to, all of the following:
1. Dated documentation of identified
deficiencies or lessons learned for each
recovery plan test or actual incident
recovery
or dated documentation stating there were
no lessons learned;
2. Dated and revised recovery plan showing
any changes based on the lessons learned;
and
3. Evidence of plan update distribution
including, but not limited to:
‐ Emails;
‐ USPS or other mail service;
‐ Electronic distribution system; or
‐ Training sign‐in sheets.
3.2 M, H No later than 60 calendar days after a
change to the roles or responsibilities,
responders, or technology that the
Responsible Entity determines would
impact the ability to execute the
recovery plan:
3.2.1. Update the recovery plan; and
3.2.2. Notify each person or group
with a defined role in the
recovery plan of the updates.
An example of evidence may include,
but is not limited to, all of the
following:
1. Dated and revised recovery plan with
changes to the roles or responsibilities,
responders, or technology; and
2. Evidence of plan update distribution
including, but not limited to:
‐ Emails;
‐ USPS or other mail service;
‐ Electronic distribution system; or
‐ Training sign‐in sheets.
32. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 32
Version 2.1
4.10.2 CEDAR Disaster Recovery Process
Disaster plan and recovery are the responsibilities of the individual system owners and
business application owners. In an event of disaster, the responsibility of ownership will
fall upon the Crisis Reponses Coordinator (CRC). The event will be notified to the CRC
via Service Operations Situation Manage (SOSM). SOSM role manages the daily
operations of all major incidents and data center operations. CRC and SOSM will closely
monitor all major incidents. Once a major incident is deemed unrecoverable, the CRC
will perform and initial event assessment. The CRC will notify an Executive Management
Team member. If the situation continues to escalate and disaster declaration is likely, the
Critical Response Team (CRT) will be notified for detailed assessment. CRT will perform
detail analysis and impact assessment. Any workaround will be assessed representing
the effected businesses and IT groups. The EMT, based on recommendation from CRT,
will declare a disaster and DR plan will be executed.
33. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 33
Version 2.1
4.10.3 Disaster Recovery Plan – Roles and Responsibilities
Each business application platform or Service Operations will maintain Disaster Recovery
plan within the Qualysis System. Attached below is a sample disaster recovery plan.
Each major system will have DR plan and owner.
CEDAR Disaster
Recovery Plan Templa
4.10.4 CEDAR Disaster Recovery Tier
Recovery will be based on the application tier of service following the Recovery Point
Objective (RTO) and Recovery Point Objective (RPO). The definition or recovery is
defined below in the event of disaster. The BES Cyber systems with High asset value
will have faster recovery time and frequent recovery point objectives measured in
minutes. Non production or development environment will be lowest tier of recovery in
the event of disaster recovery event.
34. Effecti
Versio
ive 05/23/2013
on 2.1
ܥĥ
MMJS –
•–‡”
RTO =
RPO =
RTO = B
to the a
– CEDAR Propo
‡…‘
= the am
= the am
Busines
pplicatio
osal
‘˜‡”›
ount of t
mount of t
ss RTO -
n to cont
›‹‡
ime the s
time data
time allo
tinue nor
Page 3
‡”•
system c
a would b
owed to r
rmal busi
34
can be do
be lost.
recover b
iness fun
own.
busine
nctions
35. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 35
Version 2.1
4.10.5 Live system recovery
Windows - Bear Metal Restore
Symantec System Recovery (SSR) is an industry leader in backup and recovery of
systems. The software will need to be installed in each virtual or physical computer
system. Backup is flexible for specific scheduled backup. In an event of failure, the
backup image can be restored to the original state. Each image will need to be stored off
the server in case of failure and two week backup as a policy is recommended in case of
patching or application failure. The images will be stored in Commvault for tape and
store in archive tape media. The BESR configuration on each system will be configured
to retain the last 14 images before they are overwritten. In limited cases the BESR
system can be utilized to convert Physical to Virtual conversion as an approach to
Ġ
RTO = the amount of time the system can be down. Time allowed to recover b
application to continue normal business functions once disaster declaration is m
RPO = the amount of time data would be lost.
36. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 36
Version 2.1
consolidate and reduce the physical footprint of servers in the local and remote
datacenters. SSR keeps active log of events and features notification system to incident
management for administrator follow up in the event of failure.
Any system considered High or Medium assets will be installed with BESR, where
recovery time must in hours vs days. Commvault will be utilized for any static or file
system tape of data.
Linux or Unix systems
Maksysb or Linux Redhat satellite server can be used to capture the live system through
either job scheduler to central point. Commvault can be used to backup those systems to
tape for offsite storage. Redhat Linux satellite server will be utilized for deployment
services. System administrators will be notified of any failures through scripting jobs that
will notify incident management system.
All system log files will be retained within the two week period per the retention policy.
However depending the criticality of the system, log and system event information can be
retained for longer periods
4.10.6 Data Backup
CEDAR currently has need to backup PetaByte of data. Commvault’s Simpana V10 will
be utilized to backup SSR and any file data to short term storage. The primary purpose
of the short term storage is to de-duplicate the data and store into tape devices. Based
on PetaByte of data storage requirement five tape array will be utilized in the primary
datacenter for central backup. The diagram depicts the backup architecture. Simpana
media agent and media servers will pull data for scheduled backup.
Backups will be schedule based on the category of BES cyber asset category. There are
native plug-ins for Oracle and SQL Databases. The DBA will still store the transactional
logs on the local system for quick rollback in case of user error. However the Simpana
backup will backup any stateful transactions to the backup system.
Tape hardware will include the IBM TS2900 series for remote site data backup to
TS3500 series for the data centers. The physical layer vs the software for backup
management will allow CEDAR to implement the most cost effective system at the
hardware and software layers.
Any tapes stored in offsite location must be stored in secure locked location with proper
tape labeling. Broken or bad tapes will follow the data destruction policy to properly
dispose of the tapes. Currently Iron Mountain services are used across Illinois sites for
proper disposal. A certification form will be held in record for any tapes destroyed
following the procedures for up to 3 years. Any lost tapes will be reported to local
Business Security Officer for proper notification and follow up.
Critical end user workstation may need to be backed up for compliance requirements.
Commvault DLO tape backup will be performed on these workstation utilizing the same
Simpana and IBM Tape library for backup.
37. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 37
Version 2.1
http://www.nasi.com/images/simpana-dedupe.png
4.10.7 Data De-duplication
One of the key components of Simpana V9 is de-duplication of the data. Simpana
Content Store will store the backup information into central repository. Based on the
duplication of the content, the amount actually stored will be reduced based on the
redundant data being backed up across the enterprise. The de-duplication of storage
data to tape will reduce the number of tapes required and decrease the Total Cost of
Ownership (TCO). However there is some sacrifice will be made during recovery. Since
the data are deduped, the number of tapes may increase to restore a set of data thus
reducing the amount of recovery. Someone data that require High BES Cyber asset
recovery time may be set with no deduplication to reduce the recovery time.
38. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 38
Version 2.1
http://webdocs.commvault.com/images/content/backup-and-recovery-technical.jpg
4.10.8 Alerting
Highly Available (HA) Central management console will provide alerting and event logging.
Any scheduled jobs with failed exceptions will send an alert to appropriate backup and
storage team for follow up. Any SSR failures will also generate alerts for system
administrator to follow up and determine root cause analysis and recovery of the backup
services
4.10.9 Monitoring and Backup Reports
Sample report below will be provided to IT leadership on the status of successful backups
39. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 39
Version 2.1
There will be three categories of reports that non backup admins will be able to access via
website
• Backup schedule – allow application administrators to view backup schedule of
their systems.
o Annual backup will be the last full backup of the year
o Monthly full backup will occur last full backup of each month
o Weekly backup will occur at the last full Friday of each week and will
depend on the user performance and production impact to schedule the
backups
• Filter backup report based on system name
• Determine if system is backed up by commvault system.
Below are sample reports of backup reports for individual systems. Any failures of data
backup will be alerted to backup administrators to perform root-cause analysis and
schedule backup follow evening. Any exclusion will also be available for application
owners to review and determine if they should not be backed up. This may be source
install files that can be easily downloaded from the vendor site. The report will provide
detailed information about the timing of the backup, amount of data backed up, success or
failure,
40. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 40
Version 2.1
4.10.10 Resilience Management ProgramDisaster Recovery
CEDAR will implement Disaster recovery every 1.5 years for each high and medium BES
Cyber critical systems as part of Resilience Management Program. Each DR plan will
consist of Crisis Management, Business Continuity, Disaster Recovery, and Emergency
Response Plans. Each application will be tracked under Archer for record of testing. Any
lessons learned and system gaps will be recorded and tracked to resolution. The following
schedule will be used for DR testing and formally signed off by the application business
leader.
Any changes to the DR plan will be recorded in the Archer system. Within 90 days of the
DR exercise, the team will document and disseminate any lessons learned, results, gaps,
to participants, sponsors and stakeholders. DR plans will be updated and notified to
stakeholders due to role changes within the technology or application leader ownership.
42. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 42
Version 2.1
4.10.11 Implementation cost analysis
Expense Capital Labor
Backup infrastructure HW 100K 200K 50K
Symantec System Recovery $500/server
Commvault SW maintenance –
Server (annual)
$2500/TB
Commvault SW maintenance –
Workstation (annual)
$1250/TB
4.11 Change Management
4.11.1 CIP010– Regulatory Requirements
CIP-010 outlines proper procedures for change management and vulnerability
assessments. This document outlines the minimum requirements for CEDAR and is
adapted from CIP-010-1 from NERC. All changes will be recorded in a centralized change
management system. Vulnerability assessments will be logged in the same system for
tracking purposes.
4.11.2 Change Tracking Software
1. CEDAR will utilize Microsoft System Center 2012 for tracking active, complete and
future changes
2. MS System Center requirements
a. Dual-Core x64 3.0GHz server
b. 8GB RAM
c. 50GB HDD Space
d. Windows 2008 R2
e. Separate MS SQL server
3. Server Cost (Includes purchase of server OS)
a. Management/Library Server - $3,231
b. Database Server - $4,564
4. Licensing Cost
a. System Center - $1,803.50/year
b. SQL Server 2012 - $54,995 (based on 8 cores total @ $6,874/core)
5. System Center can be used to automatically manage Microsoft based servers without
additional licensing, non-servers will need management clients installed if desired
a. Cost for non-server client is $62 for a 2-year period per device
4.11.3 Change Management Process
1. Staff will submit change requests through MS System Center
2. The change approval board will meet to discuss all changes
a. Board will meet on a twice-weekly basis
b. Board will consist of key personnel from each department
c. Changes will be approved based on risk and priority
43. MMJS – CEDAR Proposal
Effective 05/23/2013 Page 43
Version 2.1
d. Emergency changes can be approved by CIO and department manager without
change management board approval
6. Affected end-users will be notified of pending changes
7. All changes will be tested in a non-production environment, if available
8. Once change is verified a backup of affected system is created
9. Change is made to affected system
10. All changes are verified good or bad
a. If bad change is backed out to last good backup
b. If good change is considered complete with new baseline established
i. New baseline established within 30 calendar days of change
The change process must be thoroughly documented in the change management system
4.12 Information Protection
4.12.1 CIP011– Regulatory Requirements
CIP-011 outlines proper procedures for information protection. This document outlines the
minimum requirements for CEDAR and is adapted from CIP-011-1 from NERC. This
document refers to electronic and paper media.
4.12.2 Information Protection
Proper Identification of Documents –
1. Documentation of BES Cyber Systems Information shall be identified by a document
control number (DCN) and stored in a secure location.
2. All personnel will be properly trained in how to recognize sensitive BES Cyber Security
Information.
Access Control and Handling Procedures –
1. All physical and electronic BES Cyber Systems documentation shall be tracked by DCN
for information stored, transported and disposed of in a manner consistent with
documented processes.
2. All electronic copies of BES Cyber System Information shall have user access granted on
a need-to-know basis and all activities will be tracked.
3. Hardcopies of BES Cyber System Information will be stored in a secure location and
access will only be granted to authorized personnel.
Review of Protection Standards –
1. At least once every 12 calendar months internal auditors will assess adherence to BES
Cyber System Information protection processes. Thorough documentation will be
required consisting of assessment results and remediation procedures for deficiencies
identified.
2. Evidence shall include at least assessment results, action plan, evidence showing action
plan implementation.
4.12.3 Media Reuse and Disposal
Reuse of Media –