1. Lessons Learned From the
Texas Gulf Coast
ACUTA 2009
Palm Springs, California

2. About Your Speakers
Texas State University System
Dr. Michael E. Dobe, Sr., Associate Vice President,
Chief Information Officer & Adjunct Professor at
Lamar University (Beaumont, Texas)
http://www.linkedin.com/in/mdobe
Texas A & M System
Mr. Steven Conway, Director of Computing and Info
Services at Texas A&M University at Galveston
(Galveston, Texas)
http://www.linkedin.com/pub/5/b7/24a

3. About the Checklist
Steven Conway
Texas A&M University at Galveston (TAMUG)

4. ACUTA/ATIS Hurricane
Preparedness Project
Formed Committee May 2007
Reviewed and Modified ATIS document
Published ACUTA Hurricane Checklist Oct 2007

5. Hurricane Preparedness
Checklist
Hurricane Katrina, August 2005, Terra MODIS True Color
Image
Courtesy of Louisiana State University
ACUTA ‐ The Association for Information Communications Technology Professionals in
Higher Education
Developed in cooperation with
ATIS – The Alliance for Telecommunications Industry Solutions
http://www.acuta.org/?2026

6. Committee
Amy Phillips, Louisiana State University Health Sciences Center
Frank O'Quinn, Louisiana State University
Sandra Thompson, University of West Florida
James Shea, Boston University
Jeanne Spinosa, Johnson & Wales University
Garret Yoshimi, University of Hawaii
George Denbow, University of Texas at Austin
Michele Narcavage, University of Pennsylvania
Steve Ellis, University of Notre Dame
Steve Conway, Texas A&M University at Galveston
Walt Magnussen, Texas A&M University ‐ Task Force Chair
Veronica Lancaster, ATIS
Brad Ruark, ATIS /AT&T
Jeri Semer, ACUTA Executive Director
Aaron Fuehrer, ACUTA Staff Liaison

7. ACUTA Hurricane Preparedness Checklist

8. Using the Checklist As An After‐Action Review
Tool
Perspectives From Beaumont
Dr. Michael Dobe, Sr.
Lamar University

9. About Lamar University
Located in Beaumont, TX (Approx 40 Minutes from
Louisiana Border)
Central Data Center in Beaumont Serves Three
Campuses:
Lamar University
Lamar Institute of Technology
Lamar State College‐ Orange
Total Students Served – Approximately 16,000

10. Proximity of Beaumont and Galveston
119 mi – about 1 hour 57 mins

11. Rita, Edouard, Gustav, Ike
Hurricane Activity in SE Texas
Rita – September 2005 (CAT 4 With Major Damage to
Beaumont)
2006 and 2007 Atlantic Hurricane Seasons Quiet in SE
Texas
Edouard – August 2008 (Missed SE Texas)
Gustav – September 1, 2008 (Minor Impact in SE
Texas)
Ike Hit SE Texas Less Than Two Weeks After Gustav…

12. Hurricane Ike’s Path
http://en.wikipedia.org/wiki/File:Ike_2008_track.png

13. Hurricane Ike Timeline
Lamar evacuated campus 10 Sept 2008
Hurricane Ike landfall Evening of 12‐13 Sept 2008
Low Lying Areas of Beaumont Flooded, Extensive
Flooding in Orange County, Bridge City Hardest Hit
Lamar University Campus in Beaumont Did Not
Flood
Library, Family and Consumer Sciences, and
Montaigne Center Buildings Had Roof Damage
24 Sept 2008 Essential Staff Returned to Campus
25 Sept 2008 Classes Resumed at Lamar University

14. Assessing Services During Ike
LU Lessons Learned
Storm Facts Inventory
Contact Lists Physical Assets
Emergency Sites and Lodging Power and Fuel
Administration, Network Service
Communications, and Main Computer Center and
Manpower Key Telecommunications
Safety, Security and Training Hubs
Long‐Term Preparations
Post‐Storm Activities
Damage Assessment and
Post‐Storm Activities

15. Storm Facts
NOAA Website
Major News Media: CNN, Weather Channel and
Storm Pulse (http://www.stormpulse.com/)
Importance of Local News Media: KFDM and
Entergy
Government Contacts: Jefferson County and FEMA
Local Contacts: State Police and Beaumont Police

19. Contact Lists
Updated and Distributed Internal Contacts For
Gustav
Leadership Team Phone List
Pocket Card for Satellite Phone Numbers
ITS Contact List
External Contact Lists Maintained By Staff Sections

20. Emergency Sites and Lodging
Emergency Control Site Remained on Campus in
Beaumont
Temporary Lodging and Shelter Arranged by
Individuals
Disaster Recovery/Business Continuity Site at San
Marcos State University

22. Texas State University System
http://www.tsus.edu/
TSUS Institutions (East Texas Locations in Bold)
Lamar University
Sam Houston State University
Sul Ross State University
Sul Ross State University Rio Grande College
Texas State University‐San Marcos
Lamar Institute of Technology
Lamar State College‐Orange
Lamar State College‐Port Arthur

23. Sam Houston State in Hunstville
153 mi – about 2 hours 29 mins
TOO CLOSE FOR COMFORT

24. Stephen F. Austin State
128 mi – about 2 hours 36
NOT MEMBER OF TSUS

25. San Marcos State University
251 mi – about 4 hours 15 mins
During Ike – Approx. 16 hrs
LAMAR UNIVERSITY BC Site

26. Administration, Communications and
Manpower
Importance of Clear Chain of Command and
Clear Delineation of Responsibilities
ITS Set Up Two Teams: Offsite B/C Team and Onsite
Command Center Team
Communication with Employees via LU Website and
ConnectED (Now Blackboard Connect) to E‐Mail and
Phones.
Communications Methods: Cell Phones, Satellite
Phones, Land Lines and SIP Phones for Key
Employees

27. ACUTA NOTE: Effective Leadership Was Key to Success

28. ACUTA Note: Importance of Coordination of Operations Functions

29. ACUTA NOTE:
Importance of
Coordination of the
Academic Response

30. Administration, Communications and
Manpower
ITS Hunkered Down With LUPD in Beaumont
Daily Meetings for Coordination in Beaumont on
Campus
Reporting and Notification Lead By President and
Leadership Team
Conference Bridges For Coordination With Remote
Staff Managed by Network Services
Public Coordination Handled by LU PD
Managed Through Office of VP for Finance and Amin
Supplemental Personnel
Accountability and Time Reporting
Costs

31. LU IT Services Staff Disposition
Local Core Team Stayed on Campus Throughout
Remote Core Team Relocated to San Marcos Area –
Worked on Ordering Replacement Systems
Remaining Staff Dispersed With Families
AT&T Conference Bridge to Maintain Contact With
Dispersed Staff

32. Onsite IT Service Lead: John Genuardi, Customer Support Director

33. Onsite Network Support: Patrick Stewart, Manager of Network Services

34. Onsite Data Center Lead: Srinivas “Sri” Vandaraj, Senior Security Analyst

35. Safety, Security and Training
President’s Weekly Security Meeting Initiated as
Outcome of Rita
System Access Managed By Onsite Core ITS Team
(Issues With Passwords: Need One‐Time Password
System)
Physical Re‐Entry Credentials Provided by Chief of
Police
ITS and PD Coordinated
Security Cameras Shutdown and Start‐up
Training for ConnectedED

36. Inventory and Physical Assets
Impact of Key Staff Resignation
Planning: AVP (IT Infrastructure and Security)
Responsible for Availability of Emergency Supplies
and Work Supplies
AVP Had Resigned Between Gustav and Ike!
Single Point of Staff Failure – Transitioned
Responsibilities Quickly …
Equipment Inventory, To Include Network and Data
Center Backup and Recovery Plans Needed to Be
Transitioned ON THE FLY! (New Role of Onsite Lead)
Note on Vehicle Planning: Golf Carts Proved Critical
After Storm

37. Power and Fuel: Handled By
Facilities Management
Staff Continuity Maintained ‐ Gustav to Ike
Communication with Local Fuel Vendors
Communication and Coordination With Power
Companies
Engine/Genset (portable and permanent)
Fuel Tanks
Batteries
Rectifiers
Staging and Deployment

38. Network Service
Preventative Measures:
No Redundant Routes for Network (Big Exposure)
LEARN Connectivity Initiative Q 1 of 2009
Half‐Completed VoIP Migration (Half Cisco/Half
Avaya)
Priority Areas:
Network Services Spread Between Windows Services
Group (DHCP) and Unix Group (DNS).
Failure to Standardize Led to Confusion and LAN
Service Outages During the Storm (Premature AD
Shutdown).
Failure to Document All Services Creates Risk

39. Significant Telecomm Events
GOT LUCKY ‐‐ LU OC‐3 Circuit to Houston Stayed Up
Throughout Event – NO INTERNET CONNECTIVITY
OUTAGE (Gigaman to Orange Also Stayed Up)
Onsite Team Kept Campus Services Running Until
Night of Landfall (Sept. 12‐13)
Throughout Storm VoIP access via WAN using SIP
Phones (Ordered More Broadband Cards After Storm)
Cell Service Unreliable Beaumont to Remote Sites
Residential lines out for over several weeks
Satellite Phones Proved Difficult to Use

40. Main Computer Center and Key
Telecommunications Hubs
Success Story: No Systems Shutdown Sequence in
Place For Data Centers. Planning Initiated in Gustav
Continued Into Ike Lead Up
Downside to Not Running Your Own Power:
Power Failure Exposure Due to Unmet Power
Requirements in Data Center
Concerns re: Lack of Fuel Led Local Team to Bring
Down Systems After Event
Tape Backups: Contract for Off Site Had Just Lapsed!
Can You Sing “Do the Hustle”?
Wind and High Water: Failure to Stock Pile Sandbags

42. Long Term Preparations
Maintain Year‐Round
Example of Successful Planning: Replication of
Website and Content Management Service
Criticality of ERP Disaster Recovery/BC Site at San
Marcos
Hardware In Place in San Marcos
Software to Be Configured With Data Replication (Tested)
By Hurricane Season 2009
Long‐Term Plans/Prevention
Move to the Cloud: Portal and Mail Outsourcing Initiatives
Importance of Service Continuity: Redesign of Network
Services (Standardization Under Network Services Group)

43. Post‐Storm Activities Led By
Onsite Team Lead
Employee Safety
Contacts
Conference Bridges
Costs
Temporary Lodging/Shelter
Extra Work Force
Safety
Emergency Supply Availability
Power and Fuel

45. Flooding in Low-Lying Areas

47. Damage to People’s Homes

48. Damage to Local Businesses

49. Damage Assessment and Service
Restoration
Announcement Via ConnectED on Return of Staff to
Campus (September 24th), Return of Students to
Campus (September 25th)
Class Schedule Readjusted by Office of the Provost,
With Resumption of Classes on September 25th
President’s Faculty/Staff Convocation on September
25th

51. Perspectives From Galveston
Steven Conway
Texas A&M University at Galveston (TAMUG)

52. Hurricane Ike
TAMUG evacuated campus 10 Sept 2008
Hurricane Ike landfall 12 Sept 2008
75% of Galveston was flooded
TAMUG did not flood, 2 buildings had roof damage
TAMUG restarted classes at TAMU in College Station
TX Wed 24 Sept
91% of the students returned
20 Jan 2008 Classes resume in Galveston

53. Significant Telecomm Events
Cell service stayed up throughout storm‐Cat 2
UTMB Gigaman Circuit to Houston stayed up
TAMUG‐UTMB Gigaman available 24 Sept when
power returned on campus
T1’s out for over a month
Residential lines out for over 6 weeks
VoIP access to College Station via WAN

54. Hurricane Ike Flooding Galveston
A house burns uncontrolled in a flooded neighborhood as Hurricane
Ike approaches the Texas coast, Friday, Sept. 12, 2008, in
Galveston, Texas. (AP Photo/Matt Slocum)

55. TAMUG Boat on Causeway