Moodle is dead... Iain Bruce, James Blair, Michael O'Loughlin Presented at Edinburgh Moodlemoot 2014 www.moodlemoot.ie

1. Moodle is dead…

2. The Problem
• On the 30th of July 2013 the VLE, DBA and Network teams of Information
services were invited to a meeting which was to test our Business
continuity with our VLE environment (Moodle). The teams were given the
following scenario.
• All University systems have been shut down due to a full power failure which has
affected both Craiglockhart and Merchiston with no other current services at our
Sighthill campus; this also means that there is no external internet access from inside
the University.
• Moodle as being the most critical system at this time of the year is essential and has
to be back on-line as quickly as possible to allow students access to their current
course work.

3. Team brief
• The team are allowed access to any of the offsite backup systems
(VMs, Data and Databases).
• No University hardware can be used for the process.
• Staff were allowed to use their on-call laptops.
• Staff have access to budget to gain resources if needed.

4. Incident Procedure
Incident Manager
Coordinates
troubleshooting/Information
gathering
IT Incident Duty Manager
(Technical Services)
Takes ownership of Incident
through to resolution
Technical Expert(s)
Update call notes with
progress
Call Management system records the various stages of the Incident.

5. Who was involved?
Head Of
Applications
DBA?
VLE/WEB
Data Centre
&
Operations

6. Edinburgh Napier’s
Moodle Infrastructure

7. Some stats…

8. What Solution? (Approx. 2hrs)
• The following decisions were made during the initial Emergency
Incident meeting.
1. We would use Amazon Web Services and create a virtual machine in the
cloud.
 AWS provided scalable solutions, import/export options for database
2. We would look to get database and files backup from tape
3. Switch user accounts to manual
4. We discussed how to communicate out to students.

9. Tasks involved
 Obtain a database backup (Approx. 4hrs)
a. Installed Zmanda Community Edition to recover MySQL
b. No access to DBA and use of the Zmanda recovery software was
problematic(backup had to match the version of MySQL), meant a Hot
backup was obtained.
c. Truncated logs and statistics to speed up the import
d. Import Database into AWS MySQL database
 Obtain Moodle user files from backup (Approx. 5hrs)
a. Backup came from Symantec NetBackup as we found file storage was not
going to tape at the time.

10.  Register for Amazon Web Services (AWS) account
a. Purchased AWS Business support.
 Setup AWS (Approx. 30mins)
a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.
b. Create a Key Pair - AWS uses public-key cryptography to secure the
login information for your instance.
c. Communicated the AWS account credentials to key team members
to allow other aspects of the service to be configured.
 Transfer Moodle user files to Amazon (Approx. 7hrs)
a. Initial size of backup was 475GB, this was reduced to 190GB after
removal of duplicate directories and redundant course backup files.
b. Initially tried using WinSCP for transfer, this was going to take 14hrs,
switched to RSYNC transfer complete in 6hrs.
Tasks involved

11. Tasks involved
 Obtain a database backup (Approx. 4hrs)
a. Installed Zmanda Community Edition to recover MySQL
b. No access to DBA and use of the Zmanda recovery software was problematic(backup had to match the version of mysql), meant a
Hot backup was obtained.
c. Truncated logs and statistics to speed up the import
d. Import Database into AWS MySQL database
 Obtain Moodle user files from backup (Approx. 5hrs)
a. Backup came from Symantec NetBackup as we found file storage was not going to tape at the time.
 Register for Amazon Web Services (AWS) account
a. Purchased AWS Business support.
 Setup AWS (Approx. 30mins)
a. Launch Amazon console in correct region.
b. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.
c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.
 Transfer Moodle user files to Amazon (Approx. 7hrs)
a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup
files.
b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs.

12. Tasks involved
 Setup AWS (Approx. 30mins)
a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.
b. Create a Key Pair - AWS uses public-key cryptography to secure the login
information for your instance.
c. Communicated the AWS account credentials to key team members to
allow other aspects of the service to be configured.
 Transfer Moodle user files to Amazon (Approx. 7hrs)
a. Initial size of backup was 475GB, this was reduced to 190GB after removal
of duplicate directories and redundant course backup files.
b. Initially tried using WinSCP for transfer, this was going to take 14hrs,
switched to RSYNC transfer complete in 6hrs.

13. Tasks involved
 Obtain a database backup (Approx. 4hrs)
a. Installed Zmanda Community Edition to recover MySQL
b. No access to DBA and use of the Zmanda recovery software was problematic(backup had to match the version of MySQL), meant a
Hot backup was obtained.
c. Truncated logs and statistics to speed up the import
d. Import Database into AWS MySQL database
 Obtain Moodle user files from backup (Approx. 5hrs)
a. Backup came from Symantec NetBackup as we found file storage was not going to tape at the time.
 Register for Amazon Web Services (AWS) account
a. Purchased AWS Business support.
 Setup AWS (Approx. 30mins)
a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.
b. Create a Key Pair - AWS uses public-key cryptography to secure the login information for your instance.
c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.
 Transfer Moodle user files to Amazon (Approx. 7hrs)
a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup
files.
b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs.

14. Tasks involved
 Setup AWS (Approx. 30mins)
a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.
b. Create a Key Pair - AWS uses public-key cryptography to secure the login
information for your instance.
c. Communicated the AWS account credentials to key team members to
allow other aspects of the service to be configured.
 Transfer Moodle user files to Amazon (Approx. 7hrs)
a. Initial size of backup was 475GB, this was reduced to 190GB after removal
of duplicate directories and redundant course backup files.
b. Initially tried using WinSCP for transfer, this was going to take 14hrs,
switched to RSYNC transfer complete in 6hrs.

15. Tasks involved continued...
 Installation & Configuration of Services (Approx. 2hrs)
a. Apache, PHP & MySQL Installed, services started.
b. Installation of GIT
c. Cloned our Moodle code from Git Hub where all our commits are backed
up automatically.
d. Backed up ignored files separately, for example config.php
e. Ensure Apache user has correct permissions on directory.
f. Alter Moodle config.php with new database credentials.

16. Tasks involved continued...
 Moodle Administration (Approx. 1hr)
a. Recreated Moodle admin account
b. Switch Moodle user accounts to ‘Manual’ and regenerate passwords
c. Successfully access Moodle user accounts and data on AWS, various
admin/teacher/student accounts accessed and tested.

17. Moodle in the Cloud!

18. Lessons Learned
• Backups were only stored on high availability disks across two campuses
• Recommend
• Backup to tape every 2 weeks.
• Passwords for a number of the services were only with the DBAs
• Recommend
• Storing of passwords in the same way across the teams.
• Access to all passwords for the teams.
• Further Backups
• Recommend
• Additional Emergency MySQL dump
• Filter out redundant files from backup to reduce time
• Migrate only necessary tables
• Search and replace text inside sql file before import (mainly for hardcoded urls)
• Replicate data up to Cloud

19. Lessons Learned continued…
• SMS students about the downtime of the system
• Recommend
• Storing of student SMS/phone details off campus or on on-call laptops
• Moodle
• Recommend
• Create basic scripts for the processes against the database on user accounts
• Active directory is one of the main services to allow our system to work
• Recommend
• Store an active directory server in the cloud or at another universities network

20. Going forward
• Have a cloud Active Directory solution available.
• Have a parked AWS instance available when required.
• Use Vagrant/PuPHPet GUI to have consistent setup and manage virtual machines

21. Useful links
• Amazon Web Services - http://aws.amazon.com/
• Windows Azure (Cloud AD) - http://www.windowsazure.com/en-us/
• Vagrant/PuPHPet - https://puphpet.com/