Automating file transfers january 2015 Webinar Filecatalyst

1. Webinar:
Things to consider when automating file transfers
January 15, 2015

2. News flash: Unlimi-Tech wins a Emmy
| © Copyright FileCatalyst, 20152

3. Introduction
• John Tkaczewski, President and Co-Founder
• Check out our blog: http://www.filecatalyst.com/blog
| © Copyright FileCatalyst, 20153

4. Overview
• FileCatalyst Introduction
• Challenges with Automated Workflows
• Demo HotFolder
• Final Remarks
• Questions
| © Copyright FileCatalyst, 20154

5. What is File Transfer Acceleration?
• File transfer acceleration technology allows large files, or “big data”
to be sent quickly over large geographical distances without being
impeded by network impairments such as latency and packet loss
• This is accomplished by transferring files using the UDP protocol
instead of the TCP protocol, the protocol used for FTP
• The main difference between UDP and TCP is that UDP wastes no
time communicating about the receipt of a block of data so
that regardless of network latency, data transmission remains
constant, thus overcoming the peak and valley effect that occurs
with TCP based protocols on links with high latency
• Reliability and congestion control is added to UDP at the application
layer thus it doesn’t sacrifice its other desirable properties
| © Copyright FileCatalyst, 20155

6. Speed Gains with Acceleration vs. FTP
| © Copyright FileCatalyst, 20156

7. Automation, there is nothing to it!
Setup Cron Job using SCP… and I’m done right?
…. Well Perhaps Not!
| © Copyright FileCatalyst, 20157

8. Challenges of Automated Workflows
• Volume of data
• System Notifications (message fatigue)
• System properties (system clock, mounts, network drives)
• Interpreting logs and finding errors
• Folder Synchronization
• Cross platform transfers, ex: Windows to Linux
• Transfer Speed
• Data Verification (Checksum)| © Copyright FileCatalyst, 20158

9. Automation tools within FileCatalyst
9
HotFolder
© Copyright FileCatalyst, 2015
Command
Line

10. Automation tools within FileCatalyst (Cont’d)
Hotfolder features useful for data replication:
• Scheduler for tasks and bandwidth
• File System Events
• Delta (RSYNC)
• Data-set rules
• Monitoring and notifications
• ZIP chunking (Single Archive)
• MD5
• File transfer cache
• Multi-Client (multiple concurrent file transfers)
| © Copyright FileCatalyst, 201510

11. Demo

12. Platform specific gotchas
Common file attributes for all platforms:
File name, Date Created, Date Modified, Owner
| © Copyright FileCatalyst, 201512
Operating System File Attributes
MacOS Extended attributes (xattr), Resource Forks, Symlinks, Permissions
(Read Write Execute), Access Control Levels (ACL’s)
Windows Permissions, Attributes: Read Only – Archive - Hidden – System,
Security Attributes
Linux Permissions (group, user, other)

13. Data replication across operating
systems
• Extended File Attributes are LOST when moving a file
between operating systems.
• But it’s OK because extended attributes may not be always
needed.
| © Copyright FileCatalyst, 201513

14. File system events
The file system events provides a way for an application to
ask for notification when the contents of a directory hierarchy
are modified.
For example, an application can use this to quickly detect
when the user modifies a file within a folder using another
application.
• Avoid scanning large folders to detect changes! Use file system events
• File system events only work on local drives (no SMB) Most Enterprise
network drive technology does support file system events
• Transfer the initial file set without file system events, THEN turn it on.
| © Copyright FileCatalyst, 201514

15. Upcoming Events
- FileCatalyst User Forum, London, UK (around BVE)
- BVE 2015, Excel, London, UK (Feb 24-26), stand R40
- CABSAT 2015, Dubai World Trade Centre (March 10-12), stand 220
- NAB Show 2015, Las Vegas, Nevada (April 13-16), stand SL8505
And of course don’t forget our monthly webinars... And …
… our email newsletter “The Transfer Express”
| © Copyright FileCatalyst, 201515

16. Thank you!
Questions?