O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

 Active Storage - Modern File Storage? 

234 visualizações

Publicada em

Overview of what modern file storage should look like and how Active Storage fits into that definition. We look at the file life cycle and design considerations at each stage. We also take a look at alternatives like Shrine and AWS Lambda

Publicada em: Tecnologia
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

 Active Storage - Modern File Storage? 

  1. 1. 📦 Active Storage - Modern File Storage? 📦 by Michael Yagudaev
  2. 2. About Michael • Co-Founder @ Nano 3 Labs & PictureThat • Full-stack Developer Rails/React • Been working with Rails since 2012 • Web dev since 2001 • Passionate about Product Design & AR • Fun Fact: offline snowboarding 🏂
  3. 3. Outline • Battle Stories: file upload is hard… • What is Modern File Storage? • What is Active Record? • Uploading a File • Transforming a File • Downloading/Previewing a File • Alternatives to Active Storage • Conclusion
  4. 4. Battle Stories
  5. 5. Story 1: Bail on Rails • Back in 2012 (Rails 2.3) — small startup, terrible uploader. Users upload entire hi-res albums (100s of photos). Blocking all other users from doing anything on the platform! :( • Sales staff hides in hope of not running into angry users • Can you build us a better file uploader? • Me: yes! Let’s use Node.js + Direct Upload to S3 • Me 3 months later: FML… node… ;( • We created a robust uploader • auto-retry failed uploads, priority resize images users requested, pre-create resize jobs for all images sizes, custom uploader ui
  6. 6. Story 2: Rails re-discovered • 0idle - our own startup helping event organizers find venues • 2013: implemented multi-file upload • Carrierwave + Carrierwave-Direct S3 • Only took a few days • ❤ Rails • Life is good :) • Custom upload UI
  7. 7. Story 3: DDoS Day • Summer 2018 - implemented ActiveStorage for PictureThat. Loved the variants feature • Fall 2018 - • DDoS attack on one of our projects • Targeted slow pdf conversation end-points • Made us look into slow running requests • Second slowest was asset upload • A DevOps Engineer I was working with outlined “Modern File Storage” to me… it got me thinking, is Active Storage Modern?
  8. 8. What is Modern File Storage? • Modular • Supports cloud providers • Fast • Scalable • Cheap • Easy to setup • API Based
  9. 9. What is ActiveStorage? • Extracted from BaseCamp • Supports • Cloud Providers (S3, digital ocean, GCP, Azure) • Variants • Mirroring • Previews • Accessible MetaData* (sorta we will see soon) • Upload: documents, images and videos (yes!)
  10. 10. File Life Cycle
  11. 11. File Life Cycle 1. select file(s) 2. upload file 3. analyze file 4. transform file 5. preview / download
  12. 12. $ bundle open activestorage
  13. 13. File Life Cycle - AS Mapping 1. select file(s) - rails form / bring your own ui 2. upload file - activestorage/app/controllers/ 3. analyze file - activestorage/jobs/…/analyze_job.rb 4. transform file - activestorage/lib/…/ transformers.rb 5. preview / download - activestorage/models/…/ variant.rb (or) activestorage/models/…/preview
  14. 14. 1. File Selection
  15. 15. 1. Select File(s) - Traditional • Easy with rails forms • Add <%= form.file_field :files, multiple: true,%> • For S3 Direct Upload (work with Digital Ocean, GCP, etc) • <%= form.file_field :files, multiple: true, direct_upload: true,%> • Use direct upload, no long requests
  16. 16. Rails Form File Field Be Like…
  17. 17. 1. Select File(s) - BYOU • Use modern Javascript framework like React • Try uppy.io • Can work as Multi-part XHR Upload • But a two-step API based upload is better • API should support upload from iOS and Android apps
  18. 18. 2. Uploading
  19. 19. 2. Uploading • Considerations • size of files (resumable?) • # files • types of files • file selection method in step #1 • Offload away from Rails, it wasn’t meant for it • Maintain short connection timeouts. Let S3 handle upload
  20. 20. Multipart form file upload • Sends metadata & file content in one request • Chunks file and sends using a boundary, looks like • Content-Type: multipart/form-data; boundary=—— WebKitFormBoundarydmmGebTloNbZGViV • Old and meant for forms, not APIs • Can be used with XHR request
  21. 21. 2-Step API Upload • Instead use 2 requests 1. Upload Meta data (file name, size, type, etc) 2. Get Location back from step #1 for direct upload • Active Storage Direct does this! • POST http://localhost:3000/rails/active_storage/direct_uploads • { blob: { filename: …, content_type: “image/jpeg”, byte_size: 123, checksum: “aoEPYm6I0kn9sFqekqy5yg==" }} • RESPONSE: { direct_upload:{url:”<s3_signed_url>”, headers: {…}} }
  22. 22. AS Schema • Run: $ rails active_storage:install • active_storage_blobs => Meta data about file • byte_size • content_type • checksum (md5 as base64) • metadata (height, width, etc) • active_storage_attachments => Relationship between your models and blobs • blob_id • record_id • record_type (polymorphic association)
  23. 23. Custom API • Simple all you need is • Then just • POST /api/v1/files { blob: { filename, content_type, byte_size, checksum } } class Api::V1::FilesController < ActiveStorage::DirectUploadsController skip_forgery_protection end
  24. 24. 3. Analyzing
  25. 25. 3. Analyze • File has already been uploaded • File is in temp folder or cloud storage • If in temp • Easy, run appropriate tool like`identify` on images • Compare meta to client received ones • Save meta data back
  26. 26. 3. Analyze • If on cloud storage (s3), either: • Download to tmp file, and run analysis locally (AS) • Run cloud analysis and connect to DB + update or update through a webhook (no help from AS here) • Ideally run background job / cloud function • Might want to run things like a virus scan • ActiveStorage has an AnalyzeJob, use it • What happens if user requests resource right away?
  27. 27. 4. Transformations
  28. 28. 4. Transformation • Resize images, extract display frame from video • Either pre-process or lazy-evaluate (see step 5) • Active Storage uses background jobs • Alternatively, cloud function + cloud storage • But…
  29. 29. Active Storage on S3… ATM: No way to customize paths for human readable names :( no file extensions?!
  30. 30. 5. Downloading / Previewing
  31. 31. 5. Download / Preview • Viewing the original is simple • <%= image_tag @user.avatar %> • To get S3 url: @user.avatar.service_url • Return it in your API
  32. 32. Displaying Transformed Version • upload.variable? (i.e. images) • png, jpeg, etc (not SVG) • upload.previewable? (i.e. can get an image from them) • pdf, mpeg (video), etc
  33. 33. Variants • only applies to images, can change size • <%= image_tag @user.avatar.variant(resize: “400x400”) %> • 💡 dynamically generate an image size based on device • to link to full-size image simply do: • <%= link_to image_tag(@user.avatar.variant(resize: “400x400”)), @user.avatar %>
  34. 34. PictureThat AR App
  35. 35. PictureThat Backend
  36. 36. PictureThat Backend
  37. 37. Variants • ActiveStorage can transform(resize, etc) documents on the fly • in process • bad performance • works for small images only (due to connection timeouts) • in background process • dispatches a resize job
  38. 38. File Previews • As easy as this! • <%= image_tag file.preview(resize: '400x400') %> • Works for documents: pdf, videos! • Can build your own custom previewers
  39. 39. Behind the Scenes • hides your files behind a rails route • /rails/active_storage/representations/a84…3b/file_name.jpg • great for • switching between environments or mirrors • dynamically generating sizes • no more size having to call resize on all images in the database • lazily evaluated • bad for • performance, another call to rails as opposed to going to nginx or directly to S3
  40. 40. Advanced Transformation • none of these are supported by ActiveStorage • on-the-fly resize with s3 • AWS ServerLess-Thumbor -> https://github.com/ awslabs/serverless-image-handler • http://thumbor-server/filters:watermark(http:// my.site.com/img.png,10p,-20p,50)/some/image.jpg
  41. 41. Alternative: Shrine • Good • Been around for longer • Retains file extension on storage • Customizable with Plugins • Supports Cloud Processing using Plugins • Works with uppy.io out of the box • Bad • More work to setup
  42. 42. Alternative: No Rails… • Firebase Storage => easy to setup from an JS frontend • Storage as a Service: • https://transloadit.com/ • https://www.filestack.com/ • https://cloudinary.com
  43. 43. Conclusion
  44. 44. Conclusion • ActiveStorage shows potentials and works for simple cases (Admin interfaces) • For all other, start with Shrine • Don’t over-optimize. But keep your options open
  45. 45. References • https://edgeguides.rubyonrails.org/active_storage_overview.html • https://gorails.com/episodes/file-uploading-with-activestorage-rails-5-2 • https://gorails.com/episodes/how-to-create-an-active-storage-previewer • https://philsturgeon.uk/api/2016/01/04/http-rest-api-file-uploads/ • https://medium.com/typecode/a-strategy-for-handling-multiple-file-uploads-using-javascript- eb00a77e15f • https://www.netlify.com/blog/2016/11/17/serverless-file-uploads/ • https://bibwild.wordpress.com/2018/10/03/some-notes-on-whats-going-on-in-activestorage/ • https://github.com/rails/rails/issues/31419 • https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws- lambda-and-amazon-api-gateway/ • https://github.com/texpert/shrine-lambda
  46. 46. Questions? • twitter: @yagudaev, @nano3labs • come talk shop, tell me about your Rails project • Our friends @spacelist are hiring remote F/T senior ruby/rails dev

×