AWS Summit Seoul 2023 | 스타트업의 서버리스 기반 SaaS 데이터 처리 및 데이터웨어하우스 구축 사례
Denunciar
Compartilhar
•0 gostou•206 visualizações
AWS Summit Seoul 2023 | 스타트업의 서버리스 기반 SaaS 데이터 처리 및 데이터웨어하우스 구축 사례
•0 gostou•206 visualizações
Denunciar
Compartilhar
Baixar para ler offline
에이슬립은 AI 모델로 수면을 정확하게 분석하여 맞춤형 수면 서비스를 제공하는 SaaS 서비스를 제공합니다. 본 세션에서는 SaaS 서비스 인증 및 권한제어, 사용량 측정 등 필수적인 기능을 AWS 기반 서버리스 서비스를 통해 제공하여 데이터베이스 부하 없이 대규모 트래픽에도 견딜 수 있는 아키텍처를 소개합니다. 매드업은 디지털 마케팅 자동화 솔루션을 개발 및 운영하면서 대용량 데이터를 수집, 가공해서 데이터웨어하우스를 만드는 도전이 있습니다. 본 세션에서는 기존의 Amazon Athena에서 어떻게 Amazon Redshift Serverless를 통해 데이터가 필요한 조직에 정확하게 공급했는지 그 사례와 장단점을 소개합니다.
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados(20)
Similar a AWS Summit Seoul 2023 | 스타트업의 서버리스 기반 SaaS 데이터 처리 및 데이터웨어하우스 구축 사례
Similar a AWS Summit Seoul 2023 | 스타트업의 서버리스 기반 SaaS 데이터 처리 및 데이터웨어하우스 구축 사례(20)
![[보험사를 위한 AWS Data Analytics Day] 5_KB금융그룹과 계열사의 AWS 기ᄇ...](https://cdn.slidesharecdn.com/ss_thumbnails/awsdataanalyticsday5kbawsanalyticsjourneybyeong-eokkang-230608094829-c0103aff-thumbnail.jpg?width=768&fit=bounds)
Mais de Amazon Web Services Korea
Mais de Amazon Web Services Korea(19)
![[Keynote] 슬기로운 AWS 데이터베이스 선택하기 - 발표자: 강민석, Korea Database SA Manager, WWSO, A...](https://cdn.slidesharecdn.com/ss_thumbnails/d3s01aws-230704014400-3eeae447-thumbnail.jpg?width=768&fit=bounds)
AWS Summit Seoul 2023 | 스타트업의 서버리스 기반 SaaS 데이터 처리 및 데이터웨어하우스 구축 사례
- 1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. SEOUL | MAY 4, 2023
- 2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. SaaS Backend Developer
- 3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. / Liam Park System Softerware Engineer Backend Engineer AWS infra, Node JS And Python Developer TmaxTibero Software Engineer Carillon Backend Engineer Asleep, Senior Backend Engineer
- 4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Sleeptrack API 2. API 3. Sleeptrack API - / / 4. Sleeptrack API – 5.
- 5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1. Sleeptrack API
- 6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sleeptrack API
- 7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sleeptrack API
- 8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2. API
- 9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Saas API 01 AUTOMATION API 02 MINIMAL DEPENDENCY Live . 03 ACCURACY API . 04 SCALABILITY .
- 10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. client ECS server log RDS cloud watch API Gateway update per request ELB 1. ECS API 2. API DB 3. DB ,
- 11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3. Sleeptrack API / /
- 12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Athena • SQL S3 • csv, json, parquet • • RDS DocumentDB DynamoDB S3 Glue Athena
- 13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Athena 1. SERVERLESS 2. Athena 3. ( / / ) Spark/Flink Presto/Hive Athena RedShift 운영 효율성 Low Low High Medium Learning Curve High Medium Low Low 운영 비용 High Medium Low Medium
- 14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. API Firehose API Gateway API Firehose 01 02 03 04 Access log Parquet Data Athena Parquet Data Athena Firehose S3 JSON JSON Access Log Parquet Data
- 15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. API Client API Gateway ECS S3 access log Firehose RDS Athena Glue Schema ELB (1) / /
- 16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Firehose API Gateway Firehose
- 17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Access Log Firehose S3
- 18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Access Log Firehose S3
- 19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Parquet Data • • Athena S3 • 23MB JSON Data $25 JSON Data 500KB Parquet Data $5 Parquet Data Before After
- 20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Athena : 2023 5 4 2 Create Parquet access log Glue Schema Partitioned by Datetime parquet Schema For Parquet Athena CTAS Query Add Next Partition
- 21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. DB • Athena Jenkins DB Parquet Athena insert hourly Client API Gateway Jenkins aggregate query ECS RDS ELB
- 22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4. Sleeptrack API
- 23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Kinesis SQL Application • SQL • Kinesis Stream Kinesis SQL Application Firehose S3
- 24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. SQL Application
- 25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. API Firehose API Gateway API Firehose 01 02 03 SQL Application Lambda SQL Application lambda Firehose SQL Application
- 26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. API – Client API Gateway Firehose SQL Application Lambda RDS update per minute
- 27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. SQL Application Firehose SQL Application
- 28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. SQL Application SQL Application Lambda RDS
- 29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. SQL Application SQL Application Lambda RDS
- 30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5.
- 31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. DB API query string, header access log API access log SQL Application
- 32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. DW Jaeyeon Baek Tech Lead MADUP
- 33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ❏ Adtech(Advertising technology) ❏ Tech Lead at Data Platform Team ❏ Cloud GDE ❏ HashiCorp Ambassador 2023 ❏ : jybaek.tistory.com
- 34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. , ,
- 37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 05.06 install : 2,522 install : 2,599 install : 2,871 05.01 install : 2,501 06.06 06.06 . install 04 20 05.02 install : 2,872
- 38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. DW manifest delete copy commit download •data freshness •transaction size •distkey, sortkey
- 39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. , / / EKS , Pod . MWAA Redshift . ECS Fargate JupyterHub ( ) .
- 41. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. / / • • Amazon S3 , Redshift COPY bz2 ▪ raw data bucket: ▪ compressed data bucket : Redshift COPY ○ • MWAA DAG manifest Amazon Redshift Delete & COPY Internal Micro Services Amazon S3 Amazon Simple Queue Service Amazon EKS Amazon MWAA Amazon Athena Amazon Redshift
- 42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. / / average 30% • Average 30%, Maximum 90% • Counting only COPY queries. Redshift dc2.large
- 43. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. / / average 100% Normal Query • DM Load •
- 44. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. / / Redshift ETL Data Scientist Data Mart ETC
- 45. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. / / dc2.large to RA3 AQUA(Advanced Query Accelerator) 10 Amazon Managed Storage . ra3.xlplus 32TB Datashare Amazon Redshift
- 46. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 47. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Datashare Datashare AWS AWS . , Amazon Redshift
- 48. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Datashare Datashare … •inbound shares / outbound shares ▪ •producer cluster / consumer cluster ▪ •same AWS account or different AWS accounts ▪ . •Encryption ▪ producer, consumers ▪ .
- 49. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Datashare Datashare Amazon Redshift Managed Storage Producer Cluster (write data) consumer cluster (read shared data) consumer cluster (read shared data) Amazon Redshift Amazon Redshift Amazon Redshift
- 50. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 51. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Redshift Serverless DW , Amazon Redshift Serverless (2022-07 GA, ) (RPUs) , Amazon Redshift Serverless
- 52. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Redshift Serverless Datashare Redshift Serverless Amazon Redshift Managed Storage Producer Cluster (write data) consumer cluster (read shared data) consumer cluster (read shared data) Amazon Redshift Amazon Redshift Serverless Amazon Redshift Serverless
- 53. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Redshift Serverless Datashare Redshift Serverless Amazon Redshift Managed Storage Producer Cluster (write data) consumer cluster (read shared data) consumer cluster (read shared data) Amazon Redshift Amazon Redshift Serverless Amazon Redshift Serverless
- 54. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1 transaction Redshift Serverless - File (1) Table (1) x10 DELETE & COPY File (2) Table (2) DELETE & COPY File (32) Table (32) … … DELETE & COPY
- 55. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Redshift Serverless 타입 노드 스펙 월 비용 Delete & Insert 평균 시간 dc2.large x 8 - vCPU : 2 - Memory : 15 GIB - Storage : 0.16 TB 1,728 USD 224s ra3.xlplus x 2 - vCPU : 4 - Memory : 32 GIB - Storage : 32 TB 1,840.32 USD 201s ra3.xlplus x 3 - vCPU : 4 - Memory : 32 GIB - Storage : 32 TB 2,760 USD 145s Serverless x 32 RPU - RPU 시간 당 0.438 USD 매일 6시간 30일 사용 2,522.88 USD 102s Serverless x 64 RPU - RPU 시간 당 0.438 USD 매일 6시간 30일 사용 5,045.76 USD 95s - 32 (80MB) x 10
- 56. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Redshift Serverless DW Amazon MWAA Amazon Redshift Redshift Serverless Redshift Serverless RPUs Data Mart Data Scientist
- 57. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 58. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Wrap up • ▪ : DB scale-up. ▪ •Redshift Serveless RPUs ▪ ▪ / / • . ▪ ▪ ▪
- 59. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Wrap up Redshift • ▪ Athena •Athena ▪ ( Redshift ) ▪ (DAG) ▪ ETL •Athena ▪ Hive , , , and so on..
- 60. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Wrap up jybaek@madup.com :-) ?
- 61. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 감사합니다 © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.