This document summarizes a presentation about deploying millions of emails per hour using cloud-based CI/CD practices. It discusses using App Engine and SendGrid for email delivery infrastructure, integrating CI (continuous integration) and CD (continuous delivery) with tools like Gitlab, Cloud Source Repositories, and Stackdriver Debugger. It also provides a demo of these practices and learning resources for email deliverability best practices, SendGrid documentation, and GCP services like Cloud Source Repositories and Stackdriver Debugger.

1. 60分鐘完送百萬EDM，背後
雲端CI/CD實戰大公開

2. HELLO!
I am Cage Chung
I am here because I like to share my experiences.
You can find me at http://kaichu.io / QNAP 雲端應用部資深工程師

3. https://www.facebook.com/groups/GCPUG.TW/
https://plus.google.com/u/0/communities/116100913832589966421
[您知道”GCPUG”要怎麼唸嗎？為什麼會有一隻狗在 Logo裡面呢？]
Google Cloud Platform User Group的縮寫是GCPUG
GCPUG直接唸成G.C.P.U.G？當然可以！
但它也可以分開來，唸成 G.C. PUG喔～
Pug，指的是巴哥犬，所以 GCPUG的Logo中間才會有一隻可愛的巴哥犬喲。
下次聽到別人說G.C. PUG 的時候，您就可以大聲 說：「我也是G.C. PUG社團成員！」

4. 17th Meetup

5. 大綱
◉ EDM 二三事 & 送信服務架構
◉ CI (持續整合) & CD (持續部署) 整合
◉ Demo/Cloud Source Repositories, Stackdriver Debugger
◉ 小提示 & 學習資源

6. EDM 二三事 & 送信服務架構
Let’s start with the first set of slides 1

7. 寄送 Email 好簡單 - API Call
# Google App Engine Send Email example (Python)
from google.appengine.api import app_identity
from google.appengine.api import mail
def send_approved_mail(sender_address):
# [START send_mail]
mail.send_mail(sender=sender_address,
to="Albert Johnson <Albert.Johnson@example.com>",
subject="Your account has been approved",
body="""Dear Albert:
Your example.com account has been approved. You can now visit
http://www.example.com/ and sign in using your Google Account to
access new features.
Please let us know if you have any questions.
The example.com Team
""")
# [END send_mail]
send_approved_mail('{}@appspot.gserviceaccount.com'.format(app_identity.get_application_id()))

9. 客戶最關心的事 - 送達率
Sender
(Bob)
Recipients
(Alice / Sunrain)

10. Email Reputation
Sender Score Average Delivered
0-60 21%
61-70 67%
71-80 68%
81-90 92%
91-100 95%
[Your Email Reputation](http://www.slideshare.net/emaildelivered/email-reputation-2)
[5 Ways to Check Your Sending Reputation | SendGrid](https://sendgrid.com/blog/5-ways-check-sending-reputation/)
◉ Authentication
◉ Bounce management
◉ List cleanliness
◉ User engagement (recipient
feedback)

11. EDM 實際會面臨到的種種問題...
Email Reputation
Spamminess
WhiteList
Shared IP
DK(IM)/SPF
Blocks
Invalid Emails
Blacklists
SpamDedicated IP
Scores

12. [Best Practices in Email Deliverability - Act-On Software](https://www.act-on.com/ebook/email-marketing-deliverability-ebook/)

14. [垃圾邮件检测结果](http://www.mail-tester.com/web-eBR3ar&lang=zh)
Mail Tester

15. Sendgrid 如何幫我們送信
[Email Message Flow Explained - SendGrid | SendGrid](https://sendgrid.com/docs/Classroom/Basics/Email_Infrastructure/email_flow.html)

16. Sendgrid 遊戲規則
[IP Warmup Schedule - SendGrid Documentation | SendGrid](https://sendgrid.com/docs/API_Reference/Web_API_v3/IP_Management/ip_warmup_schedule.html)

17. [SendGrid | Heroku Dev Center](https://devcenter.heroku.com/articles/sendgrid)

18. 送信服務架構

19. 我們所使用到的資源
AppEngine
BigQuery
CloudStorage
DataStore
CloudEndpoints
Sendgrid
A fast, economical and fully
managed data warehouse for
large-scale data analytics
BigQuery
Cloud Datastore is a highly-
scalable NoSQL database for
your web and mobile
applications.
DataStore
A powerful platform to build
web and mobile apps that
scale automatically.
App Engine
A Powerful, Simple and Cost
Effective Object Storage
Service.
Cloud Storage
Delivering your transactional
and marketing email through
one reliable platform.
Sendgrid
Create RESTful APIs on App
Engine accessible to iOS,
Android and Javascript
clients.
Cloud Endpoints

20. AppEngine
BigQuery
CloudStorage
DataStore
CloudEndpoints
Sendgrid
UI 彈性化擴展層、排程器、 Tasks
處理。針對不同任務的機器予給優
化的資源，並可自動進行擴展
EDM Server

21. AppEngine
BigQuery
CloudStorage
DataStore
CloudEndpoints
Sendgrid
Cloud logging / sendgrid webooks
(tracking - bounce, click, open,
spam) 的資料值串接 BigQuery, 作
後序分析
資料分析

22. AppEngine
BigQuery
CloudStorage
DataStore
CloudEndpoints
Sendgrid
檔案暫存
EDM html 檔案、連絡人清單暫存區

23. AppEngine
BigQuery
CloudStorage
DataStore
CloudEndpoints
Sendgrid
RESTful API endpoint
提供 RESTful API 給前端應用程式

24. https://<gcp-project-id>.appspot.com/_ah/api/explorer
http://localhost:8080/_ah/api/explorer

25. AppEngine
BigQuery
CloudStorage
DataStore
CloudEndpoints
Sendgrid
Sendgrid
送信服務商，提供豐富的報表、
EDM 相關的 know how，亦有
webhook 可以串接

26. AppEngine
BigQuery
CloudStorage
DataStore
CloudEndpoints
Sendgrid
Metadata 儲存
Nosql datastore. 儲存 EDM 送信服
務過程中相關 Metadata. 連絡人清
單、Log 等

27. 架構
queue
MetadataEDM Server
App Engine
Task Queues
Cloud Storage
endpoints API
(OAuth 2)
Cloude
Datastore
CSV
uploadrecipients
Store/retrieve
metadata
RESTful APIs for recipients
m
ail send
cron
...

28. # dispatch.yaml
application: your-project-id
dispatch:
- url: "*/favicon.ico"
module: default
- url: "your-project-id.appspot.com/"
module: default
- url: "*/tasks/*"
module: tasks
- url: "*/googleae8f4bcce8bec00c.html"
module: ownership
- url: "*/*"
module: "default"
# default/app.yaml
automatic_scaling:
min_idle_instances: 1
# tasks/app.yaml
instance_class: F4
automatic_scaling:
min_idle_instances: 2
min_pending_latency: 30ms
max_pending_latency: 300ms
# max_concurrent_requests: 80
# max_idle_instances: automatica # default value
# min_pending_latency: 30ms # default value
# max_pending_latency: automatic
# max_concurrent_requests: 50

29. EDM Server
EDM 檔案
聯絡人清單
Queue
API
Retry
fail
Retry cron
Schedule cron Success
Log
流程示意圖 failLog

30. CI(持續整合) & CD (持續部署) 整合
Let’s start with the second set of slides 2

31. Image by[每个架构师都应该研究下康威定律](http://www.infoq.com/cn/articles/every-architect-should-study-conway-law)

32. Devops: CI & CD (Docker, gitlab, GCP)
Bitbucket
Github
Jenkins
Travis CI
CircleCI
Docker
Gitlab
Cloud Source
Repositories
Stackdriver
Debugger

33. Origin imgae from and modified [DevOps：持續整合＆持續交付（Docker、CircleCI、AWS）](http://blog.amowu.com/2015/04/devops-continuous-integration-delivery-
docker-circleci-aws-beanstalk.html)

34. 簡單腳本
◉ Local Development / Test
◉ Gitlab-ci-multi-runner
◉ Local runner exec test - docker
◉ Gitlab runner test docker
◉ gcloud deploy
◉ Google App Engine
◉ Google Cloud Source Repositories
◉ Stackdriver Debugger

35. Gitlab Project Runner

36. Gitlab Project Variables

37. Register Gitlab runner
$ gitlab-ci-multi-runner register
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci):
https://gitlab.com/ci
Please enter the gitlab-ci token for this runner:
<your-gitlab-project-runner-register-token>
Please enter the gitlab-ci description for this runner:
[Kai-Chus-MBP]: edm
Please enter the gitlab-ci tags for this runner (comma separated):
gae,python,edm
Registering runner... succeeded runner=kvwzB1cE
Please enter the executor: ssh, shell, parallels, docker, docker-ssh, virtualbox:
docker
Please enter the default Docker image (eg. ruby:2.1):
cage1016/gitlab-ci-gcp:v1.6
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

38. .gitlab-ci.yaml
image: cage1016/gitlab-ci-gcp:v1.6
before_script:
- export CLOUDSDK_CORE_DISABLE_PROMPTS=1
- export CLOUDSDK_PYTHON_SITEPACKAGES=1
- export GCP_PROJECT=cage-20160705-edm
- export GCP_PROJECT_VERSION=uat
types:
- test
- deploy
test:
stage: test
script:
- sh ./scripts/tests.sh
deploy:
stage: deploy
script:
- sh ./scripts/deploy.sh
only:
- develop

39. .gitlab-ci.yaml (./scripts/tests.sh)
#!/usr/bin/env bash
virtualenv env
source env/bin/activate
# install Google App engine test env packages from requirements.testing.txt
pip install -r requirements.testing.txt
echo $GCLOUD_SERVICE_JSON_KEY > tasks/cage-20160705-edm-cc07ec5bf115.json
# run tests
py.test tasks/tests

40. .gitlab-ci.yaml (./scripts/deploy.sh)
#!/usr/bin/env bash
echo $GCLOUD_KEY > key.json
gcloud auth activate-service-account $GCLOUD_ACCOUNT --key-file key.json
gcloud --quiet config set project $GCP_PROJECT
# tasks module
pip install -r tasks/requirements.txt -t tasks/lib/
gcloud --quiet preview app deploy tasks/app.yaml --no-promote --version $GCP_PROJECT_VERSION --project
$GCP_PROJECT
# gen-repo-info-file
gcloud app gen-repo-info-file
git push google develop

41. Local runner exec test docker
// [install gitlab-ci-multi-runner](https://goo.gl/a99Aa4)
// list and register gitlab-ci-runner
$ gitlab-ci-multi-runner list
Listing configured runners ConfigFile=/Users/cage/.gitlab-runner/config.toml
edm Executor=docker Token=<your-token> URL=https://gitlab.com/ci
// prepare a docker-machine for gitlab-ci-multi-runner
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
runner - virtualbox Running tcp://192.168.99.100:2376 v1.11.0
// run a gitlab-ci-multi-runner test via docker
$ gitlab-ci-multi-runner --debug exec docker test --env "GCLOUD_KEY=$GCLOUD_KEY" --env " GCLOUD_ACCOUNT=
$GCLOUD_ACCOUNT" --env="GCP_PROJECT_VERSION=$GCP_PROJECT_VERSION"

42. $ git remote add google https://source.developers.google.com/p/[PROJECT_ID]/r/default
// generates a file named source-context.json, which contains information about the version of the source
code used to build the application
$ gcloud preview app gen-repo-info-file
// push code to google cloud source repositories
$ git push google master Google Cloud Source Repositories - List

43. Google Cloud Source Repositories - Source Code

44. Stackdriver Debugger | Source Code

45. Stackdriver Debugger | Debug

46. Image by [每个架构师都应该研究下康威定律](http://www.infoq.com/cn/articles/every-architect-should-study-conway-law)

47. Demo/Cloud Source Repositories,
Stackdriver Debugger
Let’s start with the third set of slides 3

48. 小提示 & 學習資源
Let’s start with the fourth set of slides 4

49. “5G Google cloud storage for App
engine for FREE
<project-id>.appspot.com
staging.<project-id>.appspot.com

50. “App Engine Admin API
Cloud Source Repositories API
Google Cloud Storage

51. 學習資源
◉ [Best Practices in Email Deliverability - Act-On Software](https:
//www.act-on.com/ebook/email-marketing-deliverability-ebook/)
◉ [Getting Started - SendGrid Documentation | SendGrid](https:
//sendgrid.com/docs/index.html)
◉ [Newsletters spam test by mail-tester.com](http://www.mail-
tester.com/)
◉ [Your Email Reputation](http://www.slideshare.
net/emaildelivered/email-reputation-2)

52. 學習資源
◉ Deploying to App Engine or Managed VMs from Travis CI
◉ [Deploying to App Engine or Managed VMs from Travis CI -
YouTube](https://www.youtube.com/watch?
v=7U4jjRw_AJk&feature=youtu.be)
◉ [GoogleCloudPlatform/continuous-deployment-demo](https:
//github.com/GoogleCloudPlatform/continuous-deployment-
demo)

53. 學習資源
◉ GCP
◉ [Cloud Source Repositories - Private Git Repositories —
Google Cloud Platform](https://cloud.google.com/source-
repositories/)
◉ [Stackdriver Debugger - Production Debugging - Google Cloud
Platform — Google Cloud Platform](https://cloud.google.
com/debugger/)
◉ [Managed VMs lab · kaichu.io](http://kaichu.
io/2015/09/managed-vms-lab/)

54. DevOps (Development & Operations)
Install the necessary apps with
one-click on demand and the
system will automatically finish
the set-up process
Quickly build a development
environment ideally suited for
collaboration
One NAS with Multiple projects
Recommended apps for quick installation : Private Registry, GitLab, and Jenkins.

55. THANKS!
Any questions?
You can find me at
http://kaichu.io / cage.chung@gmail.com