O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Argoによる機械学習実行基盤の構築・運用からみえてきたこと

6.012 visualizações

Publicada em

機械学習は大量のリソースを必要とするためマネージドな環境をスケーラブルに使えることが重要です.また,機械学習システムは専門性が高く属人化してしまうことも事実です.これらの課題を解決するためにリクルートテクノロジーズではArgoワークフローを導入し,属人化や環境依存性が高いといった問題に取り組んでいます.Argoはコンテナベースのワークフローエンジンで,ワークフローの各ステップをコンテナとして実装することを可能にします.また, 弊社ではArgoプロジェクトに含まれるArgoCDと組み合わせることで,CI/CD パイプラインを構築し,GitOps を実現しています.Argo Workflowsによる機械学習実行基盤を運用していく上で見えてきた,使用者側の視点・運用する側の視点両方からの知見を共有します!
[1C1] 13:20 - 14:00
河野 晋策 (Shinsaku Kono) / 機械学習エンジニア/データサイエンティスト/検索エンジニア, リクルートテクノロジーズ
稲村 秀人 (Hideto Inamura) / システムアーキテクト, リクルートテクノロジーズ
Cloud Native Days Tokyo 2019 / OpenStack Days Tokyo 2019
#CNDT2019 / #OSDT2019
https://cloudnativedays.jp/cndt2019/

cndt2019 / osdt2019 / cndt / osdt / argo / workflow engine / workflow / gitpos / argo workflows / argo cd / 機械学習 / 機械学習実行基盤 / リクルート / リクルートテクノロジーズ / mlops / data science / machine learning / deep learning / google cloud platform / 深層学習 / cloud native days tokyo 2019 / cloud native days / open stack days tokyo 2019 / open stack days / digdag / argo events

Publicada em: Tecnologia
  • You can hardly find a student who enjoys writing a college papers. Among all the other tasks they get assigned in college, writing essays is one of the most difficult assignments. Fortunately for students, there are many offers nowadays which help to make this process easier. The best service which can help you is ⇒ www.HelpWriting.net ⇐
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • If you’re struggling with your assignments like me, check out ⇒ www.WritePaper.info ⇐. My friend sent me a link to to tis site. This awesome company. After I was continuously complaining to my family and friends about the ordeals of student life. They wrote my entire research paper for me, and it turned out brilliantly. I highly recommend this service to anyone in my shoes. ⇒ www.WritePaper.info ⇐.
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • (Unlimited)....ACCESS WEBSITE Over for All Ebooks ................ accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn }
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • (Unlimited)....ACCESS WEBSITE Over for All Ebooks ................ accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn }
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THAT BOOKS/FILE INTO AVAILABLE FORMAT - (Unlimited) ......................................................................................................................... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes Christian, Classics, Comics, Contemporary, Cookbooks, Art, Biography, Business, Chick Lit, Children's, Manga, Memoir, Music, Science, Science Fiction, Self Help, History, Horror, Humor And Comedy, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Argoによる機械学習実行基盤の構築・運用からみえてきたこと

  1. 1. G A A Q Q R D Google Kubernetes Engine MR N S C T O D 2 # 9 @2 0 A 0 / # :. 01 0 / # / # . / # 19 @2:0 A 20 7 1:2 #
  2. 2. )0 • A 0 J i s o d pl8D CC A C J ot l J E ) • J C l s ber4 J E J B A I • R R M- c u l s dr s T n kh S R l/ 1 / IJ D CA ACAJ 1E AE AE R i k S 2 22 0 ) ( 0 ? @9 : u u R u R R u R R ) ( & R P R R P s g a dP m l sch P y y P
  3. 3. ( ) 3 1 ? @ 0 00 0 : 7 / > > - 2 5 3 015 38
  4. 4. • ? ? • ? C ? CT • U A ? A / ? C UWL • ? / ? C p • ? / ? C • D • U D 4 • k lGcWtrw Ff e ibl • G W • G T A ? A / ? C • P ys W • ? xv I • G W ys ( E I • - T • m jGdn • / F gka c o F m • I A 23 . • ? / ? C - • ? D ) • ? A - • ? / ? C - • ? - 4 3 5 1
  5. 5. • • • • ?A • E • - • -
  6. 6. Argoproj is a collection of tools for getting work done with Kubernetes. • Argo Workflows - Container-native Workflow Engine • Argo CD - Declarative GitOps Continuous Delivery • Argo Events - Event-based Dependency Manager • Argo Rollouts(NEW!) - additional deployment strategies https://argoproj.github.io/argo/
  7. 7. O D CR W G 7 K D 7 A
  8. 8. L + - - - • arfi - - - - 8- + -, +? • arfi 8 gc or Wb n c M ehW • kp cfel arfi E cfel cfels w V kp tEd ca h v
  9. 9. )G E EGA ELH - 8 G H • ) EG FH 8H 8G8 ED E LEGA ELH • )G 8 H FFEG • )G 8 EG / G8L • F K DF E F H • 8G 8 H F8G8C GH • 1EEFH • 8G8C G 8 ED • ED ED8 H • C E H H F LEGA EL K • 3 G H F LEGA EL K • 3 H C C CE • HF D 3 H C & • 8D 8 ED • 0(H G HE G EG H G8 ED • , /EEAH DE 8 EDH 8D F • 8G 8 E ED E ECF LEGA EL • D • 8 D E G8 EDH DE H EGH • E C H F C G8 H D • 8G8 HC C H • 8 CED H FH • D E A G D E A G • G F H FH
  10. 10. 1 - - 1 H E
  11. 11. - 12 E
  12. 12. 1 3
  13. 13. • A F E • c d • c - - - - - • We a • • c We a
  14. 14. 5 / 1 /
  15. 15. 2 6 C B C 2 C 2 11 Co B
  16. 16. 7 1 1 • c P • S a E • • A I • c •
  17. 17. 8 - - -1 - 1- -1 • e U D • ca k nt D • g c g c L • a EU D T • D C g g cP • p i r i L S I I • a EU • g T IP a - 1- -1 • DI g c • g - -- - L E
  18. 18. :9- 8. - 2 / 02 10 • 9 9 1
  19. 19. 0 :9- 8. - 2 / 02 10 • 0 2 • 8 L l I m k 8 S m g l j d j o • j • j j j iy l k b l j a j C j k s j G C j k
  20. 20. e a ui g • L 1 2 fa • 1 1 p c 1 1 1 • 1 1 D n • ( ) ( • e 1 og 1L 1 S m
  21. 21. sm Csm s LI lknt t ( ) ( ( y ) sa p l o 8 l s l l 8 R h y D S
  22. 22. 3 2 3 2
  23. 23. • / • E D • IC • L
  24. 24. L I ) 3 2L 5 E r C reh l C jc L • ro_m C • ( r in p C • 0:: 9 /. 1: :/.0 .6 2 / 9 ) (- ) 0: 3 reh P 1/ V jc
  25. 25. /2 ( & (& &( ) 3 2 • steps.{step-name}.outputs.parameters • steps.{step-name}.outputs.artifacts
  26. 26. 8 4 8 4 1 2 4 / 8 4 • .99 8 ::: 9 71/9 5 0 /9 79/ 2 8 ( 4 :8 ) .9 2 • .99 8 ::: 9 71/9 5 0 /9 79/ 2 8 4 :8 .9 2
  27. 27. 2 /4
  28. 28. T 3E / 0 P S P 4
  29. 29. C G 3 1 / 4 RC A F E O . PI : AWS GCS Minio : : / .
  30. 30. 3 2 W / 1 2 W / 1 / % %
  31. 31. 3 2/ 2 A B /
  32. 32. 4 / H S H B A R E R / 3
  33. 33. AC ( 3 3 ( ( ) 5 / 2 1 5 (
  34. 34. 6 / 2 2 3 • DC MO C z AC CK • D g mbR V rP D BK • Snv S S vR C E DC DC DCBK • A DB K • lI kIS R & C BK • D A C B gc b f b SYhI uRMH / 2 • C f b RL st D A C B i IRo Pp • DBC G R , S x • i I SaekY • p S x • /
  35. 35. 3 . • )/ • ( )/ • )/ • )/ • )/
  36. 36. 3 )3 138 (/ ( 2 /32 • ( 2 /32 )3 138 A • ( ( ES -1 g t w t • l AdeE ) yf yf A Rn T kpAES • o DP ES uBr h uBr h uBr h h h h
  37. 37. 3 3 / 39 2 T / 3 A 3 W P ( ( )) 2 T
  38. 38. E /0 G P - / Ec A • / -/ E T a C A PE S • 00 4 / 4 4 / / /- E UG TC
  39. 39. / 3 1 / 3 1 4 / 1 A
  40. 40. / 3 / 3 4 2 / A • • S G G • C M G
  41. 41. 3 4 4 • h E A N • /. GH C H 3. T c ge ./ 3 /. i /. a • / 4 i • . 4 /.
  42. 42. 21 /1 G G A io s h e W g h ea ei P b h m rlE d g k nu P bg h E p ht argo submit -f ci/hello.yaml git clone spec: templates: - name: approve suspend: {} 2 4
  43. 43. 2 / / n / D w P p t / ID C r a c e o d d / Ck d i hm / Csp DC Cn - name: build sidecars: - name: dind image: docker:18-dind securityContext: privileged: true mirrorVolumeMounts: true container: image: docker:18 env: - name: DOCKER_HOST value: 127.0.0.1 command: - sh - -c args: [" until docker ps; do sleep 3; done; ¥ docker build -t example . && ¥ docker save example > example.tar "] I 4 C 5 DA
  44. 44. 1 26 D C SO e D f g d PR R ia Be cCe A C i a Be Argo CD Sealed Secrets Controller Secrets Sealed Secrets Custom Resource Kubernetes API Server apply sync D 4 1241 / 24 : 4 / A 1 C A
  45. 45. / cg S A l n 2 . S 2 RFR voS t production develop spec: destination: server: https://kubernetes.default.svc source: targetRevision: master spec: destination: server: https://kubernetes.default.svc source: targetRevision: develop kunm Svo a e bgSrh P 2 1 py KJSr 2 dd o si o apiVersion: batch/v1 kind: Job metadata: generateName: post-sync- annotations: argocd.argoproj.io/hook: PostSync 47 7 4 2 4
  46. 46. 5 4 .
  47. 47. A 9 / / I 4/ /4 1 31
  48. 48. w s A / i g l / 5 32 i l l / 5 32 / / e t • 5 32 r vo k n • l e • TMG c E M gd G E c G MiN M M Ma fk 5 2 / 0t • E E / 5 32 u fk o r • TMG M e TA E / 5 32 p o
  49. 49. u Wo P 5 hEe la / 9 : C l iDat v R P PESE • kD cjD m W Ps SE • - • • / 9 : C.3 : 3 T P r S PE • 57 A1 2 5 5 7 A3 • 57 A1 2 5 5 A:: kD cjD bhgD W s E p . W :A 3f la bhgD E E w W s SE
  50. 50. 1 0 r @ A i ? 7 e d o h Q ng 0
  51. 51. 4 + 5 55 4 RS g u c g 4 o CTA T apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: my-project spec: description: Example Project sourceRepos: - '*' destinations: - namespace: guestbook server: https://kubernetes.default.svc clusterResourceWhitelist: - group: '' kind: Namespace namespaceResourceBlacklist: - group: '' kind: ResourceQuota 4 i P 4 mCTA R G p P tl m 44 g h s BG T D 44 4 5 O e r j apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd labels: app.kubernetes.io/name: argocd-cm data: dex.config: | connectors: - type: oidc id: google name: Google config: {} helm.repositories: | - url: https://kubernetes- charts.storage.googleapis.com name: sealed-secrets

×