O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

KDD2014勉強会 発表資料

3.316 visualizações

Publicada em

Li, Mu, et al. "Efficient mini-batch training for stochastic optimization." Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2014.

http://www.cs.cmu.edu/~muli/file/minibatch_sgd.pdf

KDD2014勉強会関西会場: http://www.ml.ist.i.kyoto-u.ac.jp/kdd2014reading

Publicada em: Dados e análise
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

KDD2014勉強会 発表資料

  1. 1. Efficient Mini-batch Training for Stochastic Optimization Li, M., Zhang, T., Chen, Y., & Smola, A. J. @ KDD2014読み会 関西会場 京大石井研 M2 小山田創哲 @sotetsuk
  2. 2. 紹介する論文
  3. 3. SGDと最近の傾向 ビッグデータと計算機性能の向上を背景に 膨大なパラメータ数の識別器を,大量のデータで SGD(Stochastic Gradient Descent) によって学習する場面が目立つ 例えば Deep learning [Taigman et al., 2014] [Le et al., 2012] [Szegedy et al., 2014]
  4. 4. SGD Stochastic Gradient Descent 【問題】次のコスト関数の最小化をしたい Ex. 【SGD】次のように逐次的に最適化
  5. 5. MSGD Mini-batch SGD
  6. 6. Distributed implementation 応用上,パラメータ数やサンプルサイズが 膨大な場面では,分散処理が行われる. 【問題点】更新式の実行毎にコミュニケー ションコストが発生する コミュニケーションの回数を減らして, たくさんのデータをみるため,batch size を大きくしたい
  7. 7. MSGDのトレードオフ コミュニケーションコストを減らして,たくさん データをみるため,batch sizeを大きくしたいが, batch sizeを大きくしすぎると,たくさんのデー タをみても,なかなか学習が進まなくなる. n コスト関数が凸な場合について,理論的に収束 が遅くなることが示される [Dekel et al., 2012]
  8. 8. 提案手法 Batch sizeを大きくしても学習が遅くならない MSGDを提案 基本アイディア: Mini-batch毎の一回の更新の質を向上させる 各mini-batchにおけるパラメータの更新を 正則化項を含むコスト関数の最小化問題の近似 だと捉える(次で説明)
  9. 9. 提案手法の基本アイディア Mini-batch毎の一回の更新の質を向上させる は と等価.さらにこれを [ ]の中をwで微分 して0とおくと… の一次近似とみなす.これをMini-batch毎に解く. Mini-batch毎の更新を正則化項を含むコスト関数の最小化問題 (部分問題)へと発想を逆転させる
  10. 10. EMSO(提案手法) 各mini-batch毎に定義される部分問題を また勾配法で解く n EMSO-GD n Gradient Descentで解く n EMSO-CD n Coordinate Descentで解く 図はh"p://en.wikipedia.org/から参照
  11. 11. 並列計算でのEMSO Mini-batchのデータを分割して, それぞれについて をEMSO-GDまたはCDで 並列に計算し,それらの平均をとって更新する
  12. 12. 数値実験 n データセット n KDD04 n URL n CTR n モデル n Logistic regressionによる2値分類 表は論文より参照
  13. 13. 単一ノードでの結果 URLデータセット EMSO-CD が10倍早い すべて10^7サンプル学習後 Batch sizeは各手法 毎に最適なものを使用 図は論文より参照
  14. 14. 複数ノードでの結果 単一ノードの時と同じく EMSO-CDが10倍早い ノード数を変えた時 EMSO-CDのコストが 特定の値まで下がるまでに 要した時間の比較 図,表は論文より参照
  15. 15. 総括 【手法】 Mini-batch毎の更新を,正則化項を含むコス ト関数の最適化問題で置き換え,より繊細に更新を 行うことによって,batch-sizeを大きくしても 学習が遅くならない手法を提案している 【理論】 凸なコスト関数を仮定した場合,bが大きい時の収 束速度の改善が証明されている(今回触れず) 【数値実験】 二値分類Logistic regressionにおける数値実 験でEMSO-CDが他の手法より高い性能を示した

×