O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

高速・省メモリにlibsvm形式で ダンプする方法を研究してみた

3.600 visualizações

Publicada em

JapanR 2016でのLTのときの資料です。
xgboost::xgb.DMatrixを作るときのメモリ消費量を抑えるために、一旦libsvm形式のファイルを出力してから読み込む方法を使うようにしました。このとき高速にlibsvm形式でダンプするために、data.table::fwriteを魔改造したと言うお話です。

Publicada em: Engenharia
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

高速・省メモリにlibsvm形式で ダンプする方法を研究してみた

  1. 1. libsvm 2016/11/27 hskksk @ JapanR 2016
  2. 2. • : • : R, Python, C++ • : • :
  3. 3. xgboost kaggler
  4. 4. Bosch Production Line Performance 15
  5. 5. xgboost
  6. 6. xgb.DMatrix
  7. 7. # feature label = readRDS("label.rds") feature_set_A = readRDS("feature_set_A.rds") feature_set_B = readRDS("feature_set_B.rds") # feature cbind mat = cbind( feature_set_A, feature_set_B ) ↑ # DMatrix dmat = xgb.DMatrix(mat, label=label)
  8. 8. cbind
  9. 9. ※cbind rm(vars); gc()
  10. 10. xgb.DMatrix
  11. 11. Python
  12. 12. libsvm ※R
  13. 13. 1. cbind libsvm 2. DMatrix
  14. 14. cbind libsvm data.table::fwrite_libsvm(list_of_matrices, file)
  15. 15. data.table fork fwrite
  16. 16. # feature label = readRDS("label.rds") feature_set_A = fread("feature_set_A.csv") feature_set_B = fread("feature_set_B.csv") # feature list # 1 label matrices = list(label, feature_set_A, feature_set_B) # libsvm fwrite_libsvm(matrices, "libsvm.txt") # DMatrix dmat = xgb.DMatrix("libsvm.txt")
  17. 17. fwrite OpenMP 8.5GB/120sec @ Xeon 2.5GHz ✕ 8
  18. 18. data.table PR
  19. 19. https://github.com/hskksk/data.table kaggler !!
  20. 20. Enjoy Kaggling with R !!

×