SF Big Analytics: Machine Learning with Presto by Christopher Berner
•Transferir como PPTX, PDF•
3 gostaram•2,148 visualizações
PrestoML allows users to perform machine learning tasks like classification and regression directly in SQL queries using Presto. It works by adding machine learning functions as Presto plugins that can be called from SQL. This allows ML tasks to be integrated with querying and analysis of data stored in databases and other data sources. Some key benefits are that ML code does not need to be written separately, and ML models can be stored directly as tables to be queried. The demo shows classifying emails as spam by learning a model directly from a SQL query. Future work may include more algorithms, hyperparameter tuning, and other ML techniques.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (15)
Semelhante a SF Big Analytics: Machine Learning with Presto by Christopher Berner
Semelhante a SF Big Analytics: Machine Learning with Presto by Christopher Berner (20)
Mais de Chester Chen
Mais de Chester Chen (20)
Último
Último (20)
SF Big Analytics: Machine Learning with Presto by Christopher Berner
- 1. PrestoML Machine Learning in SQL Christopher Berner
- 2. Outline 1. What is Presto? 2. Why ML in SQL? 3. Demo 4. How PrestoML works 5. Architecture of Presto 6. Future work
- 3. What is Presto? • Distributed SQL engine • Optimized for low-latency, interactive analysis • ANSI SQL • Extensible connector system
- 4. Why ML in SQL? Databases & Other Sources Databases & Other Sources Databases & Other Sources #include <svm.h> …
- 5. Why ML in SQL? Databases & Other Sources Databases & Other Sources Databases & Other Sources #include <svm.h> … VS CREATE TABLE spam_model AS SELECT learn_classifier(is_spam, text) FROM emails
- 6. Why ML in SQL? Databases & Other Sources Databases & Other Sources Databases & Other Sources #include <svm.h> … VS CREATE TABLE spam_model AS SELECT learn_classifier(is_spam, text) FROM emails
- 7. Why ML in SQL? Databases & Other Sources Databases & Other Sources Databases & Other Sources #include <svm.h> … VS CREATE TABLE spam_model AS SELECT learn_classifier(is_spam, text) FROM emails
- 8. Why ML in SQL? Databases & Other Sources Databases & Other Sources Databases & Other Sources #include <svm.h> … VS CREATE TABLE spam_model AS SELECT learn_classifier(is_spam, text) FROM emails
- 9. Demo Time!
- 10. How PrestoML works • Presto Plugin ▪ Functions ▪ Types • Regression • Classification
- 11. Planning a query CREATE TABLE spam_models AS SELECT user, learn_classifier(is_spam, text) model FROM emails GROUP BY user SQL query
- 12. Planning a query TableScan emails GroupedAggregatio n learn_classifier() CreateTable spam_models Logical plan CREATE TABLE spam_models AS SELECT user, learn_classifier(is_spam, text) model FROM emails GROUP BY user SQL query
- 13. Architecture of Presto • Data in-memory during execution • Specialized SQL operators • Adaptive scheduling
- 14. Connectors Coordinator Worker Parser / Analyzer Planner Scheduler Internal Cassandra MySQL Kafka Hive Metadata API Internal Cassandra MySQL Kafka Hive Data Location API Internal Cassandra MySQL Kafka Hive Data Stream API
- 15. Architecture Scheduler Data Location API Parser / Analyzer Planner Metadata API Coordinator Client Worker Worker Worker Data Stream API Data Stream API
- 16. Future work • More models • Feature extraction • Hyperparameter tuning • Feature selection • Clustering • Collaborative filtering
Notas do Editor
- A project to marry the power of machine learning to the simplicity of SQL Goal: provide a frictionless environment to prototype machine learning models
- Built for prototyping and adhoc analysis, because Hive was too slow Facebook scale: lots of data, lots of machines, multiple geo regions, data spread across multiple systems
- Data in MySQL, MongoDB, proprietary stores…etc ETL and feature engineering very time consuming Need to worry about normalizing data (csv, tsv, json, utf8…etc) Data then loaded into custom binary, often C++, Python, Java…etc
- Whole pipeline can be written in a few lines of SQL “How many of you not familiar with SQL?”
- Data is already available and normalized (utf8, utf16…etc)
- Feature extraction and training directly in SQL
- Models often get lost, since they were trained on a laptop or dev machine Audiences: myself as a SWE, analysts and other people who know SQL Presto is distributed so scaling is much easier than a custom binary
- Distributed planner could merge first two stages, if partitioned on user
- Not MR, more like Spark Pipelined to reduce latency, execution like a DAG Flat structures, and bytecode generation
- One or more coordinator machines Current connectors: Hive, Cassandra, MySQL, Kafka. More being added all the time. Uses plugins to resolve functions Locate data Access data
- Pick workers “near” the data Multiple stages