Submit Search
Upload
Kiwipycon2011 async-with-gevent-redis
•
Download as ODP, PDF
•
11 likes
•
2,478 views
alexdong
Follow
Technology
Education
Slideshow view
Report
Share
Slideshow view
Report
Share
1 of 13
Download now
Recommended
API analytics with Redis and Google Bigquery. NoSQL matters edition
API analytics with Redis and Google Bigquery. NoSQL matters edition
javier ramirez
Building highly reliable data pipeline @datadog par Quentin François
Building highly reliable data pipeline @datadog par Quentin François
Paris Data Engineers !
Clickhouse at Cloudflare. By Marek Vavrusa
Clickhouse at Cloudflare. By Marek Vavrusa
Valery Tkachenko
Spark Summit EU talk by William Benton
Spark Summit EU talk by William Benton
Spark Summit
Speed Up Uber's Presto with Alluxio
Speed Up Uber's Presto with Alluxio
Alluxio, Inc.
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
InfluxData
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
InfluxData
DOWNSAMPLING DATA
DOWNSAMPLING DATA
InfluxData
Recommended
API analytics with Redis and Google Bigquery. NoSQL matters edition
API analytics with Redis and Google Bigquery. NoSQL matters edition
javier ramirez
Building highly reliable data pipeline @datadog par Quentin François
Building highly reliable data pipeline @datadog par Quentin François
Paris Data Engineers !
Clickhouse at Cloudflare. By Marek Vavrusa
Clickhouse at Cloudflare. By Marek Vavrusa
Valery Tkachenko
Spark Summit EU talk by William Benton
Spark Summit EU talk by William Benton
Spark Summit
Speed Up Uber's Presto with Alluxio
Speed Up Uber's Presto with Alluxio
Alluxio, Inc.
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
InfluxData
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
InfluxData
DOWNSAMPLING DATA
DOWNSAMPLING DATA
InfluxData
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Zhenxiao Luo
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Zekeriya Besiroglu
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Stephan Hochhaus
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Vianney FOUCAULT
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
Alluxio, Inc.
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Redis Labs
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Christopher Bradford
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
Alluxio, Inc.
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Sri Ambati
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Alluxio, Inc.
Getting to Know Airflow
Getting to Know Airflow
Rosanne Hoyem
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Zhenxiao Luo
Setting up InfluxData for IoT
Setting up InfluxData for IoT
InfluxData
Pig on spark
Pig on spark
Sigmoid
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Databricks
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Lviv Startup Club
Intro to hadoop
Intro to hadoop
Haden Pereira
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Dirk Petersen
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Databricks
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
KubeAcademy
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
grooverdan
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
DataWorks Summit
More Related Content
What's hot
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Zhenxiao Luo
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Zekeriya Besiroglu
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Stephan Hochhaus
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Vianney FOUCAULT
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
Alluxio, Inc.
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Redis Labs
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Christopher Bradford
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
Alluxio, Inc.
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Sri Ambati
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Alluxio, Inc.
Getting to Know Airflow
Getting to Know Airflow
Rosanne Hoyem
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Zhenxiao Luo
Setting up InfluxData for IoT
Setting up InfluxData for IoT
InfluxData
Pig on spark
Pig on spark
Sigmoid
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Databricks
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Lviv Startup Club
Intro to hadoop
Intro to hadoop
Haden Pereira
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Dirk Petersen
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Databricks
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
KubeAcademy
What's hot
(20)
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Getting to Know Airflow
Getting to Know Airflow
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Setting up InfluxData for IoT
Setting up InfluxData for IoT
Pig on spark
Pig on spark
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Intro to hadoop
Intro to hadoop
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
Similar to Kiwipycon2011 async-with-gevent-redis
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
grooverdan
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
DataWorks Summit
WE18_Performance_Up.ppt
WE18_Performance_Up.ppt
webhostingguy
Handout3o
Handout3o
Shahbaz Sidhu
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
DataWorks Summit/Hadoop Summit
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Bhupesh Bansal
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
Hadoop User Group
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Stuart Pook
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Cloudera, Inc.
Big data at United Airlines
Big data at United Airlines
DataWorks Summit
Eagle from eBay at China Hadoop Summit 2015
Eagle from eBay at China Hadoop Summit 2015
Hao Chen
Inroduction to Big Data
Inroduction to Big Data
Omnia Safaan
Hug syncsort etl hadoop big data
Hug syncsort etl hadoop big data
Stéphane Heckel
Syncsort et le retour d'expérience ComScore
Syncsort et le retour d'expérience ComScore
Modern Data Stack France
Composing re-useable ETL on Hadoop
Composing re-useable ETL on Hadoop
Paul Lam
Hands on Hadoop and pig
Hands on Hadoop and pig
Sudar Muthu
RAPIDS: GPU-Accelerated ETL and Feature Engineering
RAPIDS: GPU-Accelerated ETL and Feature Engineering
Keith Kraus
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
MLconf
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
DataWorks Summit/Hadoop Summit
Globo.com & Varnish
Globo.com & Varnish
lokama
Similar to Kiwipycon2011 async-with-gevent-redis
(20)
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
WE18_Performance_Up.ppt
WE18_Performance_Up.ppt
Handout3o
Handout3o
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Big data at United Airlines
Big data at United Airlines
Eagle from eBay at China Hadoop Summit 2015
Eagle from eBay at China Hadoop Summit 2015
Inroduction to Big Data
Inroduction to Big Data
Hug syncsort etl hadoop big data
Hug syncsort etl hadoop big data
Syncsort et le retour d'expérience ComScore
Syncsort et le retour d'expérience ComScore
Composing re-useable ETL on Hadoop
Composing re-useable ETL on Hadoop
Hands on Hadoop and pig
Hands on Hadoop and pig
RAPIDS: GPU-Accelerated ETL and Feature Engineering
RAPIDS: GPU-Accelerated ETL and Feature Engineering
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
Globo.com & Varnish
Globo.com & Varnish
Recently uploaded
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
RankYa
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
gvaughan
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
Slibray Presentation
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
Enterprise Knowledge
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
Miki Katsuragi
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
UiPathCommunity
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Zilliz
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
Ridwan Fadjar
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
Fwdays
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdf
SeasiaInfotech2
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
Alex Barbosa Coqueiro
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
2toLead Limited
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Mark Simos
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
Memoori
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
Stephanie Beckett
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
Fwdays
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
Fwdays
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Safe Software
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
Addepto
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
Scott Keck-Warren
Recently uploaded
(20)
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdf
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
Kiwipycon2011 async-with-gevent-redis
1.
Writing a
distributed crawler system using gevent and redis Alex Dong @alexdong
2.
3.
Async 101
4.
Gevent: the monkey
king
5.
Redis: data structure
server
6.
Lessons learned
7.
8.
How many links
today?
9.
What was the
project Google employee #1 working on?
10.
11.
DNS resolution
12.
Request HTTP Header
13.
Download full content
14.
Store to local
file store, database and index
15.
Scheduling, throttling, status
monitoring, scale up by flicking on more servers.
16.
17.
Yield on IO/socket,
but
18.
19.
Context switch overhead
20.
21.
Cooperative multitasking
22.
epollfd =
epoll_create ();
23.
epoll_ctl ( epollfd
, EPOLL_CTL_ADD, listen_sock , &ev)
24.
epoll_wait ( epollfd
, events , MAX_EVENTS, -1)
25.
26.
27.
28.
Working: Hashtable -
HSET, HDEL, HEXISTS
29.
One and only
one: Sets - SADD, SPOP
30.
31.
32.
25% code for
status update and monitoring
33.
34.
DNS is down?
35.
ISP's bandwidth?
36.
Large file download?
37.
Scheduler re-submit tasks?
38.
39.
40.
Carefully profile to
make sure socket.recv becomes the main bottleneck.
41.
The get_title
crisis
42.
Rule of thumb:
load average < 1 to saturate 10M bandwidth
43.
Twitter: @alexdong
trunk.ly/?q= from:alexdong + gevent
Download now