SlideShare a Scribd company logo
1 of 19
Pylons + Tokyo Cabinet
    by Ben Cheng from Oursky.com
About Us
• Founder of Oursky Ltd.
• Web application development services
  (Ruby on Rails, Django, Pylons, PHP,
  Wordpress, etc.);
• FONCEPT.com - a T-Shirt design sharing
  portal;
• Working on a SaaS workflow application.
Pylons
What is Pylons?

• Flexible, Modular, Extensible
• Built on established libraries
• Close to 1.0
• Foundation of TurboGears 2.0
Who is using Pylons?

• Bittorrent
• Reddit (code.reddit.com!)
• and hundreds of sites
WSGI
• WSGI (Web Server Gateway Interface)
 • Wrap a “app” inside another “app”
 • Middleware, e.g.:
   • Authentication
   • Beaker (Cache, Session)
   • Routes
The Stack
• Models: SQLAlchemy, CouchDB...
• Templates: Mako, Genshi, Jinja2, ...
• Helpers: Web Helpers, FormAlchemy, ...
• Request Dispatching: Routes, ...
• Session / Caching: Beaker, ...
• Deployment: Paste, mod_wsgi, Twisted, ...
Take advantage of
      Python Power
• TIOBE: Python is no. 8
• Such as:
 • Want interactive debugger? iPython
 • Logging? Chainsaw
 • Testing? Nose, MiniMock
 • ..... blah blah balh
Power of other Python
  Web Framework
• Tons of lib from Zope
• ToscaWidgets from TurboGears
• Authkit
• (except Django)
Community
• Pylons Book
• Quick response (e.g.):
 • We fired Ticket #620 at Pylons Trac
   • “init_model called twice in nosetests”
   • Bug get fixed in 2 weeks, committed to
      trunk and temp. solution are discussed.
Tokyo Cabinet
Introduction

• By mixi.jp (Facebook in Japan)
• TokyoCabinet (Engine)
• TokyoTyrant (Server)
• TokyoDystopia (Full-Text Search)
• Opensources, standard C
TokyoCabinet
• Key/value DB
• Insert: 0.4 sec / 1M records
• Search: 0.33 sec / 1M records
• hash / B+ Tree structure
• Key Value / Hash / B+ Tree / Fixed Length /
  Table
• Driver: Perl, Ruby, Python, PHP, Erlang...
TokyoTyrant

• Network server
• Binary / Memcache / HTTP protocol
• 17.2 sec / 1M queries
• Hot Backup, Master-to-Master replication
• Extensible by Lua
Lua - Extensions
function incr(key, i)
  i = tonumber(i)
  local old = tonumber(_get(key))
  i = old + i
  _put(key, i)
  return i
end
Lua - Extension
function expire()
  local args={}
  local cdata = string.format(“%d”, _time())
  table.insert(args, “addcond0x0NUMLE”..cdata)
  table.insert(args, “out”)

 local res = _misc(“search”, args)
 if not res then
    _log(“expire failed”)
 end
end
Extensible to...

• Inverted index
• Set operations
• (and we use it to do revision management)
LightCloud
• Distributed and persistent key-value
  database
• Opensource and used by Plurk
• Scale out by just adding nodes
• Nodes are replicated
   Elapsed for 10000 gets: 1.74538516998 seconds [memcache]
   Elapsed for 10000 gets: 3.57339096069 seconds [lightcloud]

   Elapsed for 10000 sets: 1.88236999512 seconds [memcache]
   Elapsed for 10000 sets: 9.23674893379 seconds [lightcloud]
Links

• http://www.pylonshq.com
• http://opensource.plurk.com/LightCloud/
• http://1978th.net/tokyocabinet/

• Reach me at bencheng@oursky.com

More Related Content

What's hot

Centralized Logging with syslog
Centralized Logging with syslogCentralized Logging with syslog
Centralized Logging with syslog
amiable_indian
 
Socket programming with php
Socket programming with phpSocket programming with php
Socket programming with php
Elizabeth Smith
 
Kettunen, miaubiz fuzzing at scale and in style
Kettunen, miaubiz   fuzzing at scale and in styleKettunen, miaubiz   fuzzing at scale and in style
Kettunen, miaubiz fuzzing at scale and in style
DefconRussia
 
Dicas e truques de otimização de websites python
Dicas e truques de otimização de websites pythonDicas e truques de otimização de websites python
Dicas e truques de otimização de websites python
Fabiano Weimar
 

What's hot (20)

Building GUI App with Electron and Lisp
Building GUI App with Electron and LispBuilding GUI App with Electron and Lisp
Building GUI App with Electron and Lisp
 
Using Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibanaUsing Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibana
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
 
Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB
 
ELK stack at weibo.com
ELK stack at weibo.comELK stack at weibo.com
ELK stack at weibo.com
 
Logstash-Elasticsearch-Kibana
Logstash-Elasticsearch-KibanaLogstash-Elasticsearch-Kibana
Logstash-Elasticsearch-Kibana
 
From zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and ElasticsearchFrom zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and Elasticsearch
 
Dexador Rises
Dexador RisesDexador Rises
Dexador Rises
 
On Centralizing Logs
On Centralizing LogsOn Centralizing Logs
On Centralizing Logs
 
Solving some of the scalability problems at booking.com
Solving some of the scalability problems at booking.comSolving some of the scalability problems at booking.com
Solving some of the scalability problems at booking.com
 
Caching Up and Down the Stack
Caching Up and Down the StackCaching Up and Down the Stack
Caching Up and Down the Stack
 
Centralized Logging with syslog
Centralized Logging with syslogCentralized Logging with syslog
Centralized Logging with syslog
 
Socket programming with php
Socket programming with phpSocket programming with php
Socket programming with php
 
Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open source
 
{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4
 
Kettunen, miaubiz fuzzing at scale and in style
Kettunen, miaubiz   fuzzing at scale and in styleKettunen, miaubiz   fuzzing at scale and in style
Kettunen, miaubiz fuzzing at scale and in style
 
XForms and eXist: A Perfect Couple
XForms and eXist: A Perfect CoupleXForms and eXist: A Perfect Couple
XForms and eXist: A Perfect Couple
 
Dicas e truques de otimização de websites python
Dicas e truques de otimização de websites pythonDicas e truques de otimização de websites python
Dicas e truques de otimização de websites python
 
Logstash family introduction
Logstash family introductionLogstash family introduction
Logstash family introduction
 
Web Development: The Next Five Years
Web Development: The Next Five YearsWeb Development: The Next Five Years
Web Development: The Next Five Years
 

Viewers also liked (7)

CU Geek Club 簡介
CU Geek Club 簡介CU Geek Club 簡介
CU Geek Club 簡介
 
Presentation at Hong Kong Start-Up Association Event
Presentation at Hong Kong Start-Up Association EventPresentation at Hong Kong Start-Up Association Event
Presentation at Hong Kong Start-Up Association Event
 
Slides for Teachers
Slides for TeachersSlides for Teachers
Slides for Teachers
 
Barcamp 2008 O Auth
Barcamp 2008 O AuthBarcamp 2008 O Auth
Barcamp 2008 O Auth
 
Hadoop New And Note - December 2010 TriHUG
Hadoop New And Note - December 2010 TriHUGHadoop New And Note - December 2010 TriHUG
Hadoop New And Note - December 2010 TriHUG
 
Fundraising & Crowdfunding - Crowdfunding “home made” - 5 passi per un sistem...
Fundraising & Crowdfunding - Crowdfunding “home made” - 5 passi per un sistem...Fundraising & Crowdfunding - Crowdfunding “home made” - 5 passi per un sistem...
Fundraising & Crowdfunding - Crowdfunding “home made” - 5 passi per un sistem...
 
The Outcome Economy
The Outcome EconomyThe Outcome Economy
The Outcome Economy
 

Similar to Pylons + Tokyo Cabinet

Ajax Tutorial
Ajax TutorialAjax Tutorial
Ajax Tutorial
oscon2007
 
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
adunne
 
Node js quick tour v2
Node js quick tour v2Node js quick tour v2
Node js quick tour v2
Wyatt Fang
 
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
curryon
 

Similar to Pylons + Tokyo Cabinet (20)

Ajax Tutorial
Ajax TutorialAjax Tutorial
Ajax Tutorial
 
Docker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12xDocker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12x
 
Node azure
Node azureNode azure
Node azure
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic Stack
 
Titanium Desktop Intro
Titanium Desktop IntroTitanium Desktop Intro
Titanium Desktop Intro
 
FOSDEM2018 Janus Lua plugin presentation
FOSDEM2018 Janus Lua plugin presentationFOSDEM2018 Janus Lua plugin presentation
FOSDEM2018 Janus Lua plugin presentation
 
Realtime Analytics with MongoDB Counters (mongonyc 2012)
Realtime Analytics with MongoDB Counters (mongonyc 2012)Realtime Analytics with MongoDB Counters (mongonyc 2012)
Realtime Analytics with MongoDB Counters (mongonyc 2012)
 
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
 
Multiprocessing with python
Multiprocessing with pythonMultiprocessing with python
Multiprocessing with python
 
Troubleshooting .net core on linux
Troubleshooting .net core on linuxTroubleshooting .net core on linux
Troubleshooting .net core on linux
 
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
 
Living in a multiligual world: Internationalization for Web 2.0 Applications
Living in a multiligual world: Internationalization for Web 2.0 ApplicationsLiving in a multiligual world: Internationalization for Web 2.0 Applications
Living in a multiligual world: Internationalization for Web 2.0 Applications
 
Node js quick-tour_v2
Node js quick-tour_v2Node js quick-tour_v2
Node js quick-tour_v2
 
Node js quick-tour_v2
Node js quick-tour_v2Node js quick-tour_v2
Node js quick-tour_v2
 
Node js quick tour v2
Node js quick tour v2Node js quick tour v2
Node js quick tour v2
 
Scaling a Rails Application from the Bottom Up
Scaling a Rails Application from the Bottom Up Scaling a Rails Application from the Bottom Up
Scaling a Rails Application from the Bottom Up
 
Streaming huge databases using logical decoding
Streaming huge databases using logical decodingStreaming huge databases using logical decoding
Streaming huge databases using logical decoding
 
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
 
Rails Performance
Rails PerformanceRails Performance
Rails Performance
 

Recently uploaded

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Recently uploaded (20)

AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 

Pylons + Tokyo Cabinet

  • 1. Pylons + Tokyo Cabinet by Ben Cheng from Oursky.com
  • 2. About Us • Founder of Oursky Ltd. • Web application development services (Ruby on Rails, Django, Pylons, PHP, Wordpress, etc.); • FONCEPT.com - a T-Shirt design sharing portal; • Working on a SaaS workflow application.
  • 4. What is Pylons? • Flexible, Modular, Extensible • Built on established libraries • Close to 1.0 • Foundation of TurboGears 2.0
  • 5. Who is using Pylons? • Bittorrent • Reddit (code.reddit.com!) • and hundreds of sites
  • 6. WSGI • WSGI (Web Server Gateway Interface) • Wrap a “app” inside another “app” • Middleware, e.g.: • Authentication • Beaker (Cache, Session) • Routes
  • 7. The Stack • Models: SQLAlchemy, CouchDB... • Templates: Mako, Genshi, Jinja2, ... • Helpers: Web Helpers, FormAlchemy, ... • Request Dispatching: Routes, ... • Session / Caching: Beaker, ... • Deployment: Paste, mod_wsgi, Twisted, ...
  • 8. Take advantage of Python Power • TIOBE: Python is no. 8 • Such as: • Want interactive debugger? iPython • Logging? Chainsaw • Testing? Nose, MiniMock • ..... blah blah balh
  • 9. Power of other Python Web Framework • Tons of lib from Zope • ToscaWidgets from TurboGears • Authkit • (except Django)
  • 10. Community • Pylons Book • Quick response (e.g.): • We fired Ticket #620 at Pylons Trac • “init_model called twice in nosetests” • Bug get fixed in 2 weeks, committed to trunk and temp. solution are discussed.
  • 12. Introduction • By mixi.jp (Facebook in Japan) • TokyoCabinet (Engine) • TokyoTyrant (Server) • TokyoDystopia (Full-Text Search) • Opensources, standard C
  • 13. TokyoCabinet • Key/value DB • Insert: 0.4 sec / 1M records • Search: 0.33 sec / 1M records • hash / B+ Tree structure • Key Value / Hash / B+ Tree / Fixed Length / Table • Driver: Perl, Ruby, Python, PHP, Erlang...
  • 14. TokyoTyrant • Network server • Binary / Memcache / HTTP protocol • 17.2 sec / 1M queries • Hot Backup, Master-to-Master replication • Extensible by Lua
  • 15. Lua - Extensions function incr(key, i) i = tonumber(i) local old = tonumber(_get(key)) i = old + i _put(key, i) return i end
  • 16. Lua - Extension function expire() local args={} local cdata = string.format(“%d”, _time()) table.insert(args, “addcond0x0NUMLE”..cdata) table.insert(args, “out”) local res = _misc(“search”, args) if not res then _log(“expire failed”) end end
  • 17. Extensible to... • Inverted index • Set operations • (and we use it to do revision management)
  • 18. LightCloud • Distributed and persistent key-value database • Opensource and used by Plurk • Scale out by just adding nodes • Nodes are replicated Elapsed for 10000 gets: 1.74538516998 seconds [memcache] Elapsed for 10000 gets: 3.57339096069 seconds [lightcloud] Elapsed for 10000 sets: 1.88236999512 seconds [memcache] Elapsed for 10000 sets: 9.23674893379 seconds [lightcloud]
  • 19. Links • http://www.pylonshq.com • http://opensource.plurk.com/LightCloud/ • http://1978th.net/tokyocabinet/ • Reach me at bencheng@oursky.com