SlideShare uma empresa Scribd logo
1 de 26
Baixar para ler offline
What Every Data Programmer ,[object Object],Needs to Know about Disks,[object Object],OSCON Data – July, 2011 - Portland,[object Object],Ted Dziuba,[object Object],@dozba,[object Object],tjdziuba@gmail.com,[object Object],Not proprietary or confidential. In fact, you’re risking a career by listening to me.,[object Object]
Who are you and why are you talking?,[object Object],First job: Like college but they pay you to go.,[object Object],A few years ago: Technical troll for The Register.,[object Object],Recently: Co-founder of Milo.com, local shopping engine.,[object Object],Present: Senior Technical Staff for eBay Local,[object Object]
The Linux Disk Abstraction,[object Object],Volume,[object Object],/mnt/volume,[object Object],File System,[object Object],xfs, ext,[object Object],Block Device,[object Object],HDD, HW RAID array,[object Object]
What happens when you read from a file?,[object Object],f = open(“/home/ted/not_pirated_movie.avi”, “rb”),[object Object],avi_header = f.read(56),[object Object],f.close(),[object Object],Disk,[object Object],controller,[object Object],user,[object Object],buffer,[object Object],page,[object Object],cache,[object Object],platter,[object Object]
What happens when you read from a file?,[object Object],Disk,[object Object],controller,[object Object],user,[object Object],buffer,[object Object],page,[object Object],cache,[object Object],[object Object]
Latency: 100 nanoseconds
Throughput: 12GB/sec on good hardwareplatter,[object Object]
What happens when you read from a file?,[object Object],Disk,[object Object],controller,[object Object],user,[object Object],buffer,[object Object],page,[object Object],cache,[object Object],[object Object]
Latency: 10 milliseconds
Throughput: 768 MB/sec on SATA 3
(Faster if you have a lot of money)platter,[object Object]
Sidebar: The Horror of a 10ms Seek Latency,[object Object],A disk read is 100,000 times slower than a memory read.,[object Object],100 nanoseconds,[object Object],Time it takes you to write a really clever tweet,[object Object],10 milliseconds,[object Object],Time it takes to write a novel, working full time,[object Object]
What happens when you write to a file?,[object Object],f = open(“/home/ted/nosql_database.csv”, “wb”),[object Object],f.write(key),[object Object],f.write(“,”),[object Object],f.write(value),[object Object],f.close(),[object Object],Disk,[object Object],controller,[object Object],user,[object Object],buffer,[object Object],page,[object Object],cache,[object Object],platter,[object Object]
What happens when you write to a file?,[object Object],f = open(“/home/ted/nosql_database.csv”, “wb”),[object Object],f.write(key),[object Object],f.write(“,”),[object Object],f.write(value),[object Object],f.close(),[object Object],Disk,[object Object],controller,[object Object],user,[object Object],buffer,[object Object],page,[object Object],cache,[object Object],platter,[object Object],You need to make this,[object Object],part happen,[object Object],Mark the page dirty,,[object Object],call it a day and go have a smoke.,[object Object]
Aside: Stick your finger in the Linux Page Cache,[object Object],Pre-Linux 2.6 used “pdflush”, now per-Backing Device Info (BDI) flush threads,[object Object],Dirty pages: grep –i “dirty” /proc/meminfo,[object Object],/proc/sys/vmLove:,[object Object],[object Object]
dirty_ratio: flush after some percent of memory is used
dirty_writeback_centisecs: how often to wake up and start flushingClear your page cache: echo 1 > /proc/sys/vm/drop_caches,[object Object],Crusty sysadmin’s hail-Mary pass: sync; sync; sync,[object Object]
Fsync: force a flush to disk,[object Object],f = open(“/home/ted/nosql_database.csv”, “wb”),[object Object],f.write(key),[object Object],f.write(“,”),[object Object],f.write(value),[object Object],os.fsync(f.fileno()),[object Object],f.close(),[object Object],Disk,[object Object],controller,[object Object],user,[object Object],buffer,[object Object],page,[object Object],cache,[object Object],platter,[object Object],Also note, fsync() has a cousin, fdatasync() that does not sync metadata.,[object Object]
Aside: point and laugh at MongoDB,[object Object],Mongo’s “fsync” command:,[object Object],> db.runCommand({fsync:1,async:true}); ,[object Object],wat.,[object Object],Also supports “journaling”, like a WAL in the SQL world, however…,[object Object],[object Object]
It’s not enabled by default.,[object Object]
Fsync: bitter lies,[object Object],platter,[object Object],…it’s a cache!,[object Object],Disk,[object Object],controller,[object Object],page,[object Object],cache,[object Object],[object Object]
Writeback is the demon,[object Object]
(Just dropped in) to see what condition your caches are in,[object Object],A Good Server,[object Object],platter,[object Object],Writethrough cache,[object Object],on controller,[object Object],Writethrough cache on disk,[object Object],Disk,[object Object],controller,[object Object]
(Just dropped in) to see what condition your caches are in,[object Object],An Even Better Server,[object Object],platter,[object Object],Battery-backedwriteback,[object Object],cache on controller,[object Object],Writethrough cache on disk,[object Object],Disk,[object Object],controller,[object Object]
(Just dropped in) to see what condition your caches are in,[object Object],The Demon Setup,[object Object],platter,[object Object],Battery-backed writeback,[object Object],cache or,[object Object],Writethrough cache,[object Object],Writeback cache on disk,[object Object],Disk,[object Object],controller,[object Object]
Disks in a virtual environment,[object Object],The Trail of Tears to the Platter,[object Object],Host,[object Object],page,[object Object],cache,[object Object],Virtual,[object Object],controller,[object Object],user,[object Object],buffer,[object Object],page,[object Object],cache,[object Object],Physical,[object Object],controller,[object Object],Hypervisor,[object Object],platter,[object Object]

Mais conteúdo relacionado

Mais procurados

Postgres connections at scale
Postgres connections at scalePostgres connections at scale
Postgres connections at scaleMydbops
 
基本に戻ってInnoDBの話をします
基本に戻ってInnoDBの話をします基本に戻ってInnoDBの話をします
基本に戻ってInnoDBの話をしますyoku0825
 
トランザクションをSerializableにする4つの方法
トランザクションをSerializableにする4つの方法トランザクションをSerializableにする4つの方法
トランザクションをSerializableにする4つの方法Kumazaki Hiroki
 
TiDBのトランザクション
TiDBのトランザクションTiDBのトランザクション
TiDBのトランザクションAkio Mitobe
 
Problems with PostgreSQL on Multi-core Systems with MultiTerabyte Data
Problems with PostgreSQL on Multi-core Systems with MultiTerabyte DataProblems with PostgreSQL on Multi-core Systems with MultiTerabyte Data
Problems with PostgreSQL on Multi-core Systems with MultiTerabyte DataJignesh Shah
 
Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Mydbops
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)Altinity Ltd
 
Java Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsJava Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsBrendan Gregg
 
RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compactionMIJIN AN
 
KafkaとAWS Kinesisの比較
KafkaとAWS Kinesisの比較KafkaとAWS Kinesisの比較
KafkaとAWS Kinesisの比較Yoshiyasu SAEKI
 
MariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB plc
 
ioMemoryとAtomic Writeによるデータベース高速化
ioMemoryとAtomic Writeによるデータベース高速化ioMemoryとAtomic Writeによるデータベース高速化
ioMemoryとAtomic Writeによるデータベース高速化IIJ
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBlazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBrendan Gregg
 
まずやっとくPostgreSQLチューニング
まずやっとくPostgreSQLチューニングまずやっとくPostgreSQLチューニング
まずやっとくPostgreSQLチューニングKosuke Kida
 
NAND Flash から InnoDB にかけての話(仮)
NAND Flash から InnoDB にかけての話(仮)NAND Flash から InnoDB にかけての話(仮)
NAND Flash から InnoDB にかけての話(仮)Takanori Sejima
 
爆速クエリエンジン”Presto”を使いたくなる話
爆速クエリエンジン”Presto”を使いたくなる話爆速クエリエンジン”Presto”を使いたくなる話
爆速クエリエンジン”Presto”を使いたくなる話Kentaro Yoshida
 
Innodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライドInnodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライドYasufumi Kinoshita
 

Mais procurados (20)

Postgres connections at scale
Postgres connections at scalePostgres connections at scale
Postgres connections at scale
 
基本に戻ってInnoDBの話をします
基本に戻ってInnoDBの話をします基本に戻ってInnoDBの話をします
基本に戻ってInnoDBの話をします
 
WiredTigerを詳しく説明
WiredTigerを詳しく説明WiredTigerを詳しく説明
WiredTigerを詳しく説明
 
トランザクションをSerializableにする4つの方法
トランザクションをSerializableにする4つの方法トランザクションをSerializableにする4つの方法
トランザクションをSerializableにする4つの方法
 
TiDBのトランザクション
TiDBのトランザクションTiDBのトランザクション
TiDBのトランザクション
 
Problems with PostgreSQL on Multi-core Systems with MultiTerabyte Data
Problems with PostgreSQL on Multi-core Systems with MultiTerabyte DataProblems with PostgreSQL on Multi-core Systems with MultiTerabyte Data
Problems with PostgreSQL on Multi-core Systems with MultiTerabyte Data
 
Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
 
Java Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsJava Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame Graphs
 
RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compaction
 
KafkaとAWS Kinesisの比較
KafkaとAWS Kinesisの比較KafkaとAWS Kinesisの比較
KafkaとAWS Kinesisの比較
 
MariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & Optimization
 
ioMemoryとAtomic Writeによるデータベース高速化
ioMemoryとAtomic Writeによるデータベース高速化ioMemoryとAtomic Writeによるデータベース高速化
ioMemoryとAtomic Writeによるデータベース高速化
 
Native Memory Tracking
Native Memory TrackingNative Memory Tracking
Native Memory Tracking
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBlazing Performance with Flame Graphs
Blazing Performance with Flame Graphs
 
Vacuum徹底解説
Vacuum徹底解説Vacuum徹底解説
Vacuum徹底解説
 
まずやっとくPostgreSQLチューニング
まずやっとくPostgreSQLチューニングまずやっとくPostgreSQLチューニング
まずやっとくPostgreSQLチューニング
 
NAND Flash から InnoDB にかけての話(仮)
NAND Flash から InnoDB にかけての話(仮)NAND Flash から InnoDB にかけての話(仮)
NAND Flash から InnoDB にかけての話(仮)
 
爆速クエリエンジン”Presto”を使いたくなる話
爆速クエリエンジン”Presto”を使いたくなる話爆速クエリエンジン”Presto”を使いたくなる話
爆速クエリエンジン”Presto”を使いたくなる話
 
Innodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライドInnodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライド
 

Semelhante a What every data programmer needs to know about disks

Memory management in Linux
Memory management in LinuxMemory management in Linux
Memory management in LinuxRaghu Udiyar
 
Let Me Pick Your Brain - Remote Forensics in Hardened Environments
Let Me Pick Your Brain - Remote Forensics in Hardened EnvironmentsLet Me Pick Your Brain - Remote Forensics in Hardened Environments
Let Me Pick Your Brain - Remote Forensics in Hardened EnvironmentsNicolas Collery
 
Logical volume manager xfs
Logical volume manager xfsLogical volume manager xfs
Logical volume manager xfsSarwar Javaid
 
Data hiding and finding on Linux
Data hiding and finding on LinuxData hiding and finding on Linux
Data hiding and finding on LinuxAnton Chuvakin
 
Tuning Linux for Databases.
Tuning Linux for Databases.Tuning Linux for Databases.
Tuning Linux for Databases.Alexey Lesovsky
 
Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Tanya Denisyuk
 
Live memory forensics
Live memory forensicsLive memory forensics
Live memory forensicsMehedi Hasan
 
Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)MongoDB
 
MySQL Oslayer performace optimization
MySQL  Oslayer performace optimizationMySQL  Oslayer performace optimization
MySQL Oslayer performace optimizationLouis liu
 
Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)
Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)
Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)Hari Shankar Sreekumar
 
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...Ron Munitz
 
Recipe of a linux Live CD (archived)
Recipe of a linux Live CD (archived)Recipe of a linux Live CD (archived)
Recipe of a linux Live CD (archived)Bud Siddhisena
 
Unix fundamentals
Unix fundamentalsUnix fundamentals
Unix fundamentalsBimal Jain
 

Semelhante a What every data programmer needs to know about disks (20)

Linux Recovery
Linux RecoveryLinux Recovery
Linux Recovery
 
Memory management in Linux
Memory management in LinuxMemory management in Linux
Memory management in Linux
 
Let Me Pick Your Brain - Remote Forensics in Hardened Environments
Let Me Pick Your Brain - Remote Forensics in Hardened EnvironmentsLet Me Pick Your Brain - Remote Forensics in Hardened Environments
Let Me Pick Your Brain - Remote Forensics in Hardened Environments
 
Vmfs
VmfsVmfs
Vmfs
 
Logical volume manager xfs
Logical volume manager xfsLogical volume manager xfs
Logical volume manager xfs
 
Hardware
HardwareHardware
Hardware
 
Data hiding and finding on Linux
Data hiding and finding on LinuxData hiding and finding on Linux
Data hiding and finding on Linux
 
Tuning Linux for Databases.
Tuning Linux for Databases.Tuning Linux for Databases.
Tuning Linux for Databases.
 
Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "
 
Live memory forensics
Live memory forensicsLive memory forensics
Live memory forensics
 
Clear cache memory
Clear cache memoryClear cache memory
Clear cache memory
 
Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)
 
MySQL Oslayer performace optimization
MySQL  Oslayer performace optimizationMySQL  Oslayer performace optimization
MySQL Oslayer performace optimization
 
Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)
Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)
Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)
 
FreeBSD Portscamp, Kuala Lumpur 2016
FreeBSD Portscamp, Kuala Lumpur 2016FreeBSD Portscamp, Kuala Lumpur 2016
FreeBSD Portscamp, Kuala Lumpur 2016
 
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
 
Recipe of a linux Live CD (archived)
Recipe of a linux Live CD (archived)Recipe of a linux Live CD (archived)
Recipe of a linux Live CD (archived)
 
Hadoop Architecture
Hadoop ArchitectureHadoop Architecture
Hadoop Architecture
 
Linux filesystemhierarchy
Linux filesystemhierarchyLinux filesystemhierarchy
Linux filesystemhierarchy
 
Unix fundamentals
Unix fundamentalsUnix fundamentals
Unix fundamentals
 

Mais de iammutex

Scaling Instagram
Scaling InstagramScaling Instagram
Scaling Instagramiammutex
 
Redis深入浅出
Redis深入浅出Redis深入浅出
Redis深入浅出iammutex
 
深入了解Redis
深入了解Redis深入了解Redis
深入了解Redisiammutex
 
NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析iammutex
 
MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用iammutex
 
8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slide8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slideiammutex
 
Thoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency ModelsThoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency Modelsiammutex
 
Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告iammutex
 
redis 适用场景与实现
redis 适用场景与实现redis 适用场景与实现
redis 适用场景与实现iammutex
 
Introduction to couchdb
Introduction to couchdbIntroduction to couchdb
Introduction to couchdbiammutex
 
redis运维之道
redis运维之道redis运维之道
redis运维之道iammutex
 
Realtime hadoopsigmod2011
Realtime hadoopsigmod2011Realtime hadoopsigmod2011
Realtime hadoopsigmod2011iammutex
 
[译]No sql生态系统
[译]No sql生态系统[译]No sql生态系统
[译]No sql生态系统iammutex
 
Couchdb + Membase = Couchbase
Couchdb + Membase = CouchbaseCouchdb + Membase = Couchbase
Couchdb + Membase = Couchbaseiammutex
 
Redis cluster
Redis clusterRedis cluster
Redis clusteriammutex
 
Redis cluster
Redis clusterRedis cluster
Redis clusteriammutex
 
Hadoop introduction berlin buzzwords 2011
Hadoop introduction   berlin buzzwords 2011Hadoop introduction   berlin buzzwords 2011
Hadoop introduction berlin buzzwords 2011iammutex
 

Mais de iammutex (20)

Scaling Instagram
Scaling InstagramScaling Instagram
Scaling Instagram
 
Redis深入浅出
Redis深入浅出Redis深入浅出
Redis深入浅出
 
深入了解Redis
深入了解Redis深入了解Redis
深入了解Redis
 
NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析
 
MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用
 
8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slide8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slide
 
skip list
skip listskip list
skip list
 
Thoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency ModelsThoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency Models
 
Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告
 
redis 适用场景与实现
redis 适用场景与实现redis 适用场景与实现
redis 适用场景与实现
 
Introduction to couchdb
Introduction to couchdbIntroduction to couchdb
Introduction to couchdb
 
Ooredis
OoredisOoredis
Ooredis
 
Ooredis
OoredisOoredis
Ooredis
 
redis运维之道
redis运维之道redis运维之道
redis运维之道
 
Realtime hadoopsigmod2011
Realtime hadoopsigmod2011Realtime hadoopsigmod2011
Realtime hadoopsigmod2011
 
[译]No sql生态系统
[译]No sql生态系统[译]No sql生态系统
[译]No sql生态系统
 
Couchdb + Membase = Couchbase
Couchdb + Membase = CouchbaseCouchdb + Membase = Couchbase
Couchdb + Membase = Couchbase
 
Redis cluster
Redis clusterRedis cluster
Redis cluster
 
Redis cluster
Redis clusterRedis cluster
Redis cluster
 
Hadoop introduction berlin buzzwords 2011
Hadoop introduction   berlin buzzwords 2011Hadoop introduction   berlin buzzwords 2011
Hadoop introduction berlin buzzwords 2011
 

Último

Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 

Último (20)

Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 

What every data programmer needs to know about disks

Notas do Editor

  1. Note that the page is actually in memory twice. Mmaped files fix this, but it’s beyond the scope of this discussion.Also this is why read performance on a lot of memory only NoSQL databases beats disk-backed SQL. Duh.
  2. Equate 100 nanoseconds to about 100 seconds. Then 10 milliseconds is about 3 months.
  3. This is where a lot of NoSQL databases get their performance, but more on that in a few minutes.
  4. There are threads that wake up every now and then to flush pages to disk.
  5. Fsync blocks until the data has been written to disk.
  6. With a battery-backed RAID controller, fsync can return very quickly with little risk of data loss.
  7. You need to dive into your vendor’s control tool to find this out.
  8. VMWare server is faithful to fsync, VMWare workstation is not. Xen usually queues I/O requests after they have been issued. The point is that you have no way of knowing. Your visibility of what happens to your data after you write or fsync ends at the hypervisor.
  9. Newer intel chips have the northbridge controller on-die. Southbridge bandwidth is usually <= 10GB/sec, and you are sharing this with other customers’ network and disk I/O. That, and you may be sharing drive spindles.
  10. EBS lies about the result of fsync. This is why Reddit is down all the time. You have been warned.
  11. EBS lies about the result of fsync. This is why Reddit is down all the time. You have been warned.