CouchDB – A Database for the Web

CouchDB — A Database for the Web
Karel Minařík

Karel Minařík
→ Independent web designer and developer

→ Ruby, Rails, Git and CouchDB propagandista in .cz

→ Previously: Flash Developer; Art Director; Information Architect;… (see LinkedIn)

→ @karmiq at Twitter

→ karmi.cz


$ couchdb
Apache CouchDB has started. Time to relax.


Apache CouchDB is a distributed, fault-tolerant
and schema-free document-oriented database
accessible via a RESTful HTTP/JSON API.
http://wiki.apache.org/couchdb


ibu ted
D is t r
Apache CouchDB is a distributed, a-Free
em
fault-tolerant
e nt Sch
and cum d
Do nte
schema-free document-oriented database
rie
accessible via a RESTful ful JSO API.
O
T HTTP/JSON
ES N
R
http://wiki.apache.org/couchdb


Talk Outline

➡ The NoSQL Moniker
➡ The CouchDB Story
➡ Schema-free Document Storage
➡ HTTP From Top to Bottom
➡ RESTful API
➡ Querying With Map/reduce
➡ Fault-tolerant, Distributed, Highly-available and Concurrent
➡ Demo: Example Application (Address Book)


NOSQL

Reasons for NoSQL

NoSQL is neither a “protest movement”
nor “trendy bullshit”.
Reasons for developing new
databases are real.
Most stem from some real pain.


NOSQL

Database denormalization at Digg

SELECT `digdate`, ìd` FROM `Diggs`
WHERE ùserid` IN (1, 2, 3, 4, ... 1000000)
AND itemid = 123 ORDER BY `digdate` DESC, ìd` DESC;

“A full query can actually clock in at 1.5kb, which is many times
larger than the actual data we want. With a cold cache, this query
can take 14 seconds to execute.”

“Non-relational data stores reverse this model completely, because they don’t have the
complex read operations of SQL. The model forces you to shift your computation to the
writes, while reducing most reads to simple operations – the equivalent of SELECT *
FROM `Table`.“
http://about.digg.com/blog/looking-future-cassandra


NOSQL

Redis: Big O Notation Built–In
redis> rpush mylist 1

redis> rpush mylist 2

redis> lpop mylist
"1"

...

redis> llen mylist

(integer) 1000000

redis> lpop mylist
"2"

$ redis-benchmark
...
====== LPOP ======
10025 requests completed in 0.53 seconds
...
93.43% <= 3 milliseconds


NOSQL

Use Case: Job Queue

RPUSH

}
LPOP

O(1) Millions of items

http://github.com/defunkt/resque/blob/master/lib/resque.rb#L133-138


1 The CouchDB Story


THE COUCHDB STORY

Damien Katz: CouchDB and Me

Damien Katz
(RubyFringe 2008)

http://www.infoq.com/presentations/katz-couchdb-and-me CouchDB — A Database for the Web

THE COUCHDB STORY

Damien Katz: CouchDB and Me

In the beginning, there was C++, XML and
custom query language.
Stuff nobody ever got fired for.
Then came Erlang, HTTP, JSON and map/reduce.


2 Schema–free Documents


SCHEMA-FREE STORAGE

“Relational Data”

OH: “The world is relational!!!”
17 minutes ago via Tweetie for Mac
Retweeted by 10000 people


That does not mean the world conforms
to the third normal form.


In fact, it’s rather the exact opposite.


SCHEMA–FREE DOCUMENTS

The Textbook Example

Design a customer database.
People have names, e-mail, phone numbers, …

How many phone numbers?




Relational Databases 101

Customers
id INTEGER A N P
ﬁrst_name VARCHAR
last_name VARCHAR
phone VARCHAR

Now. What about multiple phone numbers?

http://en.wikipedia.org/wiki/First_normal_form#Domains_and_values CouchDB — A Database for the Web



The “solution”, Pt. 1

Customers
id INTEGER A N P
ﬁrst_name VARCHAR
last_name VARCHAR
phone VARCHAR

“We will use the database only from the application, anyway.”




The “solution”, Pt. 2

Customers
id INTEGER A N P
ﬁrst_name VARCHAR
last_name VARCHAR
phone_1 VARCHAR
phone_2 VARCHAR
phone_3 VARCHAR

“This is clearly better design!”

Alright. Then, please answer these questions:
• How do you search for a customers given a phone number?
• Which customers have the same phone number?
• How many phone numbers a customer has?
Then, please add the ability to store four phone numbers. Thanks.




The Right Solution

Customers CustomerPhones
id INTEGER U A N P customer_id INTEGER i N F
ﬁrst_name VARCHAR phone VARCHAR N
last_name VARCHAR




mysql> SELECT * FROM Customers LEFT JOIN CustomerPhones
                               ON Customers.id = CustomerPhones.customer_id;

+‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐+
| id | first_name | last_name | customer_id | phone |
+‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐+
|  1 | John       | Smith     | 1           | 123   |
|  1 | John       | Smith     | 1           | 456   |
+‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐+




mysql> SELECT * FROM Customers WHERE id = 1;

+‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐+
| id | first_name | last_name |
+‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐+
|  1 | John       | Smith     |
+‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐+

mysql> SELECT phone FROM CustomerPhones WHERE customer_id IN (1);

+‐‐‐‐‐‐‐+
| phone |
+‐‐‐‐‐‐‐+
| 123   |
| 456   |
+‐‐‐‐‐‐‐+



Structured data

But, damn!, I want something like this:
{
  "id"         : 1,

  "first_name" : "Clara",
  "last_name"  : "Rice",

  "phones"     : ["0000 777 888 999", "0000 123 456 789", "0000 314 181 116"]
}

“No problem, you just iterate over the rows and build your object. That’s the way it is!”
“If this would be too painful, we will put some cache there.”



Ephemeral data

Not everything needs to be done “right”. Right?

class User < ActiveRecord::Base
serialize :preferences
end



“Consistency”

Does the “Right Way“ sometimes fail?
Hell yeah.
EXAMPLE
When designing an invoicing application, you store the
customer for the invoice the “right way”, via foreign keys.

Then, the customer address changes.

Did the address on the invoice also changed?



Documents in the Real World

12
3E
Ph VE
on R
Fa e:

gs
Sk
x: 55 YW
5. H

hin
yp 55 ER
e: 444 5 E
cit .44 .55
cityr
A

nt
00 y.r 4. 55 VEN
000 ea 4
St UE
lit 444

g eali
y,

si
Cit y.l
td

de123
Fac
t ory
Str
ee t
CI
TY
,S
ty, lt
T
00
00
d.
0

UNDERGROUND RECORDS
nd re co rd s.c om
in fo @u nd er gr ou

F: 555.555.5555
ITE 000 ANGELES, U.S.A/
P: 555.555.5555123 BOULEVARDAVE, SU LOS m
c o r d s .c o
M: 777.777.7777 roundre
www .u n d e r g
M: 888.000.1111

http://guide.couchdb.org/draft/why.html#better CouchDB — A Database for the Web


{

SON
J
"_id" : "clara-rice",
"_rev" : "1-def456",

"first_name" : "Clara",
"last_name" : "Rice",

"phones" : {
"mobile" : "0000 777 888 999"
"home" : "0000 123 456 789",
"work" : "0000 314 181 116"
},

"addresses" : {
"home" : {
"street" : "Wintheiser Ports",
"number" : "789/23",
"city" : "Erinshire",
"country" : "United Kingdom"
},
},

"occupation" : "model",
"birthday" : "1970/05/01",

"groups" : ["friends", "models"],

"created_at" : "2010/01/01 10:00:00 +0000"
}


Procrustean Bed

3 RESTful HTTP


HTTP

Built “Of the Web”

Django may be built for the Web,
but CouchDB is built of the Web.
I’ve never seen soware that so completely
embraces the philosophies behind HTTP.
Jacob Kaplan-Moss, Of the Web (2007)

http://jacobian.org/writing/of-the-web/ CouchDB — A Database for the Web

HTTP


CouchDB makes Django look old-school in the
same way that Django makes ASP look
outdated.


HTTP


HTTP is the lingua anca of our age; if you
speak HTTP, it opens up all sorts of doors.
ere’s something almost subversive about
CouchDB; it’s completely language-, platform-,
and OS-agnostic.


HTTP

HTTP API

HOST=http://localhost:5984

curl ‐X GET $HOST
# {"couchdb":"Welcome","version":"0.11.0b22c551bb‐git"}

curl ‐X GET $HOST/my‐database
# {"error":"not_found","reason":"no_db_file"}

curl ‐X PUT $HOST/my‐database
# {"ok":true}

curl ‐X PUT $HOST/my‐database/abc123 ‐d '{"foo":"bar"}'
# {"ok":true,"id":"abc123","rev":"1‐4c6114c65e295552ab1019e2b046b10e"}

curl ‐X GET $HOST/my‐database/abc123
# {"_id":"abc123","_rev":"1‐4c6114c65e295552ab1019e2b046b10e","foo":"bar"}

curl ‐X DELETE $HOST/my‐database/abc123?rev=2‐d179f665eb01834faf192153dc72dcb3
# {"ok":true,"id":"abc123","rev":"1‐4c6114c65e295552ab1019e2b046b10e"}


HTTP

Easy To Wrap
require 'rubygems'
require 'ostruct'

require 'restclient' 1 HTTP library
require 'json' 2 JSON library
class Article < OpenStruct
  def self.db(path='')
    RestClient::Resource.new "http://localhost:5984/blog/#{path}",
                             :headers => { :content_type => :json, :accept => :json }
  end

  db.put '' rescue RestClient::PreconditionFailed

  def self.create(params={})
    new db.post(params.to_json)
  end

  def self.find(id)
    new JSON.parse( db(id).get )
  end

  def destroy
    self.class.db(self._id + "?rev=#{self._rev}").delete
  end

end


HTTP

Easy To Wrap

Article.create :_id   => 'my‐first‐post',
               :title => 'CouchDB is easy',
               :body  => 'So relax!',
               :tags  => ['couchdb', 'databases'] rescue RestClient::Conflict

article =  Article.find('my‐first‐post')

puts "Got an article:"
p article

puts "n‐‐‐‐‐‐"
puts "Title: %s" % article.title        + "     (class: #{article.title.class})"
puts "Tags:  %s" % article.tags.inspect + "     (class: #{article.tags.class})"
puts "‐‐‐‐‐‐nn"

puts "Deleting article..."
article.destroy


HTTP

HTTP from Top to Bottom

$ curl ‐X POST http://localhost:5984/_replicate
‐d '{"source":"database",
"target":"http://example.org/database"}'


HTTP

Making Real Use of HTTP

$ curl ‐i ‐X GET $HOST/my‐database/abc123

HTTP/1.1 200 OK
Server: CouchDB/1.0.1 (Erlang OTP/R14B)
Etag: "4‐f04f2435e031054d6b5298c5841ae052"
Date: Thu, 23 Sep 2010 12:56:37 GMT
Content‐Type: text/plain;charset=utf‐8
Content‐Length: 73
Cache‐Control: must‐revalidate

{"_id":"abc123","_rev":"4‐f04f2435e031054d6b5298c5841ae052","foo":"bar"}


$ cat /etc/squid3/squid.conf

cache_peer 192.168.100.2 parent 5984 0 no‐query originserver name=master
acl master_acl method GET POST PUT DELETE
cache_peer_access master allow master_acl

HTTP

What is “RESTful”?

REST is a set of principles that define how Web standards, such as HTTP and
URIs, are supposed to be used. (...) In summary, the five key principles are:

➡ Give every “thing” an ID
➡ Link things together
➡ Use standard methods
➡ Resources with multiple representations
➡ Communicate statelessly

Stefan Tilkov, A Brief Introduction to REST

http://www.infoq.com/articles/rest-introduction CouchDB — A Database for the Web

HTTP

What is “RESTful”?

The basic idea is even more simple, though.
HTTP is not just a “transfer protocol”.
It is the interface for interacting with “things” itself.


4 Fault-Tolerant and Concurrent


CouchDB has no off switch.
CouchDB has no repair command.

$ kill ‐9 <PID>

FAULT–TOLERANT

Erlang

Erlang!

http://www.youtube.com/watch?v=uKfKtXYLG78 CouchDB — A Database for the Web

FAULT–TOLERANT

Erlang

Erlang's main strength is support for concurrency. It has a small but powerful
set of primitives to create processes and communicate among them.
(…) a benchmark with 20 million processes has been successfully performed.

http://en.wikipedia.org/wiki/Erlang_(programming_language) CouchDB — A Database for the Web

FAULT–TOLERANT

Append–Only B–Tree

http://guide.couchdb.org/draft/btree.html CouchDB — A Database for the Web

5 Querying With Map/reduce


MAP/REDUCE

The Google Paper

http://labs.google.com/papers/mapreduce.html CouchDB — A Database for the Web

MAP/REDUCE

The Concept

module Enumerable
alias :reduce :inject unless method_defined? :reduce
end

(1..3).map { |number| number * 2 }
# => [2, 4, 6]

(1..3).reduce(0) { |sum, number| sum += number}
# => 6


MAP/REDUCE

The Simplest View

function(doc) {
  if (doc.last_name && doc.first_name) {
    emit( doc.last_name + ' ' + doc.first_name, doc )
  }
}


MAP/REDUCE

The Simplest View

INPUT

function(doc) {
  if (doc.last_name && doc.first_name) {
    emit( doc.last_name + ' ' + doc.first_name, doc )
  }
}

OUTPUT KEY VALUE


MAP/REDUCE

The Result of Map

Key Value

_id: "lottie‐armstrong",
_rev: "2‐fcb71b26096957b3ff3ffd2970f3c933",
addresses: {
home: {
city: "Murphyville"
...
"Armstrong Lottie"
}
},
first_name: "Lottie",
last_name: "Armstrong",
occupation: "programmer",

_id: "kaelyn‐bailey",
_rev: "1‐2e25e6c9448520fa796988894423a23b",
addresses: {
      home: {
        city: "Lake Dedric"
"Bailey Kaelyn"         ...
      }
},
first_name: "Kaelyn",
last_name: "Bailey",
occupation: "supermodel"

... ...


MAP/REDUCE

The Result of Map


MAP/REDUCE

Even Simpler View

function(doc) {
emit(doc.occupation, 1);
}


MAP/REDUCE

Result of Even Simpler View

http://localhost:5984/_utils/database.html?addressbook/_design/person/_view/by_occupation CouchDB — A Database for the Web

MAP/REDUCE

A Simple Reduce

function(keys, values) {
return sum(values)
}


MAP/REDUCE

Result of a Simple Reduce

http://localhost:5984/_utils/database.html?addressbook/_design/person/_view/by_occupation CouchDB — A Database for the Web

MAP/REDUCE

Built–In Erlang Reduce functions

$ couchdb
Apache CouchDB has started. Time to relax.

_count
_sum
_stats

http://wiki.apache.org/couchdb/Built-In_Reduce_Functions#Available_Build-In_Functions CouchDB — A Database for the Web

MAP/REDUCE

Map/Reduce for Counting “tag-like stuff”

function(doc) {
  for (group in doc.groups) {
    emit(doc.groups[group], 1)
  }
}

_count


MAP/REDUCE

Result of the Map phase

http://localhost:5984/_utils/database.html?addressbook/_design/person/_view/by_groups CouchDB — A Database for the Web

MAP/REDUCE

Result of the Reduce Phase

http://localhost:5984/_utils/database.html?addressbook/_design/person/_view/by_groups CouchDB — A Database for the Web

MAP/REDUCE

Group Levels

function(doc) {
var date = new Date(doc.birthday)
emit( [date.getFullYear(), date.getMonth()+1, date.getDate()], 1 )
} COMPOSITE KEY (ARRAY)

_count


MAP/REDUCE

Group Level Exact

http://localhost:5984/_utils/database.html?addressbook/_design/person/_view/by_birthday CouchDB — A Database for the Web

MAP/REDUCE

Group Level 2


MAP/REDUCE

Group Level 1


QUERYING VIEWS

Parameters for querying views

key
startkey
startkey_docid
endkey
endkey_docid
limit
stale
descending
skip
group
group_level
reduce
include_docs

QUERYING VIEWS

A Complex Map/Reduce


QUERYING VIEWS


SELECT
      COUNT(*) AS count,
        DATE_FORMAT(published_at, "%Y/%m/%d") AS date,
        keywords.value AS keyword
      FROM feed_entries
        INNER JOIN feeds ON feed_entries.feed_id = feeds.id
        INNER JOIN keywords ON feeds.keyword_id = keywords.id
      WHERE DATE_SUB(CURDATE(), INTERVAL 90 DAY) <= feed_entries.published_at
      GROUP BY date, keyword
      ORDER BY date, keyword ASC;


QUERYING VIEWS


But. We don’t need a table. We need the data in a format like this:

Streamgraph.load_data({
  max : 170,

  keywords : ['ruby', 'python', 'erlang', 'javascript', 'haskell'],

  values   : [
    { date: '2010/01/01', ruby: 50,  python: 20, erlang: 5,  javascript: 30, haskell: 50 },
    { date: '2010/06/01', ruby: 30, python: 10, erlang: 14, javascript: 170, haskell: 14 }
  ]

});


QUERYING VIEWS

The Map Phase

function(doc) {
  var fix_date = function(junk) {
      var formatted = junk.toString().replace(/‐/g,"/").replace("T"," ").substring(0,19);
      return new Date(formatted);
    };
  // Format integers to have at least two digits.
  var f = function(n) { return n < 10 ? '0' + n : n; }
  // This is a format that collates in order and tends to work with
  // JavaScript's new Date(string) date parsing capabilities, unlike rfc3339.
  Date.prototype.toJSON = function() {
      return this.getUTCFullYear()   + '/' +
           f(this.getUTCMonth() + 1) + '/' +
           f(this.getUTCDate())      + ' ' +
           f(this.getUTCHours())     + ':' +
           f(this.getUTCMinutes())   + ':' +
           f(this.getUTCSeconds())   + ' +0000';
  };
  if (doc['couchrest‐type'] == 'Mention') {
    for ( keyword in doc.keywords ) {
      var key = fix_date(doc.published_at).toJSON().substring(0,10);
      var value = {};
      value[ doc.keywords[keyword] ] = 1;
      emit( key, value);
    }
  }
}


QUERYING VIEWS

The Reduce Phase

function(keys, values, rereduce) {
  if (rereduce) {
    var result = {}
    for ( item in values ) {
      for (prop in values[item]) {
        if ( result[prop] ) { result[prop] += values[item][prop]  }
        else { result[prop] = values[item][prop]  }
      }
    }
    return result;
  }
  else {
    // Prepare the data for the re‐reduce
    var date   = keys[0][0];
    var result = {}
    for (value in values) {
      var item = values[value];
      for (prop in item) {
        if ( result[prop] ) { result[prop] += item[prop] }
        else { result[prop] = item[prop] }
      }
    }
    return result;
  }
}


QUERYING VIEWS

The Result

$ curl http://localhost:5984/customer_database/_design/Mention/_view/by_date_and_keyword?group=true

{
    "rows": [
        {
            "key": "2010/09/22",
            "value": { "ruby": 8, "python": 19 }
        },
        {
            "key": "2010/09/23",
        },
        {
            "key": "2010/09/24",
        }
    ]
}


QUERYING VIEWS

I JS. Or... don’t?


QUERYING VIEWS

Complex Queries

So… What if you need something like:

Show me all supermodels who live in Beckerborough.

Out of luck?


COMPLEX QUERIES

CouchDB–Lucene

This guy knows.

Show me all supermodels who live in Beckerborough.

COMPLEX QUERIES

foo AND bar

Couchdb-Lucene.
When you need foo AND bar.

http://github.com/rnewson/couchdb-lucene CouchDB — A Database for the Web

COUCHDB-LUCENE

Indexing function

function(doc) {

  var result = new Document();

  if (doc.occupation)     {
    result.add(doc.occupation,                    {"field":"occupation"})
  }

  if (doc.addresses)       {
    for (address in doc.addresses) {
      result.add(doc.addresses[address].city,     {"field":"city"})
    }
  }

  return result;

}

http://localhost:5984/addressbook/_fti/_design/person/search?q=occupation:supermodel AND city:Beckerborough


6 Distributed


DISTRIBUTED

Ubuntu One


DISTRIBUTED

Replication


DISTRIBUTED

Conflict Resolutions

_rev1

http://guide.couchdb.org/draft/consistency.html#study CouchDB — A Database for the Web

DISTRIBUTED

Simple Clustering With HTTP Reverse Proxies

http://ephemera.karmi.cz/post/247255194/simple-couchdb-multi-master-clustering-via-nginx CouchDB — A Database for the Web

DISTRIBUTED

Scaling Down

http://www.couchone.com/page/android

DISTRIBUTED

CouchApps


DISTRIBUTED

CouchApps

http://pollen.nymphormation.org/afgwar/_design/afgwardiary/index.html CouchDB — A Database for the Web

7 Resources


DISTRIBUTED

Resources

➡ http://guide.couchdb.org

➡ https://nosqleast.com/2009/#speaker/miller

➡ http://www.couchone.com/migrating-to-couchdb

➡ http://wiki.apache.org/couchdb/

➡ http://blog.couchone.com/

➡ http://stackoverflow.com/tags/couchdb/


8 Demo: Example Application


DEMO

Application

SOURCE CODE: http://github.com/karmi/couchdb-showcase

http://karmi.couchone.com/addressbook/_design/person/_list/all/all CouchDB — A Database for the Web

CouchDB – A Database for the Web

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a CouchDB – A Database for the Web

Semelhante a CouchDB – A Database for the Web (20)

Mais de Karel Minarik

Mais de Karel Minarik (20)

Último

Último (20)

CouchDB – A Database for the Web