My talk from the Megacomm 2012 conference in Jerusalem, on February 16th, 2012. I describe the fundamental underpinnings of the Web, how things have changed on both the browser and server sides, and what these technologies mean for users..
Modern Web technologies (and why you should care): Megacomm, Jerusalem, February 2012
1. Modern Web
Technologies (and why
you should care)
Reuven M. Lerner • reuven@lerner.co.il
Megacomm
February 15th, 2012
2. Who am I?
• Web developer since 1993
• Software architect/consultant/trainer
• Linux Journal columnist since 1996
• Mostly Ruby on Rails + PostgreSQL, but
also Python, jQuery, MySQL, and more...
• PhD candidate at Northwestern University
3. How does the Internet
(TCP/IP) work?
“Socket”
Client Server
Port Port
4. How does the Internet
(TCP/IP) work?
“Socket”
Client Server
Port Port
Client opens
connection
5. How does the Internet
(TCP/IP) work?
“Socket”
Client Server
Port Port
Client opens Server accepts
connection connection
6. Protocols
• Communication standards built on top of
TCP/IP, typically text-based
• SMTP (e-mail)
• FTP (file transfer)
• NNTP (transfer of “news” articles)
20. Submitting forms
POST /login
name=reuven&password=secret
Browser Server
21. Submitting forms
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
22. Document = request
• If an HTML page contains images, then each
is retrieved in a separate HTTP request
• Page with 30 images = 31 HTTP requests
• Page with 20 images, 10 JavaScript files,
and 5 CSS files = 36 HTTP requests
23.
24. Idea: Lie to the browser
• Don’t return a document to the user
• Rather, run a program when the user makes
a request, and send the program’s output
• If the output is in HTML, then the browser
will show it no differently than a static doc
25. Just in time production
• Wait as long as
possible to create
pages for the user
• Ideally, create them
when the user
requests them
• “Mass customization”
28. Dynamic document
GET /
Browser Server
200 OK
<html>
<head>...</head>
<body>...</body>
</html>
29. Dynamic document
GET /
Browser Server
200 OK
<html>
<head>...</head>
Program output
<body>...</body>
</html>
30.
31.
32. What we return
• Usually HTML
• Image (e.g., stock graphs, Google Analytics)
• Word/Excel doc (e.g., from Google docs)
• PDF (e.g., PDF reports)
• XML, JSON (for computers, not people)
• Basically, any defined MIME type
33. APIs and mobile apps
Computer A Computer B
(browser) (server)
34. APIs and mobile apps
GET /
Computer A Computer B
(browser) (server)
35. APIs and mobile apps
GET /
Computer A Computer B
(browser) (server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
36. APIs and mobile apps
GET /
Computer B
(server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
37. APIs and mobile apps
GET /
Computer B
(server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
38. APIs and mobile apps
GET /
Computer B
(server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
39. APIs and mobile apps
GET /
Computer B
(server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
40. What is a Web app?
• Receives its inputs via HTTP
• Sends its output via HTTP
• That’s it! A Web application can do
anything you want, within these limits
41. That’s it!
• Now you understand how the Web works.
• Really, that’s it.
• Go home.
42. OK, perhaps not...
• How do we write these programs?
• Where (and how) do we store user data?
• How have the underlying technologies
(URLs, HTTP, and HTML) advanced?
43. Early Web applications
• First server-side programs were in C
• They were compiled into binaries
• So you had the CGI source (cgi-src)
directory...
• ... and the CGI binary (cgi-bin) directory
44. “Scripting” languages
• No explicit compilation
• Cross platform
• Built-in, powerful text functions
• Do a lot in a little bit of code
• Perl, Python, PHP, Ruby
• Typically open source
45. Frameworks
• DRY (Don’t repeat yourself)
• Get rid of the drudgery
• Concentrate on your business, rather than
worrying about common Web issues
46. MVC paradigm
• Most modern frameworks use MVC
• From Smalltalk in the 1980s!
• Model — communicates with database
• View — HTML/JavaScript/CSS for user
• Controller — handles requests
• Clear division of labor
47. Web frameworks in
dynamic languages
• Programmer speed trumps execution
speed
• Community support
• Plugins for commonly requested features
• Create a domain-specific language (DSL)
for your specific needs
48. Ruby on Rails
• Ruby language
• Rails Web app framework (MVC)
• Designed for writing Web DSLs
• “Convention over configuration”
• Thousands of little improvements
• ActiveRecord — ORM
50. Where’s the definition?
• The computer takes care of it automatically
• ActiveRecord knows what columns you
have defined, and what their types are
• (More on columns later)
• Only write things that cannot be
understood automatically
51. Not only Rails
• Python (Django)
• PHP (Symfony)
• Perl (Catalyst)
• Groovy (Grails)
• Even if you don’t use Ruby on Rails, you
have benefitted from its “opinions”
52. Plugins
• Rails (and other systems) have open-source
plugins to handle common issues
• Authentication
• E-commerce
• Social networking
• Don’t write these yourself! Customize
existing code that has proved itself
53. Storage
• Applications are great!
• But what if we want to store information
about our users?
• Name, e-mail address, account balance
• We could use text files, but most people
will use a database
54. What is a database?
Store data
confidently
Database
Retrieve data
flexibly
55. Relational databases
Define tables,
store data in them
Database
Retrieve data from
related tables
57. Relational databases
• Everything is stored in 2-dimensional tables
• Data should appear only once (normalized)
• “Join” tables to connect tables
• Technology is extremely robust, fail-safe
• Not all data is a good fit for this paradigm
58. id first_name last_name phone
1 Reuven Lerner 054-496-8405
2 Charlie Kalech 02-671-9918
60. id first_name last_name email
1 Reuven Lerner reuven@lerner.co.il
2 Charlie Kalech charlie@j-town.com
person_id phone_number_type_id phone_number id type
1 2 054-496-8405
1 1 847-230-9795 1 work
2 1 02-671-9918
2 mobile
2 2 054-803-3356
2 3 501-629-8620 3 fax
4 home
61. id first_name last_name email
1 Reuven Lerner reuven@lerner.co.il
2 Charlie Kalech charlie@j-town.com
person_id phone_number_type_id phone_number id type
1 2 054-496-8405
1 1 847-230-9795 1 work
2 1 02-671-9918
2 mobile
2 2 054-803-3356
2 3 501-629-8620 3 fax
4 home
SELECT P.first_name, P.last_name, P.email,
PN.phone_number, PNT.type
FROM People P, Phone_Numbers PN,
Phone_Number_Types PNT
WHERE PN.person_id = P.id
AND PNT.id = PN.phone_number_type_id
62. Another language!
• SQL is the language of relational databases
• So a Web app will use a language (e.g.,
Ruby, Python, or PHP) + SQL
• Or use an ORM, which automatically
translates your language into SQL
Person.first.phone_number
73. Scaling problems
• Lots of requests?
• Optimize
• Even more requests?
• Buy a bigger server
• What next?
• Panic! (Or spend lots of money)
74. Sharding
• Split your data across multiple databases
• This works, but...
• Requires rewriting a lot of code
• Maintenance is a big issue
• Re-sharding as each server gets
overwhelmed can be expensive, time-
consuming
75. Non-relational
databases
• Don’t store things in tables!
• Don’t pre-define a schema
• No SQL!
• Indeed, known as “NoSQL” databases
• Only common factor: No SQL, non-
relational
77. MapReduce / Hadoop
• Google and Yahoo do it like this:
• Split data across many servers
• Run a function on all of those servers
• Collect the results
• Display to the user!
• (Too slow? Add more servers!)
78. NoSQL: Good news
• Often easier to administer, configure
• Scaling to multiple servers is a no-brainer
• No new programming language (SQL)!
• Better fit for certain kinds of data
• Better performance than a relational DB
• Lots of options to choose from!
79. NoSQL: Bad news
• Speed is in the eye of the beholder
• Is “eventually consistent” good enough?
• Non-normalized data — ugh!
• Reporting can be harder
• Less tested than relational databases
• Can you trust your data to them?
80. Meanwhile...
• Our browsers are displaying HTML
• HTML had several problems:
• Standardized set of tags
• Making it easy for programs to parse
• Semantic, display content were mixed
81. HTML standards
• HTML — several versions, several
standards, none universally accepted
• XML — lets us create HTML-like
languages, for computer conversations
• XHTML — HTML for pedantic people
• It was a big mess!
82. HTML5
• Relaxes much of the formality of XHTML,
while remaining easy for computers to read
• Backward compatible to a large degree
• Adds a number of tags for better semantics
• Best of all: Lots of new JavaScript goodies
• More on this in a moment
94. And more
• Validation — built-in validation of form
element inputs, via regular expressions
• No more JavaScript plugins!
• Sliders — more natural numeric inputs
• Canvas — draw any picture you might like,
and detect/change it with software
• Hints in text fields (e.g., “search”)
95. My favorite
• Private data in attributes!
• Any attribute starting with “data-” is
considered valid
• A great way to stash information inside of
HTML elements without violating standards
96. Oh, yeah
• These don’t all work.
• Many of them don’t work on any browser
• Most work on only some browsers.
• What to do? Wait. Or use Modernizr,
which uses JavaScript to detect features.
• If a feature isn’t there, you can fall back
97. CSS
• Cascading Style Sheets
• Split semantic markup from presentation
• One CSS file can apply to an entire site
• No more “style” tags in your text!
• Easy to move place things, create effects
100. It gets better
• You can set styles for when the user’s
mouse hovers over or clicks on an element
• In other words: Cheap animation!
• Many uses of JavaScript (e.g., some menus)
can now be done with simple CSS
• You can make beautiful sites with CSS
101. CSS3: Cool effects
• Rounded corners
• Transparency
• Text shadows and drop shadows
• Gradients
102. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
p[id^=”foo”] { background: green}
p[id$=”foo”] { background: green}
p[id*=”foo”] { background: green}
103. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
Equals
p[id^=”foo”] { background: green}
p[id$=”foo”] { background: green}
p[id*=”foo”] { background: green}
104. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
Starts with
p[id^=”foo”] { background: green}
p[id$=”foo”] { background: green}
p[id*=”foo”] { background: green}
105. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
p[id^=”foo”] { background: green}
Ends with
p[id$=”foo”] { background: green}
p[id*=”foo”] { background: green}
106. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
p[id^=”foo”] { background: green}
p[id$=”foo”] { background: green}
Contains
p[id*=”foo”] { background: green}
107. OK, that’s nice.
• But really, the big news with HTML5
doesn’t have to do with HTML at all.
• Instead, it has to do with JavaScript.
• Remember JavaScript?
• It’s the programming language that we love
to hate. (At least, I used to.)
108. JavaScript
• Originally “LiveScript,” a language that
executes programs in the browser
• Renamed “JavaScript” when an unrelated
language (“Java”) stole the thunder
• Renamed (officially) ECMAScript for
standardization purposes
• No one actually calls it this
109. HTML5 turns the
browser into a smart,
powerful, networked
application platform.
JavaScript makes it
possible.
110. Powerful? Huh?
• Didn’t I just say that I love to hate
JavaScript?
• And then I said that it was powerful?
• What gives?
111. JavaScript is
the new hotness
• Browsers are competing for fastest, best
• Google’s V8
• Mozilla’s TraceMonkey (and JägerMonkey)
• Safari’s Nitro
• IE’s Chakra
• JavaScript is faster, more stable than ever!
112. Server-side JavaScript!
• The latest JavaScript development
• Run it on your server!
• Why? Because it’s super-fast
113. Also: frameworks
• Remove the drudgery of JavaScript
• Handle differences between browsers
• Make it easy to perform common tasks
• Lots of plugins available
• For me, it’s the difference between pain and
pleasure when working with JavaScript
115. jQuery
• jQuery has taken the world by storm
• Super-easy to use
• Extremely fast
• Open source (of course!)
• Easy to write plugins
• Lots of plugins are available
116. Ajax
• One reason for JavaScript libraries: Ajax!
• Make HTTP requests from within the page
• No refresh! Just get results from the
server, and modify the page accordingly
• This has revolutionized our view of Web
pages
118. Ajax
POST /login
name=reuven&password=secret
Browser Server
119. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
120. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
121. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
122. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
123. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
124. Ajax isn’t everything
• What if I want a chat application, or
something else that stays open?
• What if I want to execute more than one
JavaScript function at a time?
• What if I want to store things locally?
• HTML5 provides all this — and more!
125. Canvas
• A complete drawing area, in your browser!
• Use JavaScript to:
• Draw arbitrary shapes
• Detect the mouse
• Detect the drawing
• The end of Flash? Maybe...
126. Geolocation
• Your browser can know where you are!
• It can send this info to the server
• It’s not perfect, but still pretty good
• To avoid privacy issues, users are always
asked if their location should be shared
127. Inter-page
communication
• Modern Web apps can span multiple pages
• HTML5 makes it easy for two pages (from
the same server) to communicate
• The receiver knows which server sent the
data — so it can filter incoming messages,
as well as screen them for security
128. Web sockets
• This is potentially the biggest deal of all
• Ajax allows for server connections. But:
• High overhead
• Stateless
• Web sockets have low overhead, and they
stay open as long as you need!
131. Using Web sockets
var weatherSocket = new
WebSocket("ws://localhost:8080");
weatherSocket.onopen = function(e)
{ alert("Opened weather socket");};
132. Using Web sockets
var weatherSocket = new
WebSocket("ws://localhost:8080");
weatherSocket.onopen = function(e)
{ alert("Opened weather socket");};
weatherSocket.onmessage =
function(e) { alert("Got message: "
+ e.data);};
133. Using Web sockets
var weatherSocket = new
WebSocket("ws://localhost:8080");
weatherSocket.onopen = function(e)
{ alert("Opened weather socket");};
weatherSocket.onmessage =
function(e) { alert("Got message: "
+ e.data);};
weatherSocket.onclose = function(e)
{ alert("Closing socket..."); };
134. What can sockets do?
• Chat servers
• Stock feeds
• Teleconferencing
• Who knows?
• Remember, HTTP was only invented after
sockets had been around for 15 years
135. Web workers
• Execute more than one thing at a time
• In other words:You can run JavaScript
functions in the background
• Process text
• Open Web sockets
• Perform calculations
136. Local storage
• Now Web apps can store data
• A little database of name-value pairs
var foo = localStorage.getItem("bar");
localStorage.setItem("bar", foo);
137. Local storage
• Now Web apps can store data
• A little database of name-value pairs
var foo = localStorage.getItem("bar");
localStorage.setItem("bar", foo);
var foo = localStorage["bar"];
138. Local storage
• Now Web apps can store data
• A little database of name-value pairs
var foo = localStorage.getItem("bar");
localStorage.setItem("bar", foo);
var foo = localStorage["bar"];
localStorage["bar"] = foo;
139. Media
• Standard (well, sort of) ways to play audio
and video
• No more Flash!
• Problem: No one format is supported by all
browsers
142. Don’t forget mobile!
• iOS and Android are growing massively
• Web site vs. native app?
• (Ask Jakob Nielsen — for now, apps are
better, but that won’t last for long)
• Ignore these users at your peril
143. Want a Web app?
• It used to be:
• “What operating system, language, and
database will I use?”
• Or:
• “How can I produce an interesting Web
site?”
144. Now it’s:
• What experience do we want to give
people?
• What will they be using to access our
system?
145. Those lead to a wide
variety of questions:
• What server language and framework?
JavaScript framework? Hosted or cloud?
• What type of database (relational, NoSQL)?
Which one? Hosted or cloud?
• Do we offer an API? A mobile app? Both?
• Which HTML5 features do we want to
use, and how do we gracefully degrade?
146. The key takeaway
• Web sites are far more than just static
text, blogs, or forums
• They’re full-fledged software applications
• Take advantage of these technologies, and
you can create a fabulous experience
• (Don’t take advantage of them, and your
competitors will!)
147. Oh, yeah: Testing
• Automated testing is amazing
• Your Web site should use it
• Most modern frameworks support some
sort of testing — if not, change framework
• Catch dumb bugs and issues before your
customers do!
• Faster and cheaper than people
148. My brain is too small!
• Yes, there’s a lot to learn
• Most of it can wait a little bit
• There are oodles of tutorials
and books that can help you
• Besides a lot of this is still
highly transitional
149. Whew!
• There’s a lot to the modern Web
• It’s fun and exciting, and continues to move
forward at breakneck speed
• Understanding as many of these parts as
possible will help you make better
decisions, and better applications!