A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and tools, both developed at Twitter and otherwise.
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Â
The Why and How of Scala at Twitter
1.
2. The Why and How
of Scala at Twitter
Alex Payne (@al3x)
Twitter Infrastructure
3. Whoâs this guy?
â˘Working at Twitter since
2007, before it was even a
company.
â˘Spent a couple years on the
API.
â˘Co-authored Programming
Scala (OâReilly, 2009).
â˘Into programming
languages, scaling stuff.
5. So, like, Ruby or Scala?
â˘Both!
â˘Ruby mostly on the
frontend.
â˘Scala mostly on the
backend.
â˘Thrift and data stores to
connect the two.
6. Elevator Pitch
⢠Itâs fun.
⢠Itâs fast.
⢠Itâs runs on a great virtual machine (JVM).
⢠Itâs got an awesome community.
⢠It can borrow from Java.
7. No, seriously, why?
⢠A rich static type system that gets out of your
way when you donât need it, is awesome when
you do.
⢠Flexible syntax makes it easy to write
readable, maintainable code.
⢠Traits for cross-cutting modularity.
⢠Lots of OOP goodness.
8. Iâm not sold yet.
⢠How about the choice between immutable and
mutable variables?
⢠Powerful functional programming: mapping,
filtering, folding, currying, so much more.
⢠Lazy values.
⢠Pattern matching.
⢠XML literals, and clean syntax for working
with XML documents built right in!
9. What about concurrency?
⢠You can do Actor concurrency (message
passing), kinda like Erlang.
⢠Or, use threads.
⢠Or, use all the java.util.concurrent tools.
⢠Or, use other JVM concurrency frameworks
(ex: Netty, Apache Mina).
13. We build services.
⢠Isolated components, can swap them out.
⢠Talk to the rest of the system via Thrift (for
now, maybe Avro in the future).
⢠Independently tested for correctness, load.
⢠Can have custom operational properties.
14. Scala Services at Twitter
⢠Kestrel: queuing.
⢠Flock (and Gizzard): social graph store.
⢠Hawkwind: people search.
⢠Hosebird: streaming API.
⢠more in the works all the time...
16. We build libraries.
⢠Reusable chunks of code that we can share
between projects.
⢠We open source them.
⢠We try to keep the âNIHâ to a minimum, but
we also have very particular requirements.
17. Ostrich
⢠Gather stats from inside your running process.
⢠Counters, gauges, and timings.
⢠Share stats many ways: JMX, JSON-over-
HTTP, plain text Telnet-style socket, log files.
22. xrayspecs
⢠A set of extensions to the fantastic Specs BDD
test framework.
⢠Handles testing concurrency, time, creating
temporary folders.
⢠The concurrency features have been rolled into
the main distribution of Specs!
23. xrayspecs Examples
âresponse arrives from serverâ in {
get(â/exampleâ) must eventually(notBe(null))
}
âtime should stopâ in {
Time.freeze()
val nowSnapshot = Time.now
Thread.sleep(30)
Time.now mustEqual nowSnapshot
}
24. scala-json
⢠A cleaned-up version of the official Scala JSON
codec.
⢠Makes clever use of parser combinators â a
good way to learn about them.
⢠Heavily tested, fixes a bunch of edge cases.
26. Other Twitter Libraries
⢠Naggati: build protocols for Apache Mina.
⢠Smile: a memcached client that uses Actors.
⢠Querulous: a nice database client.
⢠Jackhammer: a load testing framework
(coming soon).
⢠probably stuff I'm leaving out...
27. Stuff We Use
⢠There's great open source code in the Scala
community.
⢠We try to use and contribute back to third-
party projects.
⢠Follow @implicit_ly for new Scala releases.
⢠Also subscribe to Repopular Scala.
28. sbt â the simple build tool
⢠Scalaâs answer to Ant and Maven.
⢠Sets up new projects.
⢠Maintains project configuration, build tasks,
and dependencies in pure Scala. Totally open-
ended.
⢠Interactive console.
⢠Will run tasks as soon as files in your project
change â automatically compile and run tests!
29. specs
⢠As aforementioned: great way to do BDD
testing.
⢠Set up pre- and post-conditions for
individual tests, suites, etc.
⢠Extremely flexible and extensible.
⢠Great, responsive maintainer.
⢠Supports mocking (we like Mockito).
30. The IDE Question
⢠We've had the best luck with IntelliJ IDEA,
but it's still pretty rough.
⢠Most of us use a plain text editor like Emacs,
VIM, or TextMate.
⢠Combined with sbt, it's actually a really nice
workflow!