In this presentation we discuss the decision to move away from Ruby and outline the reasons we chose Scala.

1. Whitepages Practical
Experience Converting from
Ruby to Reactive
John Nestor and Dragos Manolescu
{jnestor,dmanolescu}@whitepages.com
August 13, 2014

2. Disclaimer
The following material is subjective and represents the views of
the presenters.
We hope our views and experience may be of use to others
making the transitive from Ruby to Scala, but your mileage may
vary.
May 15, 2013

3. Whitepages Typesafe Case Study
• WhitePages Rebuilds Core Parts of Application Stack with
Scala and Akka to Improve Scaling.
• http://j.mp/wpscala
May 15, 2013

4. Outline
• Background
• Technology
• People
May 15, 2013

5. Background

6. About Whitepages
• Top web and mobile site for finding phones, people and locations.
• 50M unique users per month.
• 35M search queries per day.
• 70 engineers, mostly in Seattle.
May 15, 2013

7. Why replace Ruby?
Problems with Ruby legacy systems:
• High latencies.
• High resource consumption: memory and processors.
• Poor tolerance of upstream degradation:
– Limited concurrency.
– Worker starvation when blocking on slow upstream.
– Poor connection management and healing.
May 15, 2013

8. Why replace Ruby with Scala?
• Elegant blend of Functional and OO paradigms.
• Static type system:
• Type inference avoids the excess boilerplate of Java.
• Compiler catches many errors.
• Runs on the JVM:
– Fast.
– Near seamless interop with JVM libraries.
– Reasonably mature tooling support.
• Actor-based concurrency framework: Akka.
May 15, 2013

9. Whitepages Reactive Services
• Service-oriented-architecture where communication is Thrift
or Json over HTTP.
• Both latency and throughput are very important.
• We require excellent logging and monitoring.
• Agility of development, testing, build and deployment provide
leverage and competitive advantage.
May 15, 2013

10. Results
Performance
Service p50 ms p99 ms Throughput RPS/core
DirSvc -
Scala 25 300 80
DirSvc -
Ruby
140 1200 7
May 15, 2013

11. Status of Reactive Scala Services
• We have been using Scala for a little more than 1.5 years.
• We started with 1 development team, we now have 4.
• We started with a single back end service. We now have 4 services in
production and another 6 in development.
• Our first project had 6 Scala developers, we now have over 20 Scala
developers.
• We changed not only language but also tools (more automation) and
processes (more agile).
May 15, 2013

12. Future Reactive Work
• Transition additional Ruby back-end services to Scala.
• Exploring new technologies:
• Spark and Mesos.
• A messaging system based on Kafka.
• A graph store based on Titan/Cassandra.
• A deployment and lifecycle-management system based on
Docker that uses remote actors.
May 15, 2013

13. Technology

14. • GitHub Enterprise
• IntelliJ
• Jenkins (for builds)
• Artifactory
• Yourkit
• SCoverage
• Scrooge (for thrift, from
Twitter)
Technology Selection
• Scala
• Akka
• Spray
• SBT – with custom plugins
• Typesafe Config
• Scala Test
• Coda Hale Metrics
• Persist Json
May 15, 2013

15. Whitepages Service Framework
• Configuration (based on Typesafe Config, with multi-level overrides).
• Monitoring (central actor that uses Coda Hale metrics and publishes to Graphite, Nagios
and Riemann).
• Logging (custom system with central actor, Json log content and real time streaming to
external analytics).
• Admin REST API.
• REST Thrift/JSON Server (based on Spray).
• Clients (Spray, JDBC, Redis, SOLR, Riak,… -- advanced error detection/ recovery).
May 15, 2013

16. Technology Successes
• Scala concise syntax. – Higher developer productivity.
• Use of async future based code. – Great performance.
• Immutable collections and functional programming. – Fewer bugs.
• Strong type checking. – Fewer bugs and more maintainable code (But meta-programming becomes
more difficult).
• Concurrency. – No race conditions or deadlock on complex concurrent system.
• Spray. – Great performance and an async API.
• SBT. – Able to easily customize to our needs (after steep learning curve).
• IntelliJ IDEA. – Scala support is good and keeps getting better.
• Typesafe Developer support contract. – Fast response, great answers to complex questions. (We
should have gotten it earlier!)
May 15, 2013

17. Technology Problems -1
• Poor documentation:
• SBT is the poster child.
• We still often have to read source code of Scala and Akka libraries.
• Stable APIs:
• Typesafe has made great strides here.
• Other components are still troublesome.
• We lost many weeks as a result of breaking changes in Scala Test.
• We currently can’t compile a 2.11 version of Kafka because the Zinc Scala
compiler is not backward compatible.
May 15, 2013

18. Technology Problems -2
• Performance analysis of concurrent constructs:
• We went with the Typesafe Console.
• It never fully worked and then was abandoned.
• There are some new tools, but we have not had time to evaluate.
May 15, 2013

19. Technology Problems -3
• Ecosystem:
• Not as mature as the Java ecosystem and often missing Scala specific
features. For example, Scala Mock does not support named
parameters.
• Some needed components were missing: until recently code coverage.
• In some cases there were too many options: over 10 Json libraries.
• Lots of Scala GitHub projects. Quality is variable; many become
abandoned.
May 15, 2013

20. Technology Problems -4
• Debugging:
• Debuggers work on the Java byte code, that for some Scala constructs
is quite convoluted.
• Debugging chains of futures is much harder than debugging nested
calls. Some kind of tracing is desperately needed.
• Actors have lots of power but can be exceptionally hard to debug. We
need both better approaches to building debug-able actors and better
debugging tools.
May 15, 2013

21. Technology Problems -5
• Language and Library Issues:
• Type erasure is a major flaw in the Scala language. It causes lots of
developer confusion and errors. The workarounds are awkward at
best.
• The lack of type checking on Actor constructors is an odd hole in the
type checking and a real annoyance.
• Some Scala code that looks simple and intuitive creates major
difficulties when a developer needs to understand what is really going
on. Implicits are one of many problems here.
May 15, 2013

22. Technology Overall
• The problems mentioned above create real day to day difficulties.
• In many cases Typesafe has been actively working to incrementally
correct some of these problems.
• Since we started using Scala, the number of problems has significantly
decreased and we expect that trend to continue.
• Overall, Scala/Akka is a great technology for building reactive systems
and its benefits far outweigh the problems.
May 15, 2013

23. People

24. Levels of Scala Expertise
• Basic OO features. Easy transition from most other languages including
Java and Ruby.
• Adding functional. Ruby has functional features that are similar. But
knowing when best to use functional versus OO requires experience.
• Adding concurrency. Via futures, promises, and actors. A whole new world
for Ruby programmers that requires a new way of thinking.
• API design. Not all developers need to design reusable APIs, but those that
do need a higher level of experience and feature mastery to do it well.
May 15, 2013

25. Building Scala Development Teams
• The are still not a large number of experienced Scala developers.
• Those out there are in considerable demand.
• We have followed a mixed strategy of both recruiting Scala developers and
also training existing Ruby developers to become Scala developers.
• Having an experienced Scala developer greatly accelerated the training of
Ruby developers.
Note: Using Scala, Akka and other advanced technologies makes Whitepages
very attractive to some of the very best developers.
May 15, 2013

26. Recruiting: Finding Developers
• Personal contacts. The best way to find great developers.
• Meet-ups and Events
• John Nestor is the organizer for the Seattle Scala meet-up.
• Whitepages hosts the Scala meet-up (and several others)
• Whitepages sponsors many other technology conferences and events.
• Job posts: local Seattle and internet Scala focused.
• College recruiting events. UW and other top CS schools.
May 15, 2013

27. Recruiting: Attracting Developers
• Strong support from the Whitepages CTO and CEO and other management for Scala
and other advanced technologies.
• We have a great office (recently completely redone)
• We provide developers with the best workstations (top-end MacBook pros).
• We have competitive benefits, excellent work-life balance and a collaborative
supportive culture.
May 15, 2013

28. Training: External Resources
• Books. Odersky: Programming in Scala, and others
• Coursera. Functional programming, reactive programming.
• Typesafe training classes. Both Seattle public and in-house.
• Typesafe developer support contract and consulting. Particularly valuable
if you don’t have in-house Scala expertise.
• Conferences. We sent people to Scala Days in NY and Berlin.
May 15, 2013

29. Training: Internal Resources
• Whitepages developers are in a single office and come in most days.
• This permits a high degree of personal interactions.
• Those interactions enable rapid learning and high code quality.
• This would be much harder with multiple offices and/or widely distributed
teams.
May 15, 2013

30. Training: Internal Approaches
• Expert mentoring.
• Provide quick answers to questions.
• Detailed reviews and feedback for Ruby developers initial Scala code.
• Code reviews
• Improve code quality.
• Help those learning Scala.
• Develop a shared culture and style.
• We use a mixture of group and pair code reviews.
• Electronic means like code review tools and pull requests were less effective.
May 15, 2013

31. Training: Success and Failure
• Not everyone made a successful transition from Ruby to Scala.
• Next slide give some positive and negative indicators…
May 15, 2013

32. Training: What to Look For
• Desire to learn new technologies.
• Desire to learn new ways to think.
• Prior functional language experience.
• Likes to work closely with others.
May 15, 2013

33. Scala Features to Avoid
• Scala is a very large language.
• New developers can easily get lost on the rich feature set. Often there are
several different ways of doing something, and they don’t always know
how to choose.
– We also provide the following list of features for new developers to
avoid (or to fully understand before using).
– We encourage asking more experienced developers to provide that
understanding.
– These features are also flags for discussion in code reviews.
May 15, 2013

34. Scala Features to Avoid
• Reflection and macros
• asInstanceOf
• Top level objects with mutable
state
• Lazy vals
• Scala enumerations
• ? (the ask pattern)
• onComplete (unless body
enclosed in try)
• Nulls
• Return, break, continue
• Mutable collections
• var, while, until
• Generics
• Implicits
• Actors
• Await
May 15, 2013

35. We Are Hiring*!
May 2013
*Smart people (not just technologies) are key to our success

36. Thank you!
Questions?