Apache Spark aims to solve the problem of working with large scale distributed data -- and with access to over 500 million leaked passwords we have a lot of data to dig through.

2. Analyzing Pwned
Passwords with Spark
Kelley Robinson
@kelleyrobinson
Developer Evangelist

4. +

5. BIG DATA & SECURITY @KELLEYROBINSON
Spark: then and now
The state of passwords
Spark in action
Big Data ∩ Security

6. BIG DATA & SECURITY @KELLEYROBINSON

7. BIG DATA & SECURITY @KELLEYROBINSON

8. BIG DATA & SECURITY @KELLEYROBINSON
Apache Spark Ecosystem

9. BIG DATA & SECURITY @KELLEYROBINSON
Spark Abstractions
Then
Now
RDD (Resilient Distributed Dataset)
DataFrames / Datasets

10. https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html
@KELLEYROBINSONBIG DATA & SECURITY
RDDs
• Immutable & distributed
collection
• Unstructured data
• Low-level transformation
and control

11. BIG DATA & SECURITY @KELLEYROBINSON

12. BIG DATA & SECURITY
https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html
@KELLEYROBINSON

13. https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html
@KELLEYROBINSONBIG DATA & SECURITY
Datasets
• Structured data
• Strongly typed
• Fast

14. @KELLEYROBINSONBIG DATA & SECURITY
Datasets
• Structured data
• Strongly typed
• Fast
• SQL DSLs

15. BIG DATA & SECURITY @KELLEYROBINSON
Apache Spark Ecosystem

16. BIG DATA & SECURITY @KELLEYROBINSON
Scala has the most
robust language API

17. BIG DATA & SECURITY
https://www.slideshare.net/databricks/composable-parallel-processing-in-apache-spark-and-weld
@KELLEYROBINSON

18. BIG DATA & SECURITY @KELLEYROBINSON
Spark: then and now
The state of passwords
Spark in action
Big Data ∩ Security

22. @KELLEYROBINSONBIG DATA & SECURITY
Spark: then and now
The state of passwords
Spark in action
Big Data ∩ Security

23. https://twitter.com/dog_rates/status/986762231290490881

24. Benefits
Fast
Flexible
Good for exploration
Proven for large systems
BIG DATA & SECURITY @KELLEYROBINSON

25. Challenges
Opaque error messages
Operationalizing
Documentation
http://heather.miller.am/blog/launching-a-spark-cluster-part-1.html
BIG DATA & SECURITY @KELLEYROBINSON

27. BIG DATA & SECURITY
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/
@KELLEYROBINSON
👍💯
The missing Spark documentation

28. BIG DATA & SECURITY @KELLEYROBINSON
Spark: then and now
The state of passwords
Spark in action
Big Data ∩ Security

29. BIG DATA & SECURITY @KELLEYROBINSON

30. @KELLEYROBINSON

31. BIG DATA & SECURITY

33. THANK YOU!
@kelleyrobinson

34. Spark Resources
• Apache Spark
• Jacek's Spark Documentation
• Zeppelin
• RDDs vs. Datasets
• Running Spark on a Cluster
Security Resources
• Pwned Passwords
• Reverse SHA1 hashes
• LastPass and 1Password
• 2FA Guides
@KELLEYROBINSONBIG DATA & SECURITY