This document discusses analyzing pwned passwords with Apache Spark. It provides an overview of Spark, including its evolution from RDDs to DataFrames/Datasets and Scala having the most robust API. It also discusses analyzing large datasets of pwned passwords with Spark for security purposes and exploring the intersection of big data and security.

2. Analyzing Pwned
Passwords with Spark
Kelley Robinson
@kelleyrobinson
Developer Evangelist

4. +

5. BIG DATA & SECURITY @KELLEYROBINSON
Spark: then and now
The state of passwords
Spark in action
Big Data ∩ Security

6. BIG DATA & SECURITY @KELLEYROBINSON

7. BIG DATA & SECURITY @KELLEYROBINSON

8. BIG DATA & SECURITY @KELLEYROBINSON
Apache Spark Ecosystem

9. BIG DATA & SECURITY @KELLEYROBINSON
Spark Abstractions
Then
Now
RDD (Resilient Distributed Dataset)
DataFrames / Datasets

10. https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html
@KELLEYROBINSONBIG DATA & SECURITY
RDDs
• Immutable & distributed
collection
• Unstructured data
• Low-level transformation
and control

11. BIG DATA & SECURITY
https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html
@KELLEYROBINSON

12. https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html
@KELLEYROBINSONBIG DATA & SECURITY
Datasets
• Structured data
• Strongly typed
• Fast

13. @KELLEYROBINSONBIG DATA & SECURITY
Datasets
• Structured data
• Strongly typed
• Fast
• SQL DSLs

14. BIG DATA & SECURITY @KELLEYROBINSON
Apache Spark Ecosystem

15. BIG DATA & SECURITY @KELLEYROBINSON
Scala has the most
robust language API

16. BIG DATA & SECURITY
https://www.slideshare.net/databricks/composable-parallel-processing-in-apache-spark-and-weld
@KELLEYROBINSON

17. BIG DATA & SECURITY
https://twitter.com/CamJo89/status/996497423621996544
@KELLEYROBINSON

18. BIG DATA & SECURITY @KELLEYROBINSON
Spark: then and now
The state of passwords
Spark in action
Big Data ∩ Security

22. @KELLEYROBINSONBIG DATA & SECURITY
Spark: then and now
The state of passwords
Spark in action
Big Data ∩ Security

23. https://twitter.com/dog_rates/status/986762231290490881

24. Benefits
Fast
Flexible
Good for exploration
Proven for large systems
BIG DATA & SECURITY @KELLEYROBINSON

25. Challenges
Opaque error messages
Operationalizing
Documentation
http://heather.miller.am/blog/launching-a-spark-cluster-part-1.html
BIG DATA & SECURITY @KELLEYROBINSON

27. BIG DATA & SECURITY
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/
@KELLEYROBINSON
👍💯
The missing Spark documentation

28. BIG DATA & SECURITY @KELLEYROBINSON
Spark: then and now
The state of passwords
Spark in action
Big Data ∩ Security

29. BIG DATA & SECURITY @KELLEYROBINSON

30. @KELLEYROBINSON

31. BIG DATA & SECURITY

33. THANK YOU!
@kelleyrobinson

34. Spark Resources
• Apache Spark
• Jacek's Spark Documentation
• Zeppelin
• RDDs vs. Datasets
• Running Spark on a Cluster
Security Resources
• Pwned Passwords
• Reverse SHA1 hashes
• LastPass and 1Password
• 2FA Guides
@KELLEYROBINSONBIG DATA & SECURITY