Jeff Hale presents on Prefect Blocks, a new feature of the Prefect library that allows users to create reusable configuration blocks for Python data pipelines. Blocks can be created through a graphical interface or code and stored on a server. This makes pipelines easier to collaborate on, share configurations for, and extend functionality. Blocks aim to provide the simplicity of low-code tools with the flexibility of Python. Prefect Blocks offer an alternative to hard-coding secrets and configurations or relying on environment variables by encrypting sensitive values.

1. Better Python Coding
with Prefect Blocks 🧱
Jeff Hale prefect.io
PyData Global 2022 linkedin.com/in/-jeffhale/

2. Jeff Hale
● Hawaiian shirts 🌺
● Hiking 🥾
● Ultimate frisbee 🥏

3. Jeff Hale
📊 + 🐍
● Teach 🍎
● Write ✍
● Organize events 🫂

4. I ❤ Prefect because it solves
common pain points elegantly

5. 🐍 open source library
🙂 growing company behind it

6. Why Prefect?
Usual answer:
Coordinate your dataflow! 🚀

7. Observation + Orchestration
🧐 + 🪄

8. How?
● Sprinkle in flow & task decorators
● Turn into a deployment
● Share with workspace users

9. Today:
Blocks

10. Why blocks?
● 🤗 More collaboration
● 🏛 Fewer errors
● 🔐 Increased security
● 🍴 Easier extendability
● 🕰 Time saved

11. What are blocks?
+ front-end magic
+ server
+ access controls

12. Stored in server database
● OSS Orion DB (SQLite or Postgres)
● Prefect Cloud

13. Problem ☹

14. “Data practitioners have historically been
forced to choose between the simple setup
experience of low-code data integration tools
and the flexibility of an all-Python pipeline.”

15. Simple form vs. Flexible 🐍

17. Blocks offer choices
🙂

18. Prefect Blocks are like a mullet
Structured form in front
Flexible code in back

19. Low-code connectors with a
first class code experience ✨

20. How do I get this Block
awesomeness?

21. Install
pip install -U prefect

22. Option 1
Create with HTML
form interface

23. Option 1

24. Option 1

25. Use it

26. Option 2
Create with 🐍 code

27. Use it

28. Prefect Blocks 🧱
Flexible creation and editing

29. Use case
You want to share
cloud provider
configuration
securely 🔐

30. 🔐 Options
1. Hard code secrets 🚫
Just don’t 😅

31. 2. Environment variables
● Viewable in logs
● Tied to machine

32. 3. Secrets manager
● Vendor lock-in ☁
● Steep learning curve 📈
● Missing configuration 🧭

33. 4. Build custom secrets manager
● Add configuration
● Encryption
● Server sync
● Bunch of code

34. Prefect Secret Blocks 🔐🧱
● Obfuscated pydantic SecretStr
● Encrypted at rest
● Can use HashiCorp Vault with
prefect-vault collection

35. Historical context

36. 1791 Charles Babbage
● Mathematician
● Philosopher
● Code breaker
● Inventor
By anonymous - Art UK, Public Domain,
https://commons.wikimedia.org/w/index.php?curid=97545883

37. Babbage’s Inventions
Ophthalmoscope Cowcatcher

38. Analytical engine
By Charles Babbage - Upload by Mrjohncummings 2013-08-28 15:10, CC BY-SA 2.0,
https://commons.wikimedia.org/w/index.php?curid=28024313

39. Punch cards
By Arnold Reinhold - I took this picture of an artifact in my possession. The card was created in the late 1960s or early1970s
and has no copyright notice., CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php?curid=775153

40. 1842 Ada Lovelace
First computer
program
By Margaret Sarah Carpenter - UK Government Art
Collection website.Original upload was at English wikipedia
at en:File:Ada_Lovelace.jpg, Public Domain,
https://commons.wikimedia.org/w/index.php?curid=35407

41. 1991 Guido van Rossum
Python released
By www.python.org - www.python.org, GPL,
https://commons.wikimedia.org/w/index.php?curid=34991651

42. 2018 Streamlit

43. 2018 Streamlit
Takes HTML/JavaScript/Jinja out of the
Python developer’s required skill set for
data website development! 🎉

44. 2018 Jeremiah Lowin
PyData DC talk
Task Failed Successfully

45. 2019
Prefect released

46. 2022
Prefect 2 released

47. Similar to Streamlit
Prefect Blocks allow Python developers
to build an HTML interface easily ⭐

48. Different than Streamlit
Prefect Blocks save configuration
information for reuse and sharing in
Python. 🐍

49. Block types

50. Prefect ships with many block
types

51. Basic Block Types
● String 🔤
● Date Time 📆
● JSON 📖
● Secret 🔐

52. Code Storage Block Types
● GitHub
● S3
● Azure
● GCS

53. Infrastructure Block Types
● DockerContainer
● KubernetesJob

54. Notification Block Types
● Email
● Slack
● Microsoft Teams
● Twillio
● PagerDuty

55. Zoom in 🔬
AWS S3 Block
Stores code to run on a schedule

58. A colleague who doesn’t want to
touch Python can add exactly the
configuration info needed! 🎉

59. Slack notifications

60. Need more block types?

61. Collections 🎉
● Python libraries
● Built by Prefect/community

64. Blocks from Collections
● Snowflake
● dbt
● GitLab
● Hex
● Great Expectations
● Fugue

65. Need even more block types?

66. Create your own!

67. Register them with the server
prefect block register

68. Block Type Shows in UI!

69. Prefect has helpful:
● Docs 📃
● Slack Community 📣
● Recipes 🍛
● Blog posts 📝
● Docstrings 📒
● CLI - - help 🔤

71. Block benefits

72. Fewer Errors
Blocks take the guesswork out of
configuring a connection

73. Composable
Can use multiple
blocks together in
the same code

74. Nestable

75. Nestable

76. Nestable

77. Versatile
Same core concept -
many different roles
● Storage
● Infrastructure
● Notifications

78. Reusable
Reuse a block in
multiple flows

79. Shareable
Workspace users
can share blocks

80. Extendable
“With blocks, people don't need to
fork Prefect to extend it.”

81. What else should I know
about Prefect? 🤔

82. Best option for dataflow coordination
1. Automatic logging
2. Automatic retries
3. Scheduling
4. Notifications
5. Caching
6. Easy async
7. Collaborative UI
8. Hybrid model
9. Integrations
10. Incremental adoption

83. Easy start

84. Add Prefect flow decorator

85. Want retries? 🪄

86. Want notifications? 🪄

87. Prefect Cloud
● Hosted
● Free tier with lots of goodies
● Enterprise tools

88. Prefect Cloud

89. Hybrid model
●

90. Prefect Cloud
2 billion tasks

91. One more blocks metaphor

92. ● System of pulleys & ropes
● Mechanical advantage
● Force multiplier 💪
Block and tackle

93. Easier to build great
things 🎉
By CR - Wikipedia en Español, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=38727

94. Why Blocks? 🧱
1. Create with HTML form or code
2. Use in code
3. Fewer errors
4. Secure
5. Composable
6. Nestable
7. Versatile
8. Reusable
9. Shareable
10. Extendable

95. How could your next
Python project
benefit from Blocks?

96. Thank you!
Better Python Coding with
Prefect Blocks 🧱
Jeff Hale prefect.io
PyData Global 2022 linkedin.com/in/-jeffhale/