pipelines in golang with some pitfalls to avoid and ideas for generating common patterns

1. digitalocean.com
Go Generate Pipelines

2. Pipelines?

3. Pipelines
• https://blog.golang.org/pipelines
• concurrency pattern
• “stages connected by channels"

4. Pipelines
• produce a channel of stuff
• creates at least one goroutine
• source -> [middle] -> [middle] -> sink

5. Example
• https://play.golang.org/p/4Awrr2PgFo

6. Pattern
• directional channels
• goroutine
• close what you create
• for/range to completion (previous stage closes)

7. Concurrency
• increase the concurrency by stage
• easy to add
• https://play.golang.org/p/_qQVZ0Rh73

8. Generators vs Pipeline
• generators create values when you ask for them
• pipeline produce values before you ask for them
• https://play.golang.org/p/R_XHMu1jBz

9. Real Program Concerns
• leaking goroutines
• cancelation
• errors

10. Leaking Goroutines
• every goroutine must terminate!
• pipelines create many goroutines
• https://play.golang.org/p/jUZIv6H4l2

11. Cancelation Strategies
• exit program
• close source stage let downstream drain naturally
• close all stages

12. Exit program
• easy
• really really easy

13. Close source & drain downstream
• source closing trickles down to consumers
• all goroutines can exit
• https://play.golang.org/p/pRpNspMG2z

14. Close all stages
• `out <-` must be interruptible everywhere
• every channel producing function provided “done”
• https://play.golang.org/p/Rd-hEVic0O

15. Errors
• can happen anywhere
• second output channel
• https://play.golang.org/p/aUv5MrD7Bj

16. Errors Take 2
• https://godoc.org/golang.org/x/net/context
• need a “pipeline context”
• signal done and place to write errors
• WIP idea
• https://play.golang.org/p/-sn3QGpc7X

17. Pipelines
• `out <-` always interruptible by `done`
• close what you create
• error channel/pipeline context
• often repetitive…

18. Go Generate

19. Go Generate
• since go1.4
• “comments” in your go project
• //go:generate [command] [args…]

20. Just a command
• generate say hi (for fun)
• generate rm -fr / (for evil)
• generate printenv (for understanding)

21. Not a shell…
• go:generate echo $EDITOR > /tmp/nope
• go:generate cat /tmp/file | grep something

22. But shellish…
• $GOFILE
• $GOPACKAGE
• $GOARCH
• $GOOS
• go:generate something -out=$GOARCH_something.go

23. Meant to write files
• stderr for communicating to the user
• silent in proper operation
• side effect (file) then committed to repository
• can also generate _test.go file

24. Pipeliner

25. My common pipelines
• batch(<-chan Event) (<-chan []Event)
• flatMap(<-chan gzipfile) (<-chan Event)
• from([]Event) (<-chan Event)

26. Pipeliner
• side-project
• saves me time
• may annoy you (bad error messages)
• github.com/supershabam/pipeliner

27. digitalocean.com
close(talk)

28. digitalocean.com
questions?