O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
digitalocean.com
Go Generate Pipelines
Pipelines?
Pipelines
• https://blog.golang.org/pipelines
• concurrency pattern
• “stages connected by channels"
Pipelines
• produce a channel of stuff
• creates at least one goroutine
• source -> [middle] -> [middle] -> sink
Example
• https://play.golang.org/p/4Awrr2PgFo
Pattern
• directional channels
• goroutine
• close what you create
• for/range to completion (previous stage closes)
Concurrency
• increase the concurrency by stage
• easy to add
• https://play.golang.org/p/_qQVZ0Rh73
Generators vs Pipeline
• generators create values when you ask for them
• pipeline produce values before you ask for them
...
Real Program Concerns
• leaking goroutines
• cancelation
• errors
Leaking Goroutines
• every goroutine must terminate!
• pipelines create many goroutines
• https://play.golang.org/p/jUZIv6...
Cancelation Strategies
• exit program
• close source stage let downstream drain naturally
• close all stages
Exit program
• easy
• really really easy
Close source & drain downstream
• source closing trickles down to consumers
• all goroutines can exit
• https://play.golan...
Close all stages
• `out <-` must be interruptible everywhere
• every channel producing function provided “done”
• https://...
Errors
• can happen anywhere
• second output channel
• https://play.golang.org/p/aUv5MrD7Bj
Errors Take 2
• https://godoc.org/golang.org/x/net/context
• need a “pipeline context”
• signal done and place to write er...
Pipelines
• `out <-` always interruptible by `done`
• close what you create
• error channel/pipeline context
• often repet...
Go Generate
Go Generate
• since go1.4
• “comments” in your go project
• //go:generate [command] [args…]
Just a command
• generate say hi (for fun)
• generate rm -fr / (for evil)
• generate printenv (for understanding)
Not a shell…
• go:generate echo $EDITOR > /tmp/nope
• go:generate cat /tmp/file | grep something
But shellish…
• $GOFILE
• $GOPACKAGE
• $GOARCH
• $GOOS
• go:generate something -out=$GOARCH_something.go
Meant to write files
• stderr for communicating to the user
• silent in proper operation
• side effect (file) then committed...
Pipeliner
My common pipelines
• batch(<-chan Event) (<-chan []Event)
• flatMap(<-chan gzipfile) (<-chan Event)
• from([]Event) (<-chan...
Pipeliner
• side-project
• saves me time
• may annoy you (bad error messages)
• github.com/supershabam/pipeliner
digitalocean.com
close(talk)
digitalocean.com
questions?
Próximos SlideShares
Carregando em…5
×

go generate pipeliner

344 visualizações

Publicada em

pipelines in golang

Publicada em: Software
  • Seja o primeiro a comentar

go generate pipeliner

  1. 1. digitalocean.com Go Generate Pipelines
  2. 2. Pipelines?
  3. 3. Pipelines • https://blog.golang.org/pipelines • concurrency pattern • “stages connected by channels"
  4. 4. Pipelines • produce a channel of stuff • creates at least one goroutine • source -> [middle] -> [middle] -> sink
  5. 5. Example • https://play.golang.org/p/4Awrr2PgFo
  6. 6. Pattern • directional channels • goroutine • close what you create • for/range to completion (previous stage closes)
  7. 7. Concurrency • increase the concurrency by stage • easy to add • https://play.golang.org/p/_qQVZ0Rh73
  8. 8. Generators vs Pipeline • generators create values when you ask for them • pipeline produce values before you ask for them • https://play.golang.org/p/R_XHMu1jBz
  9. 9. Real Program Concerns • leaking goroutines • cancelation • errors
  10. 10. Leaking Goroutines • every goroutine must terminate! • pipelines create many goroutines • https://play.golang.org/p/jUZIv6H4l2
  11. 11. Cancelation Strategies • exit program • close source stage let downstream drain naturally • close all stages
  12. 12. Exit program • easy • really really easy
  13. 13. Close source & drain downstream • source closing trickles down to consumers • all goroutines can exit • https://play.golang.org/p/pRpNspMG2z
  14. 14. Close all stages • `out <-` must be interruptible everywhere • every channel producing function provided “done” • https://play.golang.org/p/Rd-hEVic0O
  15. 15. Errors • can happen anywhere • second output channel • https://play.golang.org/p/aUv5MrD7Bj
  16. 16. Errors Take 2 • https://godoc.org/golang.org/x/net/context • need a “pipeline context” • signal done and place to write errors • WIP idea • https://play.golang.org/p/-sn3QGpc7X
  17. 17. Pipelines • `out <-` always interruptible by `done` • close what you create • error channel/pipeline context • often repetitive…
  18. 18. Go Generate
  19. 19. Go Generate • since go1.4 • “comments” in your go project • //go:generate [command] [args…]
  20. 20. Just a command • generate say hi (for fun) • generate rm -fr / (for evil) • generate printenv (for understanding)
  21. 21. Not a shell… • go:generate echo $EDITOR > /tmp/nope • go:generate cat /tmp/file | grep something
  22. 22. But shellish… • $GOFILE • $GOPACKAGE • $GOARCH • $GOOS • go:generate something -out=$GOARCH_something.go
  23. 23. Meant to write files • stderr for communicating to the user • silent in proper operation • side effect (file) then committed to repository • can also generate _test.go file
  24. 24. Pipeliner
  25. 25. My common pipelines • batch(<-chan Event) (<-chan []Event) • flatMap(<-chan gzipfile) (<-chan Event) • from([]Event) (<-chan Event)
  26. 26. Pipeliner • side-project • saves me time • may annoy you (bad error messages) • github.com/supershabam/pipeliner
  27. 27. digitalocean.com close(talk)
  28. 28. digitalocean.com questions?

×