Pipe2Py is a tool that converts Yahoo Pipes workflows into Python code so they can still be run if Yahoo Pipes shuts down. Each Yahoo Pipes module is coded as a separate Python module. Pipe2Py translates Yahoo Pipes into pipelines of Python generators to closely match the original data flow. This allows users to prototype workflows visually in Yahoo Pipes and then generate Python code to host themselves independently of Yahoo Pipes.

1. Introducing Pipe2Py : Converting Yahoo Pipes to Python Code Original code: Greg Gaughan Additional development: Tuukka Hastrup Based on an original idea by: Tony Hirst , Dept of Communication and Systems, The Open University

2. pipes.yahoo.com

3. But what happens if Yahoo Pipes dies?

4. Pipe2Py github.com/ggaughan/pipe2py

5. Yahoo pipelines are translated into pipelines of Python generators* to give a close match to the original data flow. * based on ideas by David Beazley http://www.dabeaz.com/generators-uk

6. Each Yahoo module is coded as a separate Python module.

7. So you can use Yahoo Pipes as a graphical rapid prototyping application , and then generate a Python code equivalent you can host yourself So what?

8. download code http://github.com/ggaughan/pipe2py to dev8d/pipes/pipe2py set path export PYTHONPATH=dev8d/pipes installation

9. simplejson* sudo easy_install simplejson dependencies * only needed for Python pre 2.6

10. test directory python testbasics.py unit tests

11. python compile.py -p pipelineid compilation - direct from Yahoo Pipes generates pipe_ pipelineid .py

12. python compile.py pipelinefile.json compilation - from a file generates pipelinefile .py

13. python pipe_ pipelineid .py command line execution runs pipe_ pipelineid .py

14. from pipe2py import Context from pipe2py.modules import * def pipe_404411a8d22104920f3fc1f428f33642(context, _INPUT, conf=None, **kwargs): "Pipeline" if conf is None: conf = {} forever = pipeforever.pipe_forever(context, None, conf=None) sw_502 = pipefetch.pipe_fetch(context, forever, conf={u'URL': {u'type': u'url', u'value': u'http://blog.ouseful.info/feed'}}) _OUTPUT = pipeoutput.pipe_output(context, sw_502, conf={}) return _OUTPUT compiled code of the form...

15. Each call to the final generator will ripple through the pipeline issuing .next() calls onto the previous generator until the source is exhausted.

16. Each item is typically passed through the whole pipeline one at a time, so: memory usage is kept to a minimum no module is waiting on an earlier module to finish processing the whole data set by adding queues between the modules they could easily be made to run in parallel, each on a different CPU, to give great scalability

17. from pipe2py import Context import pipe_9dc8014dcfd34c834a960321afde68d9 as p C=Context() r = p.pipe_9dc8014dcfd34c834a960321afde68d9(C,None) for i in r: print i print i['title'] usage - compiled pipe

18. from pipe2py.compile import parse_and_build_pipe from pipe2py import Context pipe_def = """json representation of the pipe""" p = parse_and_build_pipe(Context(), pipe_def) for i in p: print i usage - interpreted pipe

19. context = Context(describe_input=True) p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(context, None) need_inputs = p print need_inputs >>> [(u'0', u'username', u'Twitter username', u'text', u''), ... (u'1', u'statustitle', u'Status title [string] or [logo] means twitter icon', u'text', u'logo')] ''' That is, tuples of the form (position, name, prompt, type, default) ''' usage - user inputs #1 Identifying console prompts

20. C = Context(inputs={'username':'greg', 'statustitle':'logo'}, console=False) p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(C, None) for i in p: print i usage - user inputs #2 avoiding console prompts

21. Yahoo Pipes modules: Pipe2Py implementation progress

22. Yahoo Pipes modules: Pipe2Py implementation progress

23. Yahoo Pipes modules: Pipe2Py implementation progress

24. ;-) One more thing...

25. pipes-engine.appspot.com pipe2py hosting on Google App Engine

26. - generate test pipes that work of increasing complexity - generate test pipes that don't work - commit pipe2py patches for test pipes that don't work How can you help?

27. - simplify installation (easy_install?) - identify a good convention for integrating pipe2py compiled pipes in arbitrary code - - identify a good convention for inserting arbitrary python functions into, or in-between, compiled pipe2py pipelines How else can you help?

28. the next step: produce an open source front end visual editor? wireit? pypes? Anything else?

29. generate a ready-to-run instance of a Google App Engine configuration based around a compiled pipe? Anything more else?

30. Pipe2Py github.com/ggaughan/pipe2py