The next web will be about flow, this flow will be user generated pipelines through applications and services. Unlike before these Pipelines will be definable, non-proprietary and shareable by anyone

1. Pipelines: plumbing for the next web
Ian Forrester
backstage.bbc.co.uk
cubicgarden.com

2. Overview
In depth discussion of the abstract concept of flow *
•
Not discussing the merits of UNIX or XML pipelines
•
Assumes an understanding of a pipe and pipeline
•

3. Imagine. . .
Processing data from your local machine with data
•
on-line
Chaining together services you use regularly
•
Automating the process so it ran at convenient
•
times
Not requiring programming knowledge
•
A process that is open, extensible and sharable
•

4. The Landscape

5. Hot on the Horizon
REST APIs are dotted all over the landscape
•
Web applications are opening up
•
Useful web services with revenue models
•
RSS and ATOM feeds are becoming ubiquitous
•
Desktop search is standard in Vista and OSX
•
Widgets and Gadgets are becoming useful
•
New projects to organise desktop meta-data
•

6. REST XML API Everywhere

7. REST XML API Everywhere

8. Opening Up the Silos

9. Opening Up the Silos

10. Web Services with Revenue Models

11. Ubiquitous Feeds

12. Widgets and Gadgets

13. Desktop Search

14. Semantic Desktop projects

15. Semantic Desktop projects

16. File System Meta-data

17. How People are Building Things
Particls
•
Automator
•
Yahoo Pipes
•

18. Touchstone / Particls
Particls: An alerts/updates and attention
•
management platform
Ingests many different input sources via input
•
adapters
Works out importance of new information to the
•
user and displays a proportional alert
Alerts take many forms based on urgency and relevancy
•
levels
Alerts are controlled with output adapters.
•

19. Particls

20. A Pipeline Application

21. XML Configurations

22. Outline Processing Markup Language (OPML)

23. Attention Profiling Markup Language (APML)

24. Particls Internals

25. Learning from Particls
Particls Limitations:
•
Main purpose is an alert and attention engine
•
No standard underline definable language
•
Not sharable outside of the pipes application
•
Choice of inputs and outputs limited
•
No automation
•
Semi-proprietary
•
Particls Highlights
•
Breakthrough web application
•
GUI has all the right elements
•
Form level abstract is great
•
Cloning pipes is very smart
•
Pushing new attention markup standard
•

26. Automator
A beautiful graphical user interface for Applescript
•
“Accomplish all of your time-consuming, repetitive manual
•
tasks quickly, efficiently and effortlessly with Automator
Workflows. It’s simple to create custom Workflows just by
dragging items, pointing and clicking. Perform the task
once using your Automator Workflow or many times —
even share Workflows with friends.”
Apple.com
•

27. Apple Automator

28. Automator

29. Applescript

30. Automator Internals

31. Learning from Automator
Automator Limitations
•
Proprietary
•
Requires Applescript
•
Underlying definable language non-XML based
•
Works better as automation software
•
Not natively web enabled
•
Automator Highlights
•
Break through desktop application
•
Powerful Gui
•
Form level abstract is great
•
Great automation
•
Sharing of scripts very simple
•
Choice of inputs and outputs extensible via code
•

32. Yahoo Pipes
New beta service that enables users to create
•
mash-ups with a graphical user interface
Mash-ups can be saved so other people can edit
•
and re-use them
Still geeky but easier than creating mash-ups by
•
coding
“a milestone in the history of the Internetquot;
- Tim O'Reilly
•

33. Y! Pipes

34. Y! Pipes
s

35. Y! Pipes

36. Y! Pipes Internals

37. Learning from Y! Pipes
Yahoo! Pipes Limitations
•
Only has access to data and content on-line
•
No XSL transformations
•
No standard underlying definable language
•
Not sharable outside of the Y! Pipes application
•
Choice of inputs and outputs limited
•
No automation
•
Proprietary
•
Yahoo! Pipes Highlights
•
Breakthrough web application
•
GUI has all the right elements
•
Form level abstract is great
•
Cloning pipes is very smart
•

38. Fitting the Puzzle Together
Combine:
•
Underlying XML definitions of Particls
•
Slimline and smooth look of Particls
•
Form control of Automator
Power and control of Automator
•
Visual pipeline GUI of Yahoo! Pipes
•
Automation of Automator
•
Sharing of Automator and Yahoo! Pipes
•

39. flow *

40. So what is flow*?

41. flow *

42. flow *

43. The flow * System

44. Adding Widgets to the Mix

45. Separate the Desktop API

46. But. . .there is no flow *
flow * currently does not exist
•
flow * is not just an application
•
flow * is not just a service
•
flow * is a combination of good practices served up
•
for the power users

47. My Current flow * Setup

48. More Configuration Ideas

49. More Configuration Ideas

50. More Configuration Ideas

51. Flow* is. . .
The glue that allows you to break out of the silos
•
on-line and off-line
The ability to pipe services together using their
•
APIs and feeds as though programming code or
interacting with the site directly
XPROC documents with namespaces elements
(User-generated Pipelines)
Three levels of abstraction in one file
•

52. Core User-generated Pipeline Principles
Definable
•
Core pipeline defined as XML
•
Graphical
•
Levels of abstraction from the XML
•
Standard
•
Using the new W3C standard XPROC
•
Shareable
•
As easy to share as a OPML file
•
Open
•
Can be extended and plugged into widget engines
•
Non-proprietary
•
No lock-in, uses standard web and XML technologies
•

53. Three Levels

54. Three Levels of Abstraction

55. Three Levels of Users, One Purpose

56. XPROC Level

57. Pipe GUI Level

58. Task Level

59. Pipelines Options

60. The Current Landscape

61. Particls: An Attention Platform

62. Automator: Strictly for OSX Only

63. Y! Pipes: No Access to the Local System

64. Y! Pipes: Could Access Local System

65. Yahoo widgets authenticator

66. Securing the Desktop and Cloud

67. Y! Pipes: All the Right Elements

68. Microsoft just around the corner?

69. Pipelines

70. User-generated Pipelines
Pipelines generation is only limited by our own
•
creativity and the APIs available
XPROC feeds Flow* like Javascript feeds
•
Greasemonkey
User-generated Pipelines could be...
•
XPROC+SVG+XFORMS
•
XPROC+XUL
•
XPROC+RDF+XFORMS
•
XPROC+FlexML
•
XPROC+XHTML
•

71. Pipeline Example

72. Pipeline Example

73. Complex Pipeline Example

74. Complex Pipeline Example

75. Complex Pipeline Example

76. The Ecosystem of the API

77. No fully functional API?
Forget it now... your dead
•

78. Described in XPROC (part 1)
<?xml version=quot;1.0quot; encoding=quot;UTF-8quot;?>
<p:pipeline name=quot;encoder examplequot; xmlns:p=quot;http://www.w3.org/2007/03/xprocquot;>
<p:input port=quot;videoquot; href=quot;file://///127.0.0.1/my videos/quot;/>
<p:choose name=quot;free-or-payquot;>
<p:when test=quot;form:///encoder.dropdown='yes'quot;>
<p:output port=quot;http://www.amazon.com/ws/ec2quot;>
<p:parameter name=quot;amazon/ec2imagequot; value=quot;video-encodingquot;/>
<p:parameter name=quot;amazon/encodetoquot; value=quot;flvquot;/>
<p:parameter name=quot;amazon/encodetoquot; value=quot;h.264quot;/>
<p:parameter name=quot;amazon/encodetoquot; value=quot;mpeg4quot;/>
<p:parameter name=quot;amazon/accountquot; value=quot;form:///amazon.accountquot;/>
<p:parameter name=quot;amazon/passwordquot; value=quot;form:///amazon.passwordquot;/>
<p:parameter name=quot;amazon/accesskeyquot; value=quot;form:///amazon.accesskeyquot;/>
<p:parameter name=quot;amazon/piorityquot; value=quot;1quot;/>
</p:output>
</p:when>
<p:when test=quot;form:///encoder.dropdown='no'quot;>
<p:output port=quot;http://www.heywatch.com/quot;>
<p:parameter name=quot;encodequot; value=quot;flvquot;/>
<p:parameter name=quot;encodequot; value=quot;h.264quot;/>
<p:parameter name=quot;encodequot; value=quot;mpeg4quot;/>
<p:parameter name=quot;loginquot; value=quot;form:///accountquot;/>
<p:parameter name=quot;passwordquot; value=quot;form:///passwordquot;/>
</p:output>
</p:when>
<p:otherwise>
<p:output port=quot;local://start.loocal.applicationquot; href=quot;file://///127.0.0.1/C/program files/mediacleaner/mediaclean.exequot;/>
</p:otherwise>
</p:choose>

79. Described in XPROC (part 2)
<p:choose name=quot;stoarage-optionsquot;>
<p:when test=quot;form:///storage.selector='amazon'quot;>
<p:output port=quot;http://www.amazon.com/ws/s3quot;>
<p:parameter name=quot;amazon/accountquot; value=quot;form:///amazon.accountquot;/>
<p:parameter name=quot;amazon/passwordquot; value=quot;form:///amazon.passwordquot;/>
<p:parameter name=quot;amazon/accesskeyquot; value=quot;form:///amazon.accesskeyquot;/>
<p:parameter name=quot;amazon/bucketquot; value=quot;form:///amazon.bucketquot;/>
</p:output>
</p:when>
<p:when test=quot;form:///stroage.selector='bliptv'quot;>
<p:output port=quot;http://www.blip.tvquot;>
<p:parameter name=quot;loginquot; value=quot;form:///bliptv.accountquot;/>
<p:parameter name=quot;passwordquot; value=quot;form:///bliptv.passwordquot;/>
</p:output>
</p:when>
<p:when test=quot;form:///stroage.selector='youtube'quot;>
<p:output port=quot;http://www.youtube.comquot;>
<p:parameter name=quot;loginquot; value=quot;form:///youtube.accountquot;/>
<p:parameter name=quot;passwordquot; value=quot;form:///youtube.passwordquot;/>
</p:output>
</p:when>
<p:otherwise>
<p:output port=quot;resultquot; href=quot;ftp://user:pass@ftp.cubicgarden.com/myvideo/quot;/>
</p:otherwise>
</p:choose>
</p:pipeline>

80. What happens next?

81. To do
Build flow* using a Rich Internet Application
•
language like XUL or Apollo
Write an Authentication layer for the APIs – Open
•
ID based broker system for APIs?
Decide on one way to write User-generated
•
Pipelines which works for flow*
Write schema for namespaced extensions into
•
user-generated pipelines (XPROC document)
Investigate the semantic desktop projects
•
Explorer W3Cs draft Widget specification
•

82. Summary
User generated pipelines are viable for:
•
People wanting automation
•
Non-programmers
•
Power users who can't be bothered to load up their IDE
•
Flow* (although only in early development) is very
•
powerful
All the elements are in place, it just needs a developer or
•
company with a some vision

83. Final thought
Mashing-up, remixing, sharing is part of our culture
•

84. Thanks for listening, questions?
Ian Forrester - pipelines@cubicgarden.com
•
http://www.cubicgarden.com/blojsom/blog/pipelines/
•
Flickr pictures : Creative Commons Attribution-NonCommercial-NoDerivs License
•
http://www.flickr.com/photos/andidfl/
•
http://www.flickr.com/photos/_f1guy68_/
•
http://www.flickr.com/photos/coolinbox/
•
http://www.flickr.com/photos/belljar/
•
http://www.flickr.com/photos/dgbalancesrocks/
•
http://www.flickr.com/photos/whinger/
•
http://www.flickr.com/photos/wasabicube/
•
http://www.flickr.com/photos/jacy
•