SlideShare uma empresa Scribd logo
1 de 36
Baixar para ler offline
Kamaelia - Networking
  Using Generators
        Michael Sparks
  BBC Research & Development

     ACCU Python 2005, Oxford
Kamaelia
• Project to explore long term systems for large
  scale media delivery
  •   Forms a concurrency toolkit, focussed mainly on experimenting
      with network protocols.

• 2 key portions:
  •   Axon - Core Component infrastructure, based on communicating
      generators

  •   Kamaelia - Collection of components that use Axon.

• Aim: Scalable, easy & safe concurrent systems
                                                              (c) 2005 BBC R&D
Kamaelia Status
• Released as open source:
  •   http://kamaelia.sourceforge.net/

• Axon is at version 1.0.3, and considered feature
  stable.
  •   Runs on Linux, Windows (variety), Mac OS X

  •   Specialised distribution for Nokia Series 60 mobiles

• Kamaelia is at version 0.1.2, and growing
  •   Ability to write TCP & Multicast clients and servers

  •   Variety of simple servers, clients and protocols included
                                                                  (c) 2005 BBC R&D
Kamaelia Status
• Kamaelia 0.1.2:
  •   Tested on Linux, Windows (variety), Mac OS X

  •   Subset on Nokia Series 60 mobiles

• Ease of use hypothesis has been tested with 1 pre-
  university trainee, looks promising




                                                     (c) 2005 BBC R&D
Kamaelia Motivations
• Large Scale Streaming
 •   Several million streams per day

 •   Big events have tens of thousands of concurrent viewers

 •   Want to scale to handling millions of concurrent viewers

     •   Since this could happen.




                                                                (c) 2005 BBC R&D
Kamaelia Motivations
• What If 10 years from now...
• the BBC opened the entire archive?
  •   Creative Archive is NOT that ambitious! (AFAIK)

• the entire UK got broadband?
• Instantly hit long tail problems
  •   20 million homes?

  •   20 million different things?

  •   Not like 20 million people watching BBC1 !


                                                        (c) 2005 BBC R&D
Kamaelia Motivations
• Key Problems:
 •   RTP was originally concieved for A/V conferencing/telephony

 •   Aspects don’t scale well for large scale unidirectional streaming

 •   Need a platform for designing, implementing and testing new open
     standards to scale in this way.

 •   Scalability and ability to experiment often conflict.

 •   Large scale means highly parallel

 •   Scalable concurrency often has a high barrier to entry

     •   Limits new ideas, collaboration

                                                                 (c) 2005 BBC R&D
Axon
• Kamaelia's Core Concurrency System
• Key aims:
 •   Scalable appoach

 •   Reusable

 •   Simple - easy enough for novice programmer to pick up and
     produce useful systems.

     •   Novices see possibilities, not problems

 •   Safe - it should be possible to write programs without worrying
     about race hazards

     •   Non locking if possible
                                                               (c) 2005 BBC R&D
Scaling Concurrency
• quot;Threads are for people who cant program state
  machines.quot; -- Alan Cox (http://tinyurl.com/a66es)
• Processes/Threads/Build your own
  •   Processes and threads are well known to be not scalable cross
      platform.

  •   Build your own:

      •   Normally means state machines

      •   What about people who quot;cant program state machinesquot; ?
          •   (Not a dig at Alan !)



                                                               (c) 2005 BBC R&D
Scalability : State machines
• Hard to get 100% right - especially for novices
• Debugging someone else’s - twice as hard
• State machine is a piece of sequential processing that
  can release control half way and be restarted retaining
  state
• Twisted - at it’s heart very state machine based.
  •   Provides a very good framework for this and provides lots of
      high quality assistance

  •   Still has this barrier to entry (my personal opinion,YMMV)

                                                                   (c) 2005 BBC R&D
Scalability or ease ?
                                  Do we really have to choose?
• Consider:
  •   A state machine is a piece of sequential processing that can
      release control half way and be restarted

  •   A generator is a piece of sequential processing that can release
      control half way and be restarted

• Twisted also recognises this: twisted.flow
  •   Takes a different approach to composition

• Kamaelia uses generators
  •   Hypothesised this would be easier for novices


                                                               (c) 2005 BBC R&D
Kamaelia vs Twisted?
• NO!
 •   Kamaelia could be integrated into twisted (or vice versa) - we just
     haven't looked at that yet

 •   Twisted is stable, mature and usable for production
     systems

 •   Kamaelia isn't mature or suitable for production systems at
     present

     •   Won’t always be that way, but even when it isn’t we’d rather
         collaborate rather than compete.

 •   Lengthy answer in Kamaelia’s blog


                                                                 (c) 2005 BBC R&D
Concurrency is Easy ?
• Concurrency is hard
  •   ... so why do we let sys admins do it?

• Think unix pipelines:
  •   find -type f | egrep -v '/build/|^./MANIFEST' |while read i ;
      do cp ../Source/$i $i done

• This has 4 logically concurrent units!
  •   Do unix sys admins think of themselves concurrent programmers?

  •   Do you think of it that way?



                                                                     (c) 2005 BBC R&D
Unix Pipelines
• Concurrent sequential processes - linear
• Items don't know what's next in the pipeline
• Simply communicate with local file handles
• Often forgotten “hidden” details:
 • How data passes between processes
 • The system environment
                                                 (c) 2005 BBC R&D
Axon - Key classes
• Components - self pausing sequential objects that
  send data to local interfaces
• Linkages - a facility for joining interfaces, allowing
  system composition
• Scheduler - gives components CPU time
• Postman - The facility for tracking linkages, and
  handling data transferral
• Co-ordinating Assistant/Tracker (cat) - Provides
  environmental facilities akin to a Linda tuple space
                                                    (c) 2005 BBC R&D
Axon Components
• Classes with a generator method called quot;mainquot;
• Augmented by:
 •   List of Inboxes - defaults: inbox, control

 •   List of Outboxes - defaults: outbox, signal

     •   class Echo(component):
           def main(self):
              while 1:
              if self.dataReady(quot;inboxquot;):
                 data = self.recv(quot;inboxquot;)
              self.send(data,quot;outboxquot;)
              yield 1

                                                   (c) 2005 BBC R&D
Axon Scheduler
• Operation
 •   Holds a run queue containing activated components

 •   Calls the generator for each component sequentially

• Component Activation
 •   If the return value is a newComponent object the components
     contained are activated (essentially their main() method is called,
     and the resulting generator stored)

• Component Deactivation
 •   If the return value is false, the component is removed from the run
     queue
                                                                  (c) 2005 BBC R&D
Linkages
• Normally join outboxes to inboxes between
  components
 •   out-out and in-in also allowed between parent and nest
     components

• Linkages can only be create inside a component
 •   Inboxes and outboxes designed for connection to subcomponents
     are considered private and have the naming convention of a
     leading underscore

• Encourages composition and reuse
                                                              (c) 2005 BBC R&D
Linkage Example

•   class SimpleStreamingClient(component):
      def main(self):
         client=TCPClient(quot;127.0.0.1quot;,1500)
         decoder = VorbisDecode()
         player = AOAudioPlaybackAdaptor()
         self.link((client,quot;outboxquot;), (decoder,quot;inboxquot;)
         self.link((decoder,quot;outboxquot;), (player,quot;inboxquot;))

        self.addChildren(decoder, player, client)
        yield newComponent(decoder, player, client)
        while 1:
          self.pause()
          yield 1
                                                           (c) 2005 BBC R&D
Linkage Example 2
def AdHocFileProtocolHandler(filename):
 class klass(Kamaelia.ReadFileAdaptor.ReadFileAdaptor):
 def __init__(self,*argv,**argd):
   self.__super.__init__(filename, readmode=quot;bitratequot;, bitrate=400000)
   return klass

class SimpleStreamingServer(component):
  def main(self):
   server = SimpleServer(protocol=AdHocFileProtocolHandler (quot;foo.oggquot;),
                          port=clientServerTestPort)
   self.addChildren(server)
   yield _Axon.Ipc.newComponent(*(self.children))
   while 1:
     self.pause()
     yield 1
                                                               (c) 2005 BBC R&D
Linkage Example: Re-use
class SimpleMulticastStreamingClient(component):
  def main(self):
   client = Multicast_transceiver(quot;0.0.0.0quot;, 1600, quot;224.168.2.9quot;, 0)
   adapt = detuple(1)
   decoder = VorbisDecode()
   player = AOAudioPlaybackAdaptor()
   self.link((client,quot;outboxquot;), (adapt,quot;inboxquot;)
   self.link((adapt, quot;outboxquot;), (decoder,quot;inboxquot;)
   self.link((decoder,quot;outboxquot;), (player,quot;inboxquot;))

  self.addChildren(decoder, adapt, player, client)
  yield newComponent(decoder, adapt, player, client)
  while 1:
    self.pause()
    yield 1
                                                          (c) 2005 BBC R&D
Co-ordinating Assistant Tracker

• Tracking Services
  •   This allows for the concept of services

  •   A service is a mapping of name to (component, inbox) tuple

  •   Only ever quot;needquot; one 'select' statement in a program for example.
      (want is a different matter!)

  •   The Kamaelia.Internet.Selector component offers a quot;selectorquot;
      service

• Tracking Values
  •   Provides look up and modification of values for keys

  •   Use case: to enable stats collection in servers
                                                               (c) 2005 BBC R&D
Howto: Example Component

• MIME/RFC2822 type objects are common in
  network protocols
  •   Email, web, usenet, etc..

• Essentially serialised key/value pairs - much like a
  dict.
• Create a “MIME Dict” component.
  •   Accepts dict like objects, but translates them to MIME-like
      messages

  •   Accepts MIME-like messages, and converts them to dicts.


                                                                    (c) 2005 BBC R&D
MimeDictComponent
• How it was written
 •   First of all a class that could be a quot;MIME dictquot; was written

 •   Subclasses dict

 •   Always adds a __BODY__ key

 •   Replaces __str__ with something that displays the dict as an
     RFC2822/MIME style message

 •   Adds a staticmethod quot;fromStringquot; as a factory method.

• Written entirely test first without a view
  to being used as a component

                                                                    (c) 2005 BBC R&D
MimeDictComponent 2
• Wanted a component thus:
 •   control - on which we may receive a shutdown message

 •   signal - one which we will send shutdown messages

 •   demarshall - an inbox to which you send strings for turning into
     dicts

 •   marshall - an inbox to which you send objects for turning into
     strings

 •   demarshalled - an outbox which spits out parsed strings as
     dicts

 •   marshalled = an outbox which spits out translated dicts as
     strings
                                                             (c) 2005 BBC R&D
MimeDictComponent 3
• Turned out to be simpler to write a generic marshalling
  component instead, main loop looked like this:
        while 1:
         self.pause()
         if self.dataReady(quot;controlquot;):
           data = self.recv(quot;controlquot;)
           if isinstance(data, Axon.Ipc.producerFinished)
             self.send(Axon.Ipc.producerFinished(), quot;signalquot;)
             return
         if self.dataReady(quot;marshallquot;):
           data = self.recv(quot;marshallquot;)
           self.send(str(data),quot;marshalledquot;)
         if self.dataReady(quot;demarshallquot;):
           data = self.recv(quot;demarshallquot;)
           self.send(self.klass.fromString(data),quot;demarshalledquot;)
         yield 1
                                                                   (c) 2005 BBC R&D
MimeDictComponent 4
• Subclassing approach:
 •   class MimeDictMarshaller(MarshallComponent):
       def __init__(self,*argv,**argd):
          self.__super.__init__(MimeDict, *argv,**argd)

• Class decoration approach:
 •   def MarshallerFactory(klass):
      class newclass(MarshallComponent):
        def __init__(self,*argv,**argd):
         self.__super.__init__(klass, *argv,**argd)
      return newclass

     MimeDictMarshaller=MarshallerFactory(MimeDict)

                                                          (c) 2005 BBC R&D
Summary: New Components

• Longer tutorial based around a multicast
  transceiver on the website.
• Same approach:
  •   Don't worry about concurrency, write single threaded

  •   When code works, then convert to components

  •   Change control methods into inboxes/outboxes




                                                             (c) 2005 BBC R&D
Ease of use?
• Tested on Ciaran Eaton, a pre-university trainee
  •   Happy to let me call him a novice programmer (triple checked)

  •   Previous experience: A-Level computer studies - small amount of
      Visual Basic programming and Access

• 3 Month placement with our group
  •   Started off learning python & axon (2 weeks)

  •   Created a “learning system” based around parsing a Shakespeare
      play:

      •   Performs filtering, character identification, demultiplexing etc

      •   Used pygame for display, stopped short of using pyTTS...
                                                                     (c) 2005 BBC R&D
Ease of use? 2
• Ciaran’s project:
  •   Created a simplistic low bandwidth video streamer

  •   Server has an MPEG video, and takes a frame as JPEG every n
      seconds

  •   This is sent to the client over a framing protocol Ciaran designed
      and implemented

      •   The client then displays the images as they arrive

      •   On a PC this uses pygame

      •   On a series 60 mobile this uses the native image display calls

  •   The idea is this simulates previewing PVR content on a mobile
                                                                   (c) 2005 BBC R&D
Ease of use? 3
• Project was successful, Ciaran achieved the goals
• Ciaran wrote all the components for every part of
  the description.
• Relied on a “SimpleServer” and simple “TCPclient”
  components - but these only provide reliable data
  transfer over the network.
• He’s noted that it was a fun experience
  •   I find it interesting it was not frustrating given his background.



                                                                   (c) 2005 BBC R&D
CSP vs State Machines
• Is this approach inherently worse or better?
• We would suggest neither.
• State machine systems often have intermediate
  buffers (even single variables) for handoff between
  state machines
• This is akin to outboxes and inboxes. If they are
  collapsed into one, as planned, this is equivalent
  •   If we do collapse outboxes into inboxes when we create linkages,
      then the system should be as efficient as frameworks like twisted.

  •   This is currently hypothetical.
                                                               (c) 2005 BBC R&D
Integration with other systems

• Default component provides a default main, which
  calls 3 default callbacks.
• Looks like this:
  •   def main(self):
        result = self.initialiseComponent()
        if not result:
           result = 1
        yield result
        while(result):
           result = self.mainBody()
           if result:
              yield result
        yield self.closeDownComponent()
                                              (c) 2005 BBC R&D
Integration: 2
• Purpose of the 3 callback form is for 2 main
  reasons
  •   For those who find callback forms easier to work with

  •   To allow these methods to be overridden by classes written in:

      •   Pyrex

      •   C

      •   C++

      •   ie optimisation of components



                                                                (c) 2005 BBC R&D
Futures
• C++ Version.
 •   Simple “miniaxon” version including C++ based generators
     working. see: cvs:/Code/CPP/Scratch/miniaxon.cpp

• Python Axon will be optimised
• Syntactic Sugar will be added
• Automated component distribution over clusters
• Kamaelia Component Repository
• More protocols, experimental servers:
 •   RTSP/RTP initially. New protocols to follow!
                                                                (c) 2005 BBC R&D
Finally: Collaboration
• If you’re interested in working with us, please do
    •    If you find the code looks vaguely interesting, please use and give
         us feedback

    •    We’re very open to exploring changes to the system and willing to
         give people CVS commit access in order to try their ideas.

    •    Anyone working with twisted is very welcome to come and
         criticise and suggest new ideas - integration would be very nice!

•       Contacts, project blog:
    •    michaels@rd.bbc.co.uk, kamaelia-list@lists.sourceforge.net

    •    http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi

                                                                      (c) 2005 BBC R&D

Mais conteúdo relacionado

Semelhante a Kamaelia - Networking Using Generators

Bounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise EnvironmentBounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise EnvironmentAdaCore
 
Rapid Application Development with Cocoon
Rapid Application Development with CocoonRapid Application Development with Cocoon
Rapid Application Development with Cocoontcurdt
 
9th docker meetup 2016.07.13
9th docker meetup 2016.07.139th docker meetup 2016.07.13
9th docker meetup 2016.07.13Amrita Prasad
 
Brian Oliver Pimp My Data Grid
Brian Oliver  Pimp My Data GridBrian Oliver  Pimp My Data Grid
Brian Oliver Pimp My Data Griddeimos
 
It's always sunny with OpenJ9
It's always sunny with OpenJ9It's always sunny with OpenJ9
It's always sunny with OpenJ9DanHeidinga
 
Road less traveled - SREcon_Americas 2019
Road less traveled - SREcon_Americas 2019Road less traveled - SREcon_Americas 2019
Road less traveled - SREcon_Americas 2019Eduard Iacoboaia
 
Kamaelia - Fave 2005
Kamaelia - Fave 2005Kamaelia - Fave 2005
Kamaelia - Fave 2005kamaelian
 
PL/SQL Development
PL/SQL DevelopmentPL/SQL Development
PL/SQL DevelopmentThanh Nguyen
 
Road to sbt 1.0 paved with server
Road to sbt 1.0   paved with serverRoad to sbt 1.0   paved with server
Road to sbt 1.0 paved with serverEugene Yokota
 
IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...
IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...
IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...IWMW
 
Road to sbt 1.0: Paved with server (2015 Amsterdam)
Road to sbt 1.0: Paved with server (2015 Amsterdam)Road to sbt 1.0: Paved with server (2015 Amsterdam)
Road to sbt 1.0: Paved with server (2015 Amsterdam)Eugene Yokota
 
Multi Core Playground
Multi Core PlaygroundMulti Core Playground
Multi Core PlaygroundESUG
 
Advanced Techniques for Initiating the DevOps Journey
Advanced Techniques for Initiating the DevOps JourneyAdvanced Techniques for Initiating the DevOps Journey
Advanced Techniques for Initiating the DevOps JourneyCA Technologies
 
GC Tuning Confessions Of A Performance Engineer
GC Tuning Confessions Of A Performance EngineerGC Tuning Confessions Of A Performance Engineer
GC Tuning Confessions Of A Performance EngineerMonica Beckwith
 
Free GitOps Workshop + Intro to Kubernetes & GitOps
Free GitOps Workshop + Intro to Kubernetes & GitOpsFree GitOps Workshop + Intro to Kubernetes & GitOps
Free GitOps Workshop + Intro to Kubernetes & GitOpsWeaveworks
 
AIST Super Green Cloud: lessons learned from the operation and the performanc...
AIST Super Green Cloud: lessons learned from the operation and the performanc...AIST Super Green Cloud: lessons learned from the operation and the performanc...
AIST Super Green Cloud: lessons learned from the operation and the performanc...Ryousei Takano
 
Tackling non-determinism in Hadoop - Testing and debugging distributed system...
Tackling non-determinism in Hadoop - Testing and debugging distributed system...Tackling non-determinism in Hadoop - Testing and debugging distributed system...
Tackling non-determinism in Hadoop - Testing and debugging distributed system...Akihiro Suda
 
Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...
Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...
Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...Anne Nicolas
 

Semelhante a Kamaelia - Networking Using Generators (20)

Bounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise EnvironmentBounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise Environment
 
Introducing CQ 5.1
Introducing CQ 5.1Introducing CQ 5.1
Introducing CQ 5.1
 
Rapid Application Development with Cocoon
Rapid Application Development with CocoonRapid Application Development with Cocoon
Rapid Application Development with Cocoon
 
9th docker meetup 2016.07.13
9th docker meetup 2016.07.139th docker meetup 2016.07.13
9th docker meetup 2016.07.13
 
Brian Oliver Pimp My Data Grid
Brian Oliver  Pimp My Data GridBrian Oliver  Pimp My Data Grid
Brian Oliver Pimp My Data Grid
 
It's always sunny with OpenJ9
It's always sunny with OpenJ9It's always sunny with OpenJ9
It's always sunny with OpenJ9
 
Road less traveled - SREcon_Americas 2019
Road less traveled - SREcon_Americas 2019Road less traveled - SREcon_Americas 2019
Road less traveled - SREcon_Americas 2019
 
Kamaelia - Fave 2005
Kamaelia - Fave 2005Kamaelia - Fave 2005
Kamaelia - Fave 2005
 
PL/SQL Development
PL/SQL DevelopmentPL/SQL Development
PL/SQL Development
 
Road to sbt 1.0 paved with server
Road to sbt 1.0   paved with serverRoad to sbt 1.0   paved with server
Road to sbt 1.0 paved with server
 
IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...
IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...
IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...
 
Road to sbt 1.0: Paved with server (2015 Amsterdam)
Road to sbt 1.0: Paved with server (2015 Amsterdam)Road to sbt 1.0: Paved with server (2015 Amsterdam)
Road to sbt 1.0: Paved with server (2015 Amsterdam)
 
Multi Core Playground
Multi Core PlaygroundMulti Core Playground
Multi Core Playground
 
Advanced Techniques for Initiating the DevOps Journey
Advanced Techniques for Initiating the DevOps JourneyAdvanced Techniques for Initiating the DevOps Journey
Advanced Techniques for Initiating the DevOps Journey
 
GC Tuning Confessions Of A Performance Engineer
GC Tuning Confessions Of A Performance EngineerGC Tuning Confessions Of A Performance Engineer
GC Tuning Confessions Of A Performance Engineer
 
Scaling Django
Scaling DjangoScaling Django
Scaling Django
 
Free GitOps Workshop + Intro to Kubernetes & GitOps
Free GitOps Workshop + Intro to Kubernetes & GitOpsFree GitOps Workshop + Intro to Kubernetes & GitOps
Free GitOps Workshop + Intro to Kubernetes & GitOps
 
AIST Super Green Cloud: lessons learned from the operation and the performanc...
AIST Super Green Cloud: lessons learned from the operation and the performanc...AIST Super Green Cloud: lessons learned from the operation and the performanc...
AIST Super Green Cloud: lessons learned from the operation and the performanc...
 
Tackling non-determinism in Hadoop - Testing and debugging distributed system...
Tackling non-determinism in Hadoop - Testing and debugging distributed system...Tackling non-determinism in Hadoop - Testing and debugging distributed system...
Tackling non-determinism in Hadoop - Testing and debugging distributed system...
 
Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...
Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...
Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...
 

Mais de kamaelian

Kamaelia lightning2010opensource
Kamaelia lightning2010opensourceKamaelia lightning2010opensource
Kamaelia lightning2010opensourcekamaelian
 
Kamaelia Europython Tutorial
Kamaelia Europython TutorialKamaelia Europython Tutorial
Kamaelia Europython Tutorialkamaelian
 
Embracing concurrency for fun utility and simpler code
Embracing concurrency for fun utility and simpler codeEmbracing concurrency for fun utility and simpler code
Embracing concurrency for fun utility and simpler codekamaelian
 
Kamaelia Protocol Walkthrough
Kamaelia Protocol WalkthroughKamaelia Protocol Walkthrough
Kamaelia Protocol Walkthroughkamaelian
 
Sharing Data and Services Safely in Concurrent Systems using Kamaelia
Sharing Data and Services Safely in Concurrent Systems using KamaeliaSharing Data and Services Safely in Concurrent Systems using Kamaelia
Sharing Data and Services Safely in Concurrent Systems using Kamaeliakamaelian
 
Practical concurrent systems made simple using Kamaelia
Practical concurrent systems made simple using KamaeliaPractical concurrent systems made simple using Kamaelia
Practical concurrent systems made simple using Kamaeliakamaelian
 
Sociable Software
Sociable SoftwareSociable Software
Sociable Softwarekamaelian
 
Kamaelia Grey
Kamaelia GreyKamaelia Grey
Kamaelia Greykamaelian
 
Open Source at the BBC: When, Why, Why not & How
Open Source at the BBC: When, Why, Why not & HowOpen Source at the BBC: When, Why, Why not & How
Open Source at the BBC: When, Why, Why not & Howkamaelian
 
Open Source at the BBC
Open Source at the BBCOpen Source at the BBC
Open Source at the BBCkamaelian
 
SWP - A Generic Language Parser
SWP - A Generic Language ParserSWP - A Generic Language Parser
SWP - A Generic Language Parserkamaelian
 
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
Timeshift Everything, Miss Nothing - Mashup your PVR with KamaeliaTimeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaeliakamaelian
 
Scaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, GoalsScaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, Goalskamaelian
 
Kamaelia Internals
Kamaelia InternalsKamaelia Internals
Kamaelia Internalskamaelian
 
Managing Creativity
Managing CreativityManaging Creativity
Managing Creativitykamaelian
 
Building systems with Kamaelia
Building systems with KamaeliaBuilding systems with Kamaelia
Building systems with Kamaeliakamaelian
 
Free software: How does it work?
Free software: How does it work?Free software: How does it work?
Free software: How does it work?kamaelian
 
The Selfish Programmer
The Selfish ProgrammerThe Selfish Programmer
The Selfish Programmerkamaelian
 

Mais de kamaelian (18)

Kamaelia lightning2010opensource
Kamaelia lightning2010opensourceKamaelia lightning2010opensource
Kamaelia lightning2010opensource
 
Kamaelia Europython Tutorial
Kamaelia Europython TutorialKamaelia Europython Tutorial
Kamaelia Europython Tutorial
 
Embracing concurrency for fun utility and simpler code
Embracing concurrency for fun utility and simpler codeEmbracing concurrency for fun utility and simpler code
Embracing concurrency for fun utility and simpler code
 
Kamaelia Protocol Walkthrough
Kamaelia Protocol WalkthroughKamaelia Protocol Walkthrough
Kamaelia Protocol Walkthrough
 
Sharing Data and Services Safely in Concurrent Systems using Kamaelia
Sharing Data and Services Safely in Concurrent Systems using KamaeliaSharing Data and Services Safely in Concurrent Systems using Kamaelia
Sharing Data and Services Safely in Concurrent Systems using Kamaelia
 
Practical concurrent systems made simple using Kamaelia
Practical concurrent systems made simple using KamaeliaPractical concurrent systems made simple using Kamaelia
Practical concurrent systems made simple using Kamaelia
 
Sociable Software
Sociable SoftwareSociable Software
Sociable Software
 
Kamaelia Grey
Kamaelia GreyKamaelia Grey
Kamaelia Grey
 
Open Source at the BBC: When, Why, Why not & How
Open Source at the BBC: When, Why, Why not & HowOpen Source at the BBC: When, Why, Why not & How
Open Source at the BBC: When, Why, Why not & How
 
Open Source at the BBC
Open Source at the BBCOpen Source at the BBC
Open Source at the BBC
 
SWP - A Generic Language Parser
SWP - A Generic Language ParserSWP - A Generic Language Parser
SWP - A Generic Language Parser
 
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
Timeshift Everything, Miss Nothing - Mashup your PVR with KamaeliaTimeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
 
Scaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, GoalsScaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, Goals
 
Kamaelia Internals
Kamaelia InternalsKamaelia Internals
Kamaelia Internals
 
Managing Creativity
Managing CreativityManaging Creativity
Managing Creativity
 
Building systems with Kamaelia
Building systems with KamaeliaBuilding systems with Kamaelia
Building systems with Kamaelia
 
Free software: How does it work?
Free software: How does it work?Free software: How does it work?
Free software: How does it work?
 
The Selfish Programmer
The Selfish ProgrammerThe Selfish Programmer
The Selfish Programmer
 

Último

Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Paola De la Torre
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 

Último (20)

Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 

Kamaelia - Networking Using Generators

  • 1. Kamaelia - Networking Using Generators Michael Sparks BBC Research & Development ACCU Python 2005, Oxford
  • 2. Kamaelia • Project to explore long term systems for large scale media delivery • Forms a concurrency toolkit, focussed mainly on experimenting with network protocols. • 2 key portions: • Axon - Core Component infrastructure, based on communicating generators • Kamaelia - Collection of components that use Axon. • Aim: Scalable, easy & safe concurrent systems (c) 2005 BBC R&D
  • 3. Kamaelia Status • Released as open source: • http://kamaelia.sourceforge.net/ • Axon is at version 1.0.3, and considered feature stable. • Runs on Linux, Windows (variety), Mac OS X • Specialised distribution for Nokia Series 60 mobiles • Kamaelia is at version 0.1.2, and growing • Ability to write TCP & Multicast clients and servers • Variety of simple servers, clients and protocols included (c) 2005 BBC R&D
  • 4. Kamaelia Status • Kamaelia 0.1.2: • Tested on Linux, Windows (variety), Mac OS X • Subset on Nokia Series 60 mobiles • Ease of use hypothesis has been tested with 1 pre- university trainee, looks promising (c) 2005 BBC R&D
  • 5. Kamaelia Motivations • Large Scale Streaming • Several million streams per day • Big events have tens of thousands of concurrent viewers • Want to scale to handling millions of concurrent viewers • Since this could happen. (c) 2005 BBC R&D
  • 6. Kamaelia Motivations • What If 10 years from now... • the BBC opened the entire archive? • Creative Archive is NOT that ambitious! (AFAIK) • the entire UK got broadband? • Instantly hit long tail problems • 20 million homes? • 20 million different things? • Not like 20 million people watching BBC1 ! (c) 2005 BBC R&D
  • 7. Kamaelia Motivations • Key Problems: • RTP was originally concieved for A/V conferencing/telephony • Aspects don’t scale well for large scale unidirectional streaming • Need a platform for designing, implementing and testing new open standards to scale in this way. • Scalability and ability to experiment often conflict. • Large scale means highly parallel • Scalable concurrency often has a high barrier to entry • Limits new ideas, collaboration (c) 2005 BBC R&D
  • 8. Axon • Kamaelia's Core Concurrency System • Key aims: • Scalable appoach • Reusable • Simple - easy enough for novice programmer to pick up and produce useful systems. • Novices see possibilities, not problems • Safe - it should be possible to write programs without worrying about race hazards • Non locking if possible (c) 2005 BBC R&D
  • 9. Scaling Concurrency • quot;Threads are for people who cant program state machines.quot; -- Alan Cox (http://tinyurl.com/a66es) • Processes/Threads/Build your own • Processes and threads are well known to be not scalable cross platform. • Build your own: • Normally means state machines • What about people who quot;cant program state machinesquot; ? • (Not a dig at Alan !) (c) 2005 BBC R&D
  • 10. Scalability : State machines • Hard to get 100% right - especially for novices • Debugging someone else’s - twice as hard • State machine is a piece of sequential processing that can release control half way and be restarted retaining state • Twisted - at it’s heart very state machine based. • Provides a very good framework for this and provides lots of high quality assistance • Still has this barrier to entry (my personal opinion,YMMV) (c) 2005 BBC R&D
  • 11. Scalability or ease ? Do we really have to choose? • Consider: • A state machine is a piece of sequential processing that can release control half way and be restarted • A generator is a piece of sequential processing that can release control half way and be restarted • Twisted also recognises this: twisted.flow • Takes a different approach to composition • Kamaelia uses generators • Hypothesised this would be easier for novices (c) 2005 BBC R&D
  • 12. Kamaelia vs Twisted? • NO! • Kamaelia could be integrated into twisted (or vice versa) - we just haven't looked at that yet • Twisted is stable, mature and usable for production systems • Kamaelia isn't mature or suitable for production systems at present • Won’t always be that way, but even when it isn’t we’d rather collaborate rather than compete. • Lengthy answer in Kamaelia’s blog (c) 2005 BBC R&D
  • 13. Concurrency is Easy ? • Concurrency is hard • ... so why do we let sys admins do it? • Think unix pipelines: • find -type f | egrep -v '/build/|^./MANIFEST' |while read i ; do cp ../Source/$i $i done • This has 4 logically concurrent units! • Do unix sys admins think of themselves concurrent programmers? • Do you think of it that way? (c) 2005 BBC R&D
  • 14. Unix Pipelines • Concurrent sequential processes - linear • Items don't know what's next in the pipeline • Simply communicate with local file handles • Often forgotten “hidden” details: • How data passes between processes • The system environment (c) 2005 BBC R&D
  • 15. Axon - Key classes • Components - self pausing sequential objects that send data to local interfaces • Linkages - a facility for joining interfaces, allowing system composition • Scheduler - gives components CPU time • Postman - The facility for tracking linkages, and handling data transferral • Co-ordinating Assistant/Tracker (cat) - Provides environmental facilities akin to a Linda tuple space (c) 2005 BBC R&D
  • 16. Axon Components • Classes with a generator method called quot;mainquot; • Augmented by: • List of Inboxes - defaults: inbox, control • List of Outboxes - defaults: outbox, signal • class Echo(component): def main(self): while 1: if self.dataReady(quot;inboxquot;): data = self.recv(quot;inboxquot;) self.send(data,quot;outboxquot;) yield 1 (c) 2005 BBC R&D
  • 17. Axon Scheduler • Operation • Holds a run queue containing activated components • Calls the generator for each component sequentially • Component Activation • If the return value is a newComponent object the components contained are activated (essentially their main() method is called, and the resulting generator stored) • Component Deactivation • If the return value is false, the component is removed from the run queue (c) 2005 BBC R&D
  • 18. Linkages • Normally join outboxes to inboxes between components • out-out and in-in also allowed between parent and nest components • Linkages can only be create inside a component • Inboxes and outboxes designed for connection to subcomponents are considered private and have the naming convention of a leading underscore • Encourages composition and reuse (c) 2005 BBC R&D
  • 19. Linkage Example • class SimpleStreamingClient(component): def main(self): client=TCPClient(quot;127.0.0.1quot;,1500) decoder = VorbisDecode() player = AOAudioPlaybackAdaptor() self.link((client,quot;outboxquot;), (decoder,quot;inboxquot;) self.link((decoder,quot;outboxquot;), (player,quot;inboxquot;)) self.addChildren(decoder, player, client) yield newComponent(decoder, player, client) while 1: self.pause() yield 1 (c) 2005 BBC R&D
  • 20. Linkage Example 2 def AdHocFileProtocolHandler(filename): class klass(Kamaelia.ReadFileAdaptor.ReadFileAdaptor): def __init__(self,*argv,**argd): self.__super.__init__(filename, readmode=quot;bitratequot;, bitrate=400000) return klass class SimpleStreamingServer(component): def main(self): server = SimpleServer(protocol=AdHocFileProtocolHandler (quot;foo.oggquot;), port=clientServerTestPort) self.addChildren(server) yield _Axon.Ipc.newComponent(*(self.children)) while 1: self.pause() yield 1 (c) 2005 BBC R&D
  • 21. Linkage Example: Re-use class SimpleMulticastStreamingClient(component): def main(self): client = Multicast_transceiver(quot;0.0.0.0quot;, 1600, quot;224.168.2.9quot;, 0) adapt = detuple(1) decoder = VorbisDecode() player = AOAudioPlaybackAdaptor() self.link((client,quot;outboxquot;), (adapt,quot;inboxquot;) self.link((adapt, quot;outboxquot;), (decoder,quot;inboxquot;) self.link((decoder,quot;outboxquot;), (player,quot;inboxquot;)) self.addChildren(decoder, adapt, player, client) yield newComponent(decoder, adapt, player, client) while 1: self.pause() yield 1 (c) 2005 BBC R&D
  • 22. Co-ordinating Assistant Tracker • Tracking Services • This allows for the concept of services • A service is a mapping of name to (component, inbox) tuple • Only ever quot;needquot; one 'select' statement in a program for example. (want is a different matter!) • The Kamaelia.Internet.Selector component offers a quot;selectorquot; service • Tracking Values • Provides look up and modification of values for keys • Use case: to enable stats collection in servers (c) 2005 BBC R&D
  • 23. Howto: Example Component • MIME/RFC2822 type objects are common in network protocols • Email, web, usenet, etc.. • Essentially serialised key/value pairs - much like a dict. • Create a “MIME Dict” component. • Accepts dict like objects, but translates them to MIME-like messages • Accepts MIME-like messages, and converts them to dicts. (c) 2005 BBC R&D
  • 24. MimeDictComponent • How it was written • First of all a class that could be a quot;MIME dictquot; was written • Subclasses dict • Always adds a __BODY__ key • Replaces __str__ with something that displays the dict as an RFC2822/MIME style message • Adds a staticmethod quot;fromStringquot; as a factory method. • Written entirely test first without a view to being used as a component (c) 2005 BBC R&D
  • 25. MimeDictComponent 2 • Wanted a component thus: • control - on which we may receive a shutdown message • signal - one which we will send shutdown messages • demarshall - an inbox to which you send strings for turning into dicts • marshall - an inbox to which you send objects for turning into strings • demarshalled - an outbox which spits out parsed strings as dicts • marshalled = an outbox which spits out translated dicts as strings (c) 2005 BBC R&D
  • 26. MimeDictComponent 3 • Turned out to be simpler to write a generic marshalling component instead, main loop looked like this: while 1: self.pause() if self.dataReady(quot;controlquot;): data = self.recv(quot;controlquot;) if isinstance(data, Axon.Ipc.producerFinished) self.send(Axon.Ipc.producerFinished(), quot;signalquot;) return if self.dataReady(quot;marshallquot;): data = self.recv(quot;marshallquot;) self.send(str(data),quot;marshalledquot;) if self.dataReady(quot;demarshallquot;): data = self.recv(quot;demarshallquot;) self.send(self.klass.fromString(data),quot;demarshalledquot;) yield 1 (c) 2005 BBC R&D
  • 27. MimeDictComponent 4 • Subclassing approach: • class MimeDictMarshaller(MarshallComponent): def __init__(self,*argv,**argd): self.__super.__init__(MimeDict, *argv,**argd) • Class decoration approach: • def MarshallerFactory(klass): class newclass(MarshallComponent): def __init__(self,*argv,**argd): self.__super.__init__(klass, *argv,**argd) return newclass MimeDictMarshaller=MarshallerFactory(MimeDict) (c) 2005 BBC R&D
  • 28. Summary: New Components • Longer tutorial based around a multicast transceiver on the website. • Same approach: • Don't worry about concurrency, write single threaded • When code works, then convert to components • Change control methods into inboxes/outboxes (c) 2005 BBC R&D
  • 29. Ease of use? • Tested on Ciaran Eaton, a pre-university trainee • Happy to let me call him a novice programmer (triple checked) • Previous experience: A-Level computer studies - small amount of Visual Basic programming and Access • 3 Month placement with our group • Started off learning python & axon (2 weeks) • Created a “learning system” based around parsing a Shakespeare play: • Performs filtering, character identification, demultiplexing etc • Used pygame for display, stopped short of using pyTTS... (c) 2005 BBC R&D
  • 30. Ease of use? 2 • Ciaran’s project: • Created a simplistic low bandwidth video streamer • Server has an MPEG video, and takes a frame as JPEG every n seconds • This is sent to the client over a framing protocol Ciaran designed and implemented • The client then displays the images as they arrive • On a PC this uses pygame • On a series 60 mobile this uses the native image display calls • The idea is this simulates previewing PVR content on a mobile (c) 2005 BBC R&D
  • 31. Ease of use? 3 • Project was successful, Ciaran achieved the goals • Ciaran wrote all the components for every part of the description. • Relied on a “SimpleServer” and simple “TCPclient” components - but these only provide reliable data transfer over the network. • He’s noted that it was a fun experience • I find it interesting it was not frustrating given his background. (c) 2005 BBC R&D
  • 32. CSP vs State Machines • Is this approach inherently worse or better? • We would suggest neither. • State machine systems often have intermediate buffers (even single variables) for handoff between state machines • This is akin to outboxes and inboxes. If they are collapsed into one, as planned, this is equivalent • If we do collapse outboxes into inboxes when we create linkages, then the system should be as efficient as frameworks like twisted. • This is currently hypothetical. (c) 2005 BBC R&D
  • 33. Integration with other systems • Default component provides a default main, which calls 3 default callbacks. • Looks like this: • def main(self): result = self.initialiseComponent() if not result: result = 1 yield result while(result): result = self.mainBody() if result: yield result yield self.closeDownComponent() (c) 2005 BBC R&D
  • 34. Integration: 2 • Purpose of the 3 callback form is for 2 main reasons • For those who find callback forms easier to work with • To allow these methods to be overridden by classes written in: • Pyrex • C • C++ • ie optimisation of components (c) 2005 BBC R&D
  • 35. Futures • C++ Version. • Simple “miniaxon” version including C++ based generators working. see: cvs:/Code/CPP/Scratch/miniaxon.cpp • Python Axon will be optimised • Syntactic Sugar will be added • Automated component distribution over clusters • Kamaelia Component Repository • More protocols, experimental servers: • RTSP/RTP initially. New protocols to follow! (c) 2005 BBC R&D
  • 36. Finally: Collaboration • If you’re interested in working with us, please do • If you find the code looks vaguely interesting, please use and give us feedback • We’re very open to exploring changes to the system and willing to give people CVS commit access in order to try their ideas. • Anyone working with twisted is very welcome to come and criticise and suggest new ideas - integration would be very nice! • Contacts, project blog: • michaels@rd.bbc.co.uk, kamaelia-list@lists.sourceforge.net • http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi (c) 2005 BBC R&D