SlideShare a Scribd company logo
1 of 29
Download to read offline
Cover Page 

 




         Towards a New 
      Paradigm to Resolve 
       the Software Crisis 
Author: Jeffrey G. Long (jefflong@aol.com 

Date: February 5, 20033 

Forum: Talk presented at the University of North Carolina, Chapel Hill.

 

                                 Contents 
Page 1: Proposal 

Pages 2‐28: Slides (but no text) for presentation 

 


                                  License 
This work is licensed under the Creative Commons Attribution‐NonCommercial 
3.0 Unported License. To view a copy of this license, visit 
http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative 
Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA. 




                                Uploaded June 27, 2011 
Towards A New Paradigm to Resolve the Software Crisis

Various kinds of metrics have been used to indicate the scope and magnitude of the
software crisis. Some of these pertain to the high percentage of uninstalled systems;
others to the increasing length of the applications backlog in most shops; others to the
high lifetime cost of developing and maintaining software, whether custom-made or
packaged; and others to the opportunity costs.

All of these metrics indicate that the software business continues to be in a state of crisis
which persists in spite of technical advances such as database management systems;
structured code; improved and more formal development practices; the use of packaged
software, code generators, computer-aided software engineering tools; component-
based, object-oriented, multi-tier software architectures; and colossal budgets. The
problem becomes worse as society expects more from computers and becomes more
dependent upon them. User and organizational frustrations with this problem have
caused CIOs to be replaced regularly, and, increasingly, have resulted in a strong trend
to simply outsource all MIS computer and software operations -- and blame.

This talk presents a new paradigm that has been developed and tested since 1985. It is
based on an analysis of the nature of rules, their underlying common structure, and how
rules are and ought to be represented. The approach is called “Ultra-Structure”.

Rules are typically hard-coded into software applications, and the maintenance of these
rules as they change is the primary cause of subsequent software changes. One
unfortunate consequence of this standard approach is that subject experts must
somehow communicate the rules they wish to see automated to programmers who
typically are not experts in the subject matter of the application; much is lost in the
translation. Another unfortunate consequence is that the software becomes large, often
millions of lines of code, such that no one involved in the project can really comprehend
or manage it. There have been of course numerous initiatives directed at solving these
problems, but the solutions have been ineffectual and the problems they address are
actually secondary and symptomatic rather than primary.

The basic premise of this new approach is that we can resolve all of the major difficulties
with software development, and generate significant new kinds of benefits for users, by
removing most rules and all knowledge of the world from software and instead
representing them the same way we represent data, i.e. as tables in a relational
database.

This approach combines key features of the normally disparate areas of management
information systems, expert systems, and simulations, borrowing the strengths of each
and largely eliminating the known problems of each. The approach can be applied to
any kind of rule-based system, whether the application is intended to support business,
scientific, or other application areas. It has been applied thus far to business and natural
language applications, and an early prototype of its applications to biology has been
developed by Dr. Gidding’s laboratory.
Towards A New
Paradigm to Resolve the
    Software Crisis
    S f      Ci i


         Jeff Long
        Feb 5, 2003
        jefflong@aol.com
P
Proposed A
       d Agenda
             d
 Current paradigm and consequences
 C      t     di      d
 thereof (15 minutes)
 Proposed alternative paradigm and
 consequences thereof (15 minutes)
 Technical analysis of rules and their
 implementation ( minutes)
   p             (15          )
 General Q&A (15 minutes)


 2/5/03       Copyright 2003 Jeff Long   2
P di
Paradigm
 A way of looking at a subject
        f l ki     t     bj t
 An example, pattern, archetype, or
 model

 A set of unconscious assumptions we
 have about a subject



 2/5/03       Copyright 2003 Jeff Long   3
C
Current S ft
      t Software P di
                 Paradigm
 All applications are d fi d i t
         li ti        defined in terms of
                                        f
 algorithms and data



 Algorithms are the rules which are used
 to manipulate data
 The model for this is the abacus


 2/5/03       Copyright 2003 Jeff Long   4
Algorithms are implemented as software
Everything else, such as inventory
quantities or employee names is
                       names,
implemented as data

Thus most rules (e.g. work processes,
decision trees) are implemented as
software


2/5/03      Copyright 2003 Jeff Long   5
I
Immediate C
    di t Consequences
 Every application requires a l t of
 E         li ti         i     lot f
 software, often >1 million lines
     No one can really understand or manage
     this level of complexity
     There are numerous bugs in any software
     Th                    b   i       ft
     system
     The ft
     Th software has to be changed as b
                     h t b h       d   bugs
     are found and eliminated


 2/5/03         Copyright 2003 Jeff Long   6
There is a l t of “maintenance” of the
Th    i    lot f “ i t        ” f th
software
    The software has to be changed as the
    rules change
    Rules h
    R l change as users b  become more
    sophisticated
    Rules h
    R l change d t external pressures as
                  due to t     l
    well


2/5/03        Copyright 2003 Jeff Long   7
The h
Th changes t software th
             to ft    themselves
                            l
introduce new bugs
    Extensive testing is required in order to
    have any comfort level
    Extensive documentation i necessary, b t
    E t    i d           t ti is              but
    often skipped; maintenance programmers
    don t
    don’t know why system was so designed



2/5/03           Copyright 2003 Jeff Long   8
Additi
Additional A
         l Assumptions
                 ti
 Users know what they want and can
 communicate that to programmers
     Programmers who have experience in an
     industry can understand user requirements
     User requirements can be documented in a
     form that is less complex than the actual
     working system
 Software can be designed the same
 way as any other complex technology

 2/5/03         Copyright 2003 Jeff Long   9
R
Results t D t
    lt to Date
 Increasing application backlog (maybe
 use open source software)
 2/6 development projects cancelled;
 additional 3/6 considered failures
 True value of very good designers and
 programmers increasingly recognized
 Bugs have greater and greater
 consequences; viruses don’t help


 2/5/03      Copyright 2003 Jeff Long   10
High turnover of Chi f I f
Hi h t          f Chief Information
                               ti
Officers (average tenure: 18 months)
Increasing use of packages
    This typically forces changes in work
    processes, or else requires expensive
    customization (>2x package price)
Increasing use of outsourcing upon
surrender (35-45% by 2005)

2/5/03          Copyright 2003 Jeff Long   11
L
Lessons L     d?
        Learned?
 Subject experts cannot communicate all
 requirements to programmers
     their expertise took many y
             p               y years to acquire
                                          q
     their own understanding will evolve
 Subject experts must see working prototypes,
 not paper representations
   t               t ti
 Subject experts must be able to directly and
 continuously update rules as needed
 “Corporate” knowledge must be externalized


 2/5/03            Copyright 2003 Jeff Long       12
A Alt
An Alternative P di
          ti Paradigm
 Remove most rules from software


 Represent rules in canonical form, as
 data in a small set of tables
                      f
 Externalize “corporate” knowledge in
 relational k
   l i    l knowledgebase
                 l d b
 Let subject experts manage data

 2/5/03       Copyright 2003 Jeff Long   13
I
Immediate C
    di t Consequences
 Software size is reduced by 2+ orders of
 magnitude
     simpler to create, manage, understand, test,
        p              ,      g ,             ,     ,
     document, and teach
     remaining software has no knowledge of the
     world; it provides basic control logic that knows
     what tables to check in what order, how to resolve
     conflicts, etc.
 Software development team is very small and
 S ft     d   l     tt     i          ll d
 manageable


 2/5/03            Copyright 2003 Jeff Long      14
“Corporate” knowledge is externalized and is
 Corporate
in a form anyone can see and understand
    Knowledge is actionable by the computer:
    reasoning, error checking, d i i support
          i           h ki     decision
Subject experts can enter, change, and
otherwise manage rules directly without
                        directly,
programmer assistance
    Data no longer merely defines certain details of
    the overall system; it defines the identity of the
    system



2/5/03             Copyright 2003 Jeff Long       15
Programmers do not need to know or
understand all rules, just enough to
determine the classes of rules and the
proper animation procedures
Serious prototyping becomes feasible;
communications with users improves
Testing & QA can be far more rigorous
Documentation can be more complete


2/5/03       Copyright 2003 Jeff Long   16
R
Results t D t
    lt to Date
  Commercial system: 7x growth with
  $1K/month software expense


Other prototypes in language, biology, legal,
        p     yp         g g ,         gy, g ,
   games, and artificial life seem to confirm
   basic claims and expectations
Closer to SEI Vision: “Th right software,
Cl              Vi i   “The i h      f
   delivered defect free, on time and on cost,
   every time ”
          time.

  2/5/03          Copyright 2003 Jeff Long   17
A l i of R l
Analysis f Rules
 Statement of rules and device for
 St t      t f l        dd i f
 executing them can be different; need
 not be software for both
   tb     ft     f b th
 Rules can be reformulated into a
 canonical form of “If a and b and c...
 then consider x and y and z”



 2/5/03       Copyright 2003 Jeff Long   18
“If” values specify conditions under
        l          if    diti      d
which each rule is examined; these are
called “f t ” in Ult St t
    ll d “factors” i Ultra-Structure Theory
                                     Th
(UST)
“Then consider” values specify
additional criteria that must be
considered; these are called
“considerations” in UST

2/5/03        Copyright 2003 Jeff Long   19
Factors become primary keys in thi d
F t     b          i      k     i third-
normal form RDBMS
Alternate keys can be specified if useful
Control logic (
          g (“animation p  procedures”))
reads relevant rules, including rules
about selecting rules, and carries out
               g     ,
specified actions


2/5/03       Copyright 2003 Jeff Long   20
One i f
O informal, “molecular” rule (
             l “ l     l ” l (e.g. ththree
strikes and you’re out in baseball) may be
translated as many atomic rules
Basic process is to
    define what exists (existential rules)
    define relations between these (network &
    authorization rules)
    define processes (protocol & meta-protocol rules)




2/5/03            Copyright 2003 Jeff Long     21
Tens of thousands of rules can be
grouped into a small number of classes
based on their syntax and semantics
These classes can be managed easily
by the tools of a RDBMS
Design can proceed by iterative
prototype; small prototypes can easily
evolve to necessary level of complexity


2/5/03       Copyright 2003 Jeff Long   22
I l     t ti
Implementation
 Separation of rules i t classes d fi
 S       ti    f l into l           defines
 tables in the RDBMS; there is no practical
 limit on number of rows in a table
 Referential integrity and field edits ensure
 that rules maintain integrity
 Queries, report writers and forms make
 access to rules easy for authorized users
                      y




 2/5/03         Copyright 2003 Jeff Long   23
The R l f
Th Ruleform H
            Hypothesis
                th i
 Complex system structures are created by not-
     necessarily complex processes; and these
     p
     processes are created by the animation of
                              y
     competency rules. Competency rules can be
     grouped into a small number of classes whose
     form is prescribed by "ruleforms". While the
             p            y
     competency rules of a system change over time,
     the ruleforms remain constant. A well-designed
     collection of ruleforms can anticipate all logically
                                       p          g     y
     possible competency rules that might apply to the
     system, and constitutes the deep structure of the
     system.
      y

 2/5/03             Copyright 2003 Jeff Long       24
The C RE Hypothesis
Th CoRE H    th i
We can create “Competency Rule Engines”, or CoREs,
  consisting of <50 ruleforms, that are sufficient to
  represent all rules found among systems sharing
    p                             g y               g
  broad family resemblances, e.g. all corporations.
  Their definitive deep structure will be permanent,
  unchanging, and robust for all members of the family,
          g g,                                         y,
  whose differences in manifest structures and
  behaviors will be represented entirely as differences
  in competency rules. The animation p
         p       y                       procedures for
  each engine will be relatively simple compared to
  current applications, requiring less than 100,000 lines
  of code in a third generation language.
                     g              g g

  2/5/03            Copyright 2003 Jeff Long      25
Ultra-Structure is an Example of
Notational Engineering
 Many problems i science, government, th
 M         bl     in i                 t the
 arts, and engineering are caused by the way
 we represent them
 These problems cannot be resolved with
 more computing power or more money
 They require a new abstraction which can be
 the basis of a notational revolution




 2/5/03        Copyright 2003 Jeff Long   26
R f
References
 Long, J.,      Denning, D., Ultra-Structure:
 Long J and Denning D “Ultra-Structure: A design theory for
 complex systems and processes.” In Communications of the
 ACM (January 1995)
 Long, J., “A new notation for representing business and other
     g, ,                          p         g
 rules.” In Long, J. (guest editor), Semiotica Special Issue on
 Notational Engineering, Volume 125-1/3 (1999)
 Long, J., “How could the notation be the limitation?” In Long, J.
 (guest editor), S i ti S
 (     t dit ) Semiotica Special Ii l Issue on N t ti
                                               Notational
                                                        l
 Engineering, Volume 125-1/3 (1999)
 Long, J., "Automated Identification of Sensitive Information in
 Documents Using Ultra-Structure". In Proceedings of the 20th
                      Ultra-Structure
 Annual ASEM Conference, American Society for Engineering
 Management (October 1999)



 2/5/03                Copyright 2003 Jeff Long            27

More Related Content

Similar to Towards a new paradigm to resolve the software crisis

Case study of rules as relational data
Case study of rules as relational dataCase study of rules as relational data
Case study of rules as relational data
Jeff Long
 
How to save on software maintenance costs
How to save on software maintenance costsHow to save on software maintenance costs
How to save on software maintenance costs
FrancisJansen
 
Towards preventing software from becoming legacy a road map
Towards preventing software from becoming legacy a road mapTowards preventing software from becoming legacy a road map
Towards preventing software from becoming legacy a road map
IAEME Publication
 

Similar to Towards a new paradigm to resolve the software crisis (20)

Applying a new software development paradigm to biology
Applying a new software development paradigm to biologyApplying a new software development paradigm to biology
Applying a new software development paradigm to biology
 
Case study of rules as relational data
Case study of rules as relational dataCase study of rules as relational data
Case study of rules as relational data
 
Case study of rules as relational data
Case study of rules as relational dataCase study of rules as relational data
Case study of rules as relational data
 
How to save on software maintenance costs
How to save on software maintenance costsHow to save on software maintenance costs
How to save on software maintenance costs
 
Bt0081 software engineering
Bt0081 software engineeringBt0081 software engineering
Bt0081 software engineering
 
Instant message
Instant  messageInstant  message
Instant message
 
Elements of legacy program complexity
Elements of legacy program complexityElements of legacy program complexity
Elements of legacy program complexity
 
Dit yvol3iss3
Dit yvol3iss3Dit yvol3iss3
Dit yvol3iss3
 
Four ways to represent computer executable rules
Four ways to represent computer executable rulesFour ways to represent computer executable rules
Four ways to represent computer executable rules
 
Mingle box - Online Job seeking System
Mingle box - Online Job seeking SystemMingle box - Online Job seeking System
Mingle box - Online Job seeking System
 
Software copy
Software   copySoftware   copy
Software copy
 
Sd Revision
Sd RevisionSd Revision
Sd Revision
 
Vol 1 issue 2 june 2015
Vol 1 issue 2 june 2015Vol 1 issue 2 june 2015
Vol 1 issue 2 june 2015
 
Public voice
Public voicePublic voice
Public voice
 
Using Computer-Aided Tools in Information Systems Development
Using Computer-Aided Tools in Information Systems DevelopmentUsing Computer-Aided Tools in Information Systems Development
Using Computer-Aided Tools in Information Systems Development
 
Gov civilworkshop
Gov civilworkshopGov civilworkshop
Gov civilworkshop
 
Defect effort prediction models in software
Defect effort prediction models in softwareDefect effort prediction models in software
Defect effort prediction models in software
 
Towards preventing software from becoming legacy a road map
Towards preventing software from becoming legacy a road mapTowards preventing software from becoming legacy a road map
Towards preventing software from becoming legacy a road map
 
Slcm sharbani bhattacharya
Slcm sharbani bhattacharyaSlcm sharbani bhattacharya
Slcm sharbani bhattacharya
 
EDM Systems In Depth Review
EDM Systems In Depth ReviewEDM Systems In Depth Review
EDM Systems In Depth Review
 

More from Jeff Long

Notational engineering and the search for new intellectual primitives
Notational engineering and the search for new intellectual primitivesNotational engineering and the search for new intellectual primitives
Notational engineering and the search for new intellectual primitives
Jeff Long
 

More from Jeff Long (20)

Notational systems and the abstract built environment
Notational systems and the abstract built environmentNotational systems and the abstract built environment
Notational systems and the abstract built environment
 
Managing and benefiting from multi million rule systems
Managing  and benefiting from multi million rule systemsManaging  and benefiting from multi million rule systems
Managing and benefiting from multi million rule systems
 
Ten lessons from a study of ten notational systems
Ten lessons from a study of ten notational systemsTen lessons from a study of ten notational systems
Ten lessons from a study of ten notational systems
 
Notational systems and cognitive evolution
Notational systems and cognitive evolutionNotational systems and cognitive evolution
Notational systems and cognitive evolution
 
Notational systems and abstractions
Notational systems and abstractionsNotational systems and abstractions
Notational systems and abstractions
 
Notational engineering and the search for new intellectual primitives
Notational engineering and the search for new intellectual primitivesNotational engineering and the search for new intellectual primitives
Notational engineering and the search for new intellectual primitives
 
Understanding complex systems
Understanding complex systemsUnderstanding complex systems
Understanding complex systems
 
The hunt for new abstractions
The hunt for new abstractionsThe hunt for new abstractions
The hunt for new abstractions
 
Why we dont understand complex systems
Why we dont understand complex systemsWhy we dont understand complex systems
Why we dont understand complex systems
 
Issues in the study of abstractions
Issues in the study of abstractionsIssues in the study of abstractions
Issues in the study of abstractions
 
Automated identification of sensitive information
Automated identification of sensitive informationAutomated identification of sensitive information
Automated identification of sensitive information
 
Mathematics rules and scientific representations
Mathematics rules and scientific representationsMathematics rules and scientific representations
Mathematics rules and scientific representations
 
Notational engineering
Notational engineeringNotational engineering
Notational engineering
 
The evolution of abstractions
The evolution of abstractionsThe evolution of abstractions
The evolution of abstractions
 
A metaphsical system that includes numbers rules and bricks
A metaphsical system that includes numbers rules and bricksA metaphsical system that includes numbers rules and bricks
A metaphsical system that includes numbers rules and bricks
 
The nature of notational engineering
The nature of notational engineeringThe nature of notational engineering
The nature of notational engineering
 
The co evolution of symbol systems and society
The co evolution of symbol systems and societyThe co evolution of symbol systems and society
The co evolution of symbol systems and society
 
New ways to represent complex systems and processes
New ways to represent complex systems and processesNew ways to represent complex systems and processes
New ways to represent complex systems and processes
 
Representing emergence with rules
Representing emergence with rulesRepresenting emergence with rules
Representing emergence with rules
 
The evolution of symbol systems and society
The evolution of symbol systems and societyThe evolution of symbol systems and society
The evolution of symbol systems and society
 

Recently uploaded

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Earley Information Science
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Recently uploaded (20)

Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 

Towards a new paradigm to resolve the software crisis

  • 1. Cover Page    Towards a New  Paradigm to Resolve  the Software Crisis  Author: Jeffrey G. Long (jefflong@aol.com  Date: February 5, 20033  Forum: Talk presented at the University of North Carolina, Chapel Hill.   Contents  Page 1: Proposal  Pages 2‐28: Slides (but no text) for presentation    License  This work is licensed under the Creative Commons Attribution‐NonCommercial  3.0 Unported License. To view a copy of this license, visit  http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative  Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.  Uploaded June 27, 2011 
  • 2. Towards A New Paradigm to Resolve the Software Crisis Various kinds of metrics have been used to indicate the scope and magnitude of the software crisis. Some of these pertain to the high percentage of uninstalled systems; others to the increasing length of the applications backlog in most shops; others to the high lifetime cost of developing and maintaining software, whether custom-made or packaged; and others to the opportunity costs. All of these metrics indicate that the software business continues to be in a state of crisis which persists in spite of technical advances such as database management systems; structured code; improved and more formal development practices; the use of packaged software, code generators, computer-aided software engineering tools; component- based, object-oriented, multi-tier software architectures; and colossal budgets. The problem becomes worse as society expects more from computers and becomes more dependent upon them. User and organizational frustrations with this problem have caused CIOs to be replaced regularly, and, increasingly, have resulted in a strong trend to simply outsource all MIS computer and software operations -- and blame. This talk presents a new paradigm that has been developed and tested since 1985. It is based on an analysis of the nature of rules, their underlying common structure, and how rules are and ought to be represented. The approach is called “Ultra-Structure”. Rules are typically hard-coded into software applications, and the maintenance of these rules as they change is the primary cause of subsequent software changes. One unfortunate consequence of this standard approach is that subject experts must somehow communicate the rules they wish to see automated to programmers who typically are not experts in the subject matter of the application; much is lost in the translation. Another unfortunate consequence is that the software becomes large, often millions of lines of code, such that no one involved in the project can really comprehend or manage it. There have been of course numerous initiatives directed at solving these problems, but the solutions have been ineffectual and the problems they address are actually secondary and symptomatic rather than primary. The basic premise of this new approach is that we can resolve all of the major difficulties with software development, and generate significant new kinds of benefits for users, by removing most rules and all knowledge of the world from software and instead representing them the same way we represent data, i.e. as tables in a relational database. This approach combines key features of the normally disparate areas of management information systems, expert systems, and simulations, borrowing the strengths of each and largely eliminating the known problems of each. The approach can be applied to any kind of rule-based system, whether the application is intended to support business, scientific, or other application areas. It has been applied thus far to business and natural language applications, and an early prototype of its applications to biology has been developed by Dr. Gidding’s laboratory.
  • 3. Towards A New Paradigm to Resolve the Software Crisis S f Ci i Jeff Long Feb 5, 2003 jefflong@aol.com
  • 4. P Proposed A d Agenda d Current paradigm and consequences C t di d thereof (15 minutes) Proposed alternative paradigm and consequences thereof (15 minutes) Technical analysis of rules and their implementation ( minutes) p (15 ) General Q&A (15 minutes) 2/5/03 Copyright 2003 Jeff Long 2
  • 5. P di Paradigm A way of looking at a subject f l ki t bj t An example, pattern, archetype, or model A set of unconscious assumptions we have about a subject 2/5/03 Copyright 2003 Jeff Long 3
  • 6. C Current S ft t Software P di Paradigm All applications are d fi d i t li ti defined in terms of f algorithms and data Algorithms are the rules which are used to manipulate data The model for this is the abacus 2/5/03 Copyright 2003 Jeff Long 4
  • 7. Algorithms are implemented as software Everything else, such as inventory quantities or employee names is names, implemented as data Thus most rules (e.g. work processes, decision trees) are implemented as software 2/5/03 Copyright 2003 Jeff Long 5
  • 8. I Immediate C di t Consequences Every application requires a l t of E li ti i lot f software, often >1 million lines No one can really understand or manage this level of complexity There are numerous bugs in any software Th b i ft system The ft Th software has to be changed as b h t b h d bugs are found and eliminated 2/5/03 Copyright 2003 Jeff Long 6
  • 9. There is a l t of “maintenance” of the Th i lot f “ i t ” f th software The software has to be changed as the rules change Rules h R l change as users b become more sophisticated Rules h R l change d t external pressures as due to t l well 2/5/03 Copyright 2003 Jeff Long 7
  • 10. The h Th changes t software th to ft themselves l introduce new bugs Extensive testing is required in order to have any comfort level Extensive documentation i necessary, b t E t i d t ti is but often skipped; maintenance programmers don t don’t know why system was so designed 2/5/03 Copyright 2003 Jeff Long 8
  • 11. Additi Additional A l Assumptions ti Users know what they want and can communicate that to programmers Programmers who have experience in an industry can understand user requirements User requirements can be documented in a form that is less complex than the actual working system Software can be designed the same way as any other complex technology 2/5/03 Copyright 2003 Jeff Long 9
  • 12. R Results t D t lt to Date Increasing application backlog (maybe use open source software) 2/6 development projects cancelled; additional 3/6 considered failures True value of very good designers and programmers increasingly recognized Bugs have greater and greater consequences; viruses don’t help 2/5/03 Copyright 2003 Jeff Long 10
  • 13. High turnover of Chi f I f Hi h t f Chief Information ti Officers (average tenure: 18 months) Increasing use of packages This typically forces changes in work processes, or else requires expensive customization (>2x package price) Increasing use of outsourcing upon surrender (35-45% by 2005) 2/5/03 Copyright 2003 Jeff Long 11
  • 14. L Lessons L d? Learned? Subject experts cannot communicate all requirements to programmers their expertise took many y p y years to acquire q their own understanding will evolve Subject experts must see working prototypes, not paper representations t t ti Subject experts must be able to directly and continuously update rules as needed “Corporate” knowledge must be externalized 2/5/03 Copyright 2003 Jeff Long 12
  • 15. A Alt An Alternative P di ti Paradigm Remove most rules from software Represent rules in canonical form, as data in a small set of tables f Externalize “corporate” knowledge in relational k l i l knowledgebase l d b Let subject experts manage data 2/5/03 Copyright 2003 Jeff Long 13
  • 16. I Immediate C di t Consequences Software size is reduced by 2+ orders of magnitude simpler to create, manage, understand, test, p , g , , , document, and teach remaining software has no knowledge of the world; it provides basic control logic that knows what tables to check in what order, how to resolve conflicts, etc. Software development team is very small and S ft d l tt i ll d manageable 2/5/03 Copyright 2003 Jeff Long 14
  • 17. “Corporate” knowledge is externalized and is Corporate in a form anyone can see and understand Knowledge is actionable by the computer: reasoning, error checking, d i i support i h ki decision Subject experts can enter, change, and otherwise manage rules directly without directly, programmer assistance Data no longer merely defines certain details of the overall system; it defines the identity of the system 2/5/03 Copyright 2003 Jeff Long 15
  • 18. Programmers do not need to know or understand all rules, just enough to determine the classes of rules and the proper animation procedures Serious prototyping becomes feasible; communications with users improves Testing & QA can be far more rigorous Documentation can be more complete 2/5/03 Copyright 2003 Jeff Long 16
  • 19. R Results t D t lt to Date Commercial system: 7x growth with $1K/month software expense Other prototypes in language, biology, legal, p yp g g , gy, g , games, and artificial life seem to confirm basic claims and expectations Closer to SEI Vision: “Th right software, Cl Vi i “The i h f delivered defect free, on time and on cost, every time ” time. 2/5/03 Copyright 2003 Jeff Long 17
  • 20. A l i of R l Analysis f Rules Statement of rules and device for St t t f l dd i f executing them can be different; need not be software for both tb ft f b th Rules can be reformulated into a canonical form of “If a and b and c... then consider x and y and z” 2/5/03 Copyright 2003 Jeff Long 18
  • 21. “If” values specify conditions under l if diti d which each rule is examined; these are called “f t ” in Ult St t ll d “factors” i Ultra-Structure Theory Th (UST) “Then consider” values specify additional criteria that must be considered; these are called “considerations” in UST 2/5/03 Copyright 2003 Jeff Long 19
  • 22. Factors become primary keys in thi d F t b i k i third- normal form RDBMS Alternate keys can be specified if useful Control logic ( g (“animation p procedures”)) reads relevant rules, including rules about selecting rules, and carries out g , specified actions 2/5/03 Copyright 2003 Jeff Long 20
  • 23. One i f O informal, “molecular” rule ( l “ l l ” l (e.g. ththree strikes and you’re out in baseball) may be translated as many atomic rules Basic process is to define what exists (existential rules) define relations between these (network & authorization rules) define processes (protocol & meta-protocol rules) 2/5/03 Copyright 2003 Jeff Long 21
  • 24. Tens of thousands of rules can be grouped into a small number of classes based on their syntax and semantics These classes can be managed easily by the tools of a RDBMS Design can proceed by iterative prototype; small prototypes can easily evolve to necessary level of complexity 2/5/03 Copyright 2003 Jeff Long 22
  • 25. I l t ti Implementation Separation of rules i t classes d fi S ti f l into l defines tables in the RDBMS; there is no practical limit on number of rows in a table Referential integrity and field edits ensure that rules maintain integrity Queries, report writers and forms make access to rules easy for authorized users y 2/5/03 Copyright 2003 Jeff Long 23
  • 26. The R l f Th Ruleform H Hypothesis th i Complex system structures are created by not- necessarily complex processes; and these p processes are created by the animation of y competency rules. Competency rules can be grouped into a small number of classes whose form is prescribed by "ruleforms". While the p y competency rules of a system change over time, the ruleforms remain constant. A well-designed collection of ruleforms can anticipate all logically p g y possible competency rules that might apply to the system, and constitutes the deep structure of the system. y 2/5/03 Copyright 2003 Jeff Long 24
  • 27. The C RE Hypothesis Th CoRE H th i We can create “Competency Rule Engines”, or CoREs, consisting of <50 ruleforms, that are sufficient to represent all rules found among systems sharing p g y g broad family resemblances, e.g. all corporations. Their definitive deep structure will be permanent, unchanging, and robust for all members of the family, g g, y, whose differences in manifest structures and behaviors will be represented entirely as differences in competency rules. The animation p p y procedures for each engine will be relatively simple compared to current applications, requiring less than 100,000 lines of code in a third generation language. g g g 2/5/03 Copyright 2003 Jeff Long 25
  • 28. Ultra-Structure is an Example of Notational Engineering Many problems i science, government, th M bl in i t the arts, and engineering are caused by the way we represent them These problems cannot be resolved with more computing power or more money They require a new abstraction which can be the basis of a notational revolution 2/5/03 Copyright 2003 Jeff Long 26
  • 29. R f References Long, J., Denning, D., Ultra-Structure: Long J and Denning D “Ultra-Structure: A design theory for complex systems and processes.” In Communications of the ACM (January 1995) Long, J., “A new notation for representing business and other g, , p g rules.” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999) Long, J., “How could the notation be the limitation?” In Long, J. (guest editor), S i ti S ( t dit ) Semiotica Special Ii l Issue on N t ti Notational l Engineering, Volume 125-1/3 (1999) Long, J., "Automated Identification of Sensitive Information in Documents Using Ultra-Structure". In Proceedings of the 20th Ultra-Structure Annual ASEM Conference, American Society for Engineering Management (October 1999) 2/5/03 Copyright 2003 Jeff Long 27