April 15, 2003: “Leveraging Notational Systems: Developing applications that stand the test of time”. Presented at the Collaborative Expedition workshop, sponsored by the U.S. General Services Administration as part of the federal Chief Information Officers’ Council.

1. Cover Page
Developing Applications that
Stand the Test of Time
Author: Jeffrey G. Long (jefflong@aol.com)
Date: April 15, 2003
Forum: Talk presented at the Collaborative Expedition workshop, sponsored by
the U.S. General Services Administration as part of the federal Chief Information
Officers’ Council.
Contents
Pages 1‐31: Slides (but no text) for presentation
License
This work is licensed under the Creative Commons Attribution‐NonCommercial
3.0 Unported License. To view a copy of this license, visit
http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative
Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
Uploaded June 27, 2011

2. Leveraging Notational Systems:
Developing applications th t stand th t t of ti
D l i li ti that t d the test f time
Jeff Long
April 15, 2003
jefflong@aol.com

3. Proposed Agenda
 1: Current software development
paradigm and consequences thereof
(15 minutes)
 2: Proposed alternative paradigm and
consequences thereof (15 minutes)
 3: How to implement rules as data (15
minutes)
i t )
4/15/03 Copyright 2003 Jeff Long 2

4. Fundamental Hypothesis of
Notational Engineering
Many problems in government science the
government, science,
arts, and engineering exist solely because of
the way we currently represent them. These
problems present an apparent “complexity
barrier” and cannot be resolved with more
computing power or more money. Their
resolution requires a new abstraction which
becomes th b i of a notational revolution
b the basis f t ti l l ti
and solves a whole class of previously-
intractable problems
problems.
4/15/03 Copyright 2003 Jeff Long 3

5. A New Notational System Often
Requires a Change of Paradigm
 A way of looking at a subject
 An example, pattern, archetype, or
y
model
 A set of unconscious assumptions we
have about a subject
4/15/03 Copyright 2003 Jeff Long 4

6. Part 1 The Current
P t 1: Th C t
Software Development
Paradigm

7. Current Paradigm
 Computer applications are defined in
terms of algorithms and data
 Algorithms are the rules which are used
to
t manipulate the d t d t and rules
i l t th data; data d l
are distinct
 The model for this is the abacus
4/15/03 Copyright 2003 Jeff Long 6

8.  Algorithms are implemented as software
 Everything else, such as inventory
y g , y
quantities or employee names, is
implemented as data
p
 Thus most rules (e g work processes,
(e.g. processes
decision trees) are implemented as
software
4/15/03 Copyright 2003 Jeff Long 7

9. Immediate Consequences
 Every application requires a lot of
software, often >1 million lines
– No one can really understand or manage
this level of complexity
– There are numerous bugs in any software
system
– Special tools and techniques are required
to create and/or manage large bodies of
code; these add to overall costs
4/15/03 Copyright 2003 Jeff Long 8

10.  There is a lot of “maintenance” of the
software
– The software has to be changed as bugs
are found and eliminated
– Th software has to be changed as th
The ft h t b h d the
rules change
– Rules change as users become more
sophisticated
– Rules change due to external p
g pressures as
well (e.g. new languages or architectures;
government requirements; competition)
4/15/03 Copyright 2003 Jeff Long 9

11.  The changes to software themselves
introduce new bugs
– Extensive testing is required in order to
have any comfort level
– Extensive documentation is necessary, but
often skipped; maintenance p g
pp ; programmers
don’t know why system was so designed
4/15/03 Copyright 2003 Jeff Long 10

12. Additional Assumptions
 Users know what they want and can
communicate that to programmers
– Programmers who have experience in an
industry can understand user requirements
– User requirements can be documented in a
form that is less complex than the actual
working system
g y
 Software can be designed the same
way as any other complex technology
4/15/03 Copyright 2003 Jeff Long 11

13. Results to Date
 Increasing application backlog
 2/6 development projects cancelled;
additional 3/6 considered failures
 True importance of very good designers
and programmers i
d increasingly
i l
recognized
 Bugs have greater and greater
consequences; viruses don’t help
4/15/03 Copyright 2003 Jeff Long 12

14.  High turnover of Chief Information
Officers (average tenure: 18 months)
( g )
 Increasing use of packages
– This typically forces changes in work
processes, or else requires expensive
customization (>2x package p
( p g price))
 Increasing use of offshore outsourcing
(>35% by 2005?)
4/15/03 Copyright 2003 Jeff Long 13

15. Lessons Learned?
 Subject experts cannot communicate all
requirements to programmers
– their expertise took many years to acquire
– their own understanding will evolve
 Subject experts must see working prototypes,
not paper representations
 Subject experts must be able to directly and
continuously update rules as needed
 “Corporate” knowledge must be externalized
4/15/03 Copyright 2003 Jeff Long 14

16. Part 2: The Ultra-Structure
Development P di
D l t Paradigm

17. Where and How Should Rules be
Represented in a System?
 Remove 99% of all rules from the
software
 Represent them in a standard If/Then
form (multiple Ifs, multiple Thens)
 Represent them as records of data
within a very small set of tables
 Distinction between rules and data
largely disappears!
4/15/03 Copyright 2003 Jeff Long 16

18. Immediate Consequences
 Software size is reduced by 2+ orders
2
of magnitude
– simpler to create, manage, understand,
p , g , ,
test, document, and teach
– remaining software has no knowledge of
the
th world; it provides b i control l i
ld id basic t l logic
that knows what tables to check in what
order, how to resolve conflicts, etc.
 Software development team is very
small and manageable
4/15/03 Copyright 2003 Jeff Long 17

19.  “Corporate” knowledge is externalized and is
in a form anyone can see and understand
y
 Subject experts can enter, change, and
otherwise manage rules (corporate
knowledge) di
k l d ) directly, without going to
l ih i
programmers for assistance
 Knowledge is actionable not only by subject
experts (e.g. as an encyclopedia) but also by
the computer, for reasoning, decision
support, etc.
4/15/03 Copyright 2003 Jeff Long 18

20.  Programmers do not need to know or
understand all rules, just enough to
determine the classes of rules and the
proper animation procedures
 Serious prototyping becomes feasible;
p
communications with users improves
 Testing & QA can be far more rigorous
 Documentation can be more complete
4/15/03 Copyright 2003 Jeff Long 19

21. Results to Date
 First commercial system built with this
approach grew from $13M to $100M from
1986-2002, with $1K/month software
expense; became largest independent
wholesaler in their industry partly due to low
operating costs
 Other prototypes in language, biology, legal,
games, and artificial life seem to confirm
basic claims and expectations
4/15/03 Copyright 2003 Jeff Long 20

22. Part 3: How to Implement
Rules
R l as D t (A P i
Data Primer)
)

23. Analysis of Rules
 Statement of rules and device for executing
them can be different; need not be software
for both
 Rules can be reformulated into a canonical
form of “If a and b and c... then consider x
and y and z”
 One informal, “molecular” rule (e.g. “three
strikes and you’re out” i b
ik d ’ ” in baseball) may b
b ll) be
translated as many “atomic” rules
4/15/03 Copyright 2003 Jeff Long 22

24.  “If” values specify conditions under
which each rule is examined; these are
called “factors”
 “Then consider” values specify addit
Then consider addit-
ional criteria that must be considered
before deciding what to do; these are
called “considerations”
4/15/03 Copyright 2003 Jeff Long 23

25.  Factors become primary keys in third-
normal form
normal-form RDBMS tables
 Alternate keys can be specified if useful
TTens of thousands of rules can be
f th d f l b
grouped into 10-50 classes based on
their
th i syntax and semantics
t d ti
 These classes can be managed easily
by the tools of a RDBMS
4/15/03 Copyright 2003 Jeff Long 24

26.  Control logic (“animation procedures”)
reads relevant rules, including rules
, g
about selecting rules, and carries out
specified actions
p
 Once established, control logic should
not need to change; all changes are
made by subject experts who directly
update the data (rules) of the system
4/15/03 Copyright 2003 Jeff Long 25

27.  Separation of rules into categories
defines tables in the RDBMS
 There is no practical limit on number of
rows in a table (i.e. > 1,000 rules is no
problem)
 Referential integrity and field edits
ensure that rules maintain integrity
 Queries, report writers and forms ease
the review of rules by authorized users
f
4/15/03 Copyright 2003 Jeff Long 26

28.  Design can proceed by iterative prototype;
small prototypes can easily evolve to
necessary level of complexity
 Basic design process is to:
– define what exists (existential rules)
– define relations between these (network &
authorization rules)
– d fi processes ( t
define (protocol & meta-protocol rules)
l t t l l )
 The application essentially becomes an
Expert System using a RDBMS
4/15/03 Copyright 2003 Jeff Long 27

29. The Ruleform Hypothesis
Complex system structures are created by not-
necessarily complex processes; and these
processes are created by the animation of
competency rules. Competency rules can be
rules
grouped into a small number of classes whose
form is prescribed by "ruleforms". While the
competency rules of a system change over time time,
the ruleforms remain constant. A well-designed
collection of ruleforms can anticipate all logically
possible competency rules that might apply to the
system, and constitutes the deep structure of the
system.
4/15/03 Copyright 2003 Jeff Long 28

30. The Theory Offers a Different Way to
Look at Complex Systems and Processes
observables surface structure
generates
rules middle structure
constrains
groups of rules
f l deep structure
4/15/03 Copyright 2003 Jeff Long 29

31. The CoRE Hypothesis
We can create “Competency Rule Engines”, or CoREs,
Competency Engines CoREs
consisting of <50 ruleforms, that are sufficient to
represent all rules found among systems sharing
broad family resemblances, e g all corporations
resemblances e.g. corporations.
Their definitive deep structure will be permanent,
unchanging, and robust for all members of the family,
whose differences in manifest structures and
behaviors will be represented entirely as differences
in competency rules. The animation procedures for
each engine will be relatively simple compared to
current applications, requiring less than 100,000 lines
of code in a third generation language.
4/15/03 Copyright 2003 Jeff Long 30

32. References
 Long, J., and Denning, D., “Ultra-Structure: A design theory for
complex systems and processes.” In Communications of the
ACM (January 1995)
 Long, J., “A new notation for representing business and other
rules.” In Long, J. (guest editor), Semiotica Special Issue on
Notational Engineering, Volume 125-1/3 (1999)
 Long, J., “How could the notation be the limitation?” In Long, J.
(guest editor), Semiotica Special Issue on Notational
Engineering, Volume 125-1/3 (1999)
 Long, J., "Automated Identification of Sensitive Information in
Documents Using Ultra-Structure". In Proceedings of the 20th
Annual ASEM Conference, American Society for Engineering
Management (October 1999)
4/15/03 Copyright 2003 Jeff Long 31