Abstract
The general problem at hand is that of supervised learning. This is to say, can computers
learn by example? Specifically, this problem entails learning from data that may not
have a pre-specified form, rendering it difficult to solve through classical statistical
methods. In addition, many non-classical methods, such as feed-forward neural nets and
decision trees, encounter difficulty in learning from data that does not follow an
[obvious] pattern. In this project, this problem is attacked with a recurrent neural
network. This is a simplified model inspired by the human brain, consisting of models of
neurons connected to each other in an arbitrary fashion. While the flexibility of this
architecture makes general-purpose algorithms for analyzing it difficult to develop, it is
hoped that this flexible architecture will also allow recurrent neural net to solve more
complex learning problems than traditional approaches.

Acknowledgement
Iowa State’s Adaptive Computing Laboratory (ACL) will provide much of the
framework for the recurrent neural net. In particular, much of the knowledge of the
technical details of neural net operation is owed to Dr. Eric Bartlett.

Definition of Terms
ACL – Iowa State’s Adaptive Computing Laboratory

Artificial Neural Net – A simple model of learning consisting of two or more models of
neurons connected to each other, as a graph.

Feed-Forward Net – A neural net architecture in which the neurons of the network are
connected in a multi-partite structure, with the neurons divided into layers. The layers of
neurons, besides the last layer containing the output neuron, are known as hidden layers.

Network Model - The way the neurons of the network are connected (the architecture).

Neuron Model – A simple model of a biological neuron. It takes multiple inputs,
multiplies each of them by a separate weight, sums the products together, and outputs a
transfer function of this sum. The transfer function determines the neuron model.

Recurrent Net – A neural network architecture in which every neuron may be connected
to every other neuron. The output neuron is chosen arbitrarily.

Toolkit – The application developed by the ACL consisting of a collection of neural
network data modeling programs.

Introduction
•General background - The machine learning problem is one of the oldest, and still
one of the most daunting, in computer science. In general, can a computer, given
examples of input data and output data, learn to predict future output given only the
input? The applications of a solution to this general problem are nearly without limit.
Given recorded weather patterns from previous days, can a computer predict the
weather for the next day? Given pictures of a person from different angles, and with
different facial expressions, can a computer pick that person out of a picture of a
crowd? Given enough time to play Gary Kasparov in chess, could a computer learn
to improve its performance sufficiently to beat him, without any previous knowledge
of chess strategy? Traditionally, the learning problem has been analyzed through
classical statistical methods. These methods, however, are limited in their usefulness.
Consequently, researchers have turned to other approaches. One approach, neural
networks, is inspired by the human brain’s ability to learn seemingly complex
problems, such as facial recognition, with comparatively limited data. The Adaptive
Computing Laboratory’s (ACL) Neural Toolkit currently contains a neural network
architecture known as a feed-forward neural net. The goal of this project is to
implement a recurrent neural network for use in the Toolkit. Simplistically, the main
difference between a feed-forward net and a recurrent net is complexity. Feed-
forward nets are simpler and easier to analyze, but the complexity of problems they
can solve is limited. Recurrent nets mimic more closely the design of the brain and
may allow computers to solve more challenging problems, yet are more difficult to
analyze and have more inefficient learning algorithms.
•Technical problem - The recurrent neural net will be implemented in software. An
algorithm will need to be chosen to train the neural net. An algorithm will also need
to be chosen to determine the output of the neural net. These algorithms must then be
implemented in C++. Once these objectives have been accomplished, the network
will be integrated into the existing Toolkit framework using a common user interface.
•Operating environment - The operating environment is the Toolkit. The network
will depend on the Toolkit to format input and output data and to display results. The
Toolkit runs on the Microsoft Windows 95/98/2000/NT 4.0 operating system.
•Intended user(s) and use(s) - The primary user of the Toolkit is the ACL. Other
users might include clients of the ACL. Uses include learning problems in which
other learning techniques have been shown to be ineffective. Experiments will need
to be done to determine for what uses a recurrent neural net is more effective than
other methods.
•Assumptions and limitations
•Assumptions:
•Users of this module will already know how to use the Toolkit.
•There is a plausible method to do this and we will figure out how to do it.
•There is some practical use in doing this.
•Limitations:
•We have five months to work on the project.

Design Requirements
•Design objectives
• Develop a neuron model - This is the transfer function of the neurons.
• Develop a network model - This dictates the interconnection of neurons.
• Develop training algorithm - This algorithm will adjust the weights of the
neurons to minimize the cost function.
•Functional requirements
• Be callable from a system call. This allows the Toolkit to run it.
• Implement I/O interfaces - Formats of input and output files should match
those of the Toolkit.
• Should terminate on a definite stopping condition.
• Should report progress to the user so that the user may decide intelligently
whether to stop it before the stopping condition is reached.
•Design constraints -
• Uses only Windows system calls
• Works only with normalized data
• Runs with the minimum specifications of the Toolkit
•Measurable milestones
• Have a preliminary neuron model and network model
• Complete testing on network model
• Have a preliminary training algorithm
• Complete testing of training algorithm on various data sets
• Incorporate into the toolkit

End-Product Description
We have implemented a recurrent neural network module that integrates with the existing
ACL Neural Network Toolkit. It analyzes normalized training sets to infer dependencies
between the input and output of the training set. It then uses these dependencies to predict
output given arbitrary inputs.

Approach and Design
•Technical approaches - One possible approach is the Rapid Prototype Model of
software development. In this model, we design a prototype recurrent network and
analyze its performance. The design goes through many iterations of design and
analysis before moving to final implementation. Another possible approach is the
Waterfall Model. In this model we break the design into five sections: requirements
analysis, general design, detailed design, coding, and testing. The project proceeds
linearly through the five phases. The following criteria will be used in deciding
between these two: flexible design schedule, ability to provide measurable results in a
timely manner, minimization of overhead.
•Technical design - The following items must be defined:
•Input format – This is the format of the data to input into the system.
•Output format – This is the format of the data to output from the system.
•Data structures – These will hold information about the neural net.
•Evaluation algorithm – This will calculate the output from the neural net.
•Cost functions – These will calculate the error of the model over the data.
•Training algorithm – This will develop the model from the data.
•Testing description - The effectiveness of the project results will be measured by the
performance of the recurrent neural net. This performance will be measured using
common cost functions, and these results will be compared to the performance of
traditional methods. Computational efficiency will not be a significant factor in
measuring performance, for this reason: The recurrent neural net is not expected to
beat the performance of methods that are already in place. The purpose of the project
is to lay a framework from which future efficiency- and performance-enhancing
modifications to the recurrent neural net may easily be added.
•Risks and risk management - Some of the risks we may potentially encounter are
inability of our chosen development model to deliver a working recurrent neural
network on time, loss of a team member, failure of our developed product to meet our
expectations for performance, and an inability of our product to reach a solution to a
test problem. Proper documentation will ensure that future ACL researchers will be
able to continue the work easily and hopefully find a solution.

Financial Budget
Table 1 Financial budget
Item Original Estimated Cost
Poster $50.00
Total estimated cost $50.00

Table 2 Personnel effort budget
Personnel Original Estimated Effort
David Doty 159 hours
Michael Kuehl 154 hours
Corey Folkerts 157 hours
Andrew Brown 163 hours
Total estimated effort 615 hours

Project Schedule
Feb 11, '01 Feb 18, '01 Feb 25, '01 Mar 4, '01 Mar 11, '01 Mar 18, '01 Mar 25, '01 Apr 1, '01 Apr 8, '01 Apr 15, '01
ID Task Name TWT F SSMTWT F SSMTWT F S SMTWT F S SMTWT F S SMTWT F S SMTWT F SS MTWT F S S MTWT F S SMTW
1 Project Poster

2 Design Preliminary Network Structure

3 Test Preliminary Network Structure

4 Decide on Network Structure

5 Design Preliminary Training Algorithm

6 Test Preliminary Training Algorithm

7 Decide on Training Algorithm

8 Design Report

9 Research Toolkit Implementation

Figure 1 – Gantt Chart of Timeline for Project Plan

Project Team Information:
David Doty
4541 Steinbeck, #2 Ames, IA, 50014
515-292-8261
ddoty@iastate.edu
CprE

Corey Folkerts
Friley 3349 Noble, Ames, IA, 50012
515-572-5519
gumby145@iastate.edu
CprE

Michael Kuehl
4541 Steinbeck, #2 Ames, IA, 50014
515-292-8261
mkuehl@iastate.edu
CprE

Andrew Brown
610 7th St., Ames, IA, 50010
515-233-3465
ajbrown@iastate.edu
CprE

Summary
Because recurrent neural networks are able to solve problems of such a type or
complexity that other methods have proved insufficient or infeasible this project will
offer a necessary addition to the diversity of the ACL's Neural Toolkit. By clearly
defining the goals of this project and scheduling such that we continuously make progress
towards those goals via the milestones defined it is the belief that this project will come
to fruition.

References
Eric B Bartlett, Associate Professor
Department of Electrical and Computer Engineering
Iowa State University

Machine Learning, Tom Mitchell, McGraw-Hill, 1997

Table of Contents

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (9)

Semelhante a Table of Contents

Semelhante a Table of Contents (20)

Mais de butest

Mais de butest (20)

Table of Contents