Semi-automatic Picture Book Generation based on Story Model and Agent-based Simulation

International Journal of Modern Research in Engineering & Management (IJMREM)
||Volume|| 1 ||Issue|| 11 || 42-52 || December 2018 || ISSN: 2581-4540
www.ijmrem.com IJMREM Page 42
 Semi-automatic Picture Book Generation based on Story Model
and Agent-based Simulation
1,
Kiyohito Fukuda, 2,
Naoki Mori, 3,
Keinosuke Matsumoto
1,2,3,
Osaka Prefecture University
-----------------------------------------------------------------ABSTRACT-----------------------------------------------------
In the fields of artificial intelligence, automatic narrative generation has attracted considerable interest. Lots of
studies on narrative generation have been reported such as novel, comic, and picture book. However, most of
those reuse original stories, pictures, and sentences. As a result, there is a problem that generated narratives
infringe on copyright. In this paper, we focus on the picture book as the narrative because picture book is a mix of
images and language. As a first step of automatic story generation without representation mediums, we propose a
novel semi-automatic picture book generation method based on story model and agent-based simulation. The
computational experiments are carried out to confirm the effectiveness of the proposed method.
KEYWORDS: agent-based simulation, narrative engineering, picture book generation, story model
---------------------------------------------------------------------------------------------------------------------------------------
Date of Submission: Date, 6 December 2018 Date of Publication: Date 13 December 2018
--------------------------------------------------------------------------------------------------------------------------------------
I. INTRODUCTION
Recently, automatic narrative generation by the computer has attracted considerable interest as a challenging
problem in the fields of artificial intelligence and natural language processing. Narrative [1] is a creation based on
emotions of human and is comprised of a story and a representation medium. When we represent the story by
language, narrative is called novels; and it is called comics when we represent the story in images mainly. In this
paper, we focus on the picture book which is one of the simplest comics, as narrative. With a view toward realizing
effective automatic narrative generation, lots of studies on automatic novel generation based on a case-based
reasoning (CBR) [2][3][4] and problem-solving process [5], and automatic comic generation based on picture
model to express pictures and images [6][7] have been reported. However, most of those studies have a critical
problem that they reuse original stories and sentences directly. As a result, there is a problem that generated
narratives infringe on copyright. If we can generate coherent and unexpected narratives without reusing original
stories directly, we can solve this problem. Therefore, we have proposed the story model in order to model stories
from log data of computational simulation. In this paper, we propose a novel semi-automatic picture book
generation method based on story model and agent-based simulation (ABS). Here, ABS is a simulation to
investigate actions and interactions between autonomous agents [8][9]. A new picture book is generated by
representing a story model which is generated by the simulation log of the ABS.
II. STORY MODEL
We generate picture books based on our story model. The most important part of that is called a scene. There are
lots of different scenes in the story model. Those differences among scenes are represented as objects that appear in
the scene. Each scene has actions and emotions of the object, time, and environment. The story is generated by
selecting the next scenes from the all scenes set and transiting scenes from the start scene to the end scene. The
ability of modeling branched stories is one of the important features of our story model.
The scene has the following informations that are necessary for story generation in hierarchical structure.
 The scene such as scene name, place and time.

Semi-automatic Picture Book Generation based…
 The objects that appear in the scene such as name, age and emotion.
 The relations between the objects such as action, identification and causal relation.
Fig. 1 shows the scene structure of story model.
III. PROPOSED METHOD
In this paper, we propose a semi-automatic picture book generation method based on story model and ABS. The
proposed method proceeds as follows. Each step is explained in detail in 3.1 - 3.4.
Step 1: Settings of the ABS
The environment and parameters of the ABS are set in order to generate log data. Several parameters of the ABS
are determined by the user.
Step 2: Log data generation via ABS
Two types of log data, namely status logs and action logs are generated via the ABS.
Step 3: Story model generation
Story model is generated from action logs and status logs on the basis of a specific character's log data.
Step 4: Picture book generation
Picture book is generated based on the story model by using languages and images as representing mediums.
Steps 1-3 of the proposed method can be performed automatically. However, step 4 requires human's help in the
current system. Thus, the former are referred to as the “computer part” and the latter is referred to as the “user part”.
3.1. Setting of the ABS (Step 1): Fig. 2 shows the flowchart of the ABS algorithm. The ABS algorithm is
implemented for the proposed method. Here we represent a real value according to the normal distribution
and the uniform distribution of as and .
The details of the ABS are explained as follows:
Parameter settings There are two types of parameters in the ABS, namely system parameters and user parameters.
Figure 1. scene structure of story model

Figure 2. flowchart of the ABS
System parameters:
, : The field size of the ABS.
, : The maximum and minimum value of the friendship value.
, : The maximum and minimum value of the familiarity value.
, : The maximum and minimum value of emotion.
, : The maximum and minimum value of personality.
, : The maximum and minimum value of health condition.
: Variable coefficient.
: Variation range coefficient.
: The current number of turns.
: The number of character agents.
: The number of item agents.
: The number of scenes.
: Action number.
User parameters:
: Each agent's name.
: Each character agent's age.
: Each character agent's sex.
Space construction In the ABS, the agent moving space is constructed by a field, which is two-dimensional
torus-shaped spaces. All agents and scenes are set on random positions in a certain range of the field. Each of
agents and scenes can not be set on the same position.
Time management In the ABS, time is managed by turn t and action number . The turn denotes a system
parameter that manages the time course in the environment and is updated after all agents performed all actions.

The action number denotes a system parameter that manages the order of actions performed in the same turn and
is updated each time some agent performs an action.
Scenes and agent’s construction First, character agents and item agents are constructed.
A character agent is an agent that represents the main character in the stories generated by the proposed method and
has the following specific internal states:
 attributes: age , sex , personalities related to interaction and movement , emotion , and health
condition , where is a natural number, is a binary of man or woman, and are real number of
, is a real number of , is a real number of ;
 friendship values toward other agent and familiarity values toward other agent
where is a real number of and is a real number of ;
 the list of its own item agents.
An item agent is an agent that represents the item that is appeared in the stories and has the following specific
internal state:
 information on character agent that is its owner.
Then, both character and item agents have the following internal state:
 attributes: names , role in the story , where is “hero”, “enemy”, “ally”, “food”, or “weapon”;
 influence value on other agent's movement , where is a real positive value and different for each character
agent and item agent;
 position in the field .
Second, scenes are constructed. A scene is a special position in the field. That is important for the proposed
method to generate story model and has the following parameters: role in the story and influence value on other
agent's moving , where is “event” or “goal” and is a real positive number.
Agent selection An agent is randomly selected from agents that do not perform an action in this turn. If there are
character agents in them, agent is randomly selected from them. The selected agent is defined as .
Movement An agent always performs an action of movement. It moves to the Neumann neighborhood in the
direction of the value according to influence values of other agents and scenes in turn , represented by
, Euclidean distance between them, and its own personality. After performed the
action of movement, the action number is updated to .
Interaction If an agent is a character agent and it meets some conditions after movement, it performs an action
of interaction with other agent. The algorithm of interaction is given below.
1. If a character agent is set at the same position of other character agents in the field, a character agent is
randomly selected form them and is defined as . The friendship value and familiarity value of toward in
turn , represented by and , and the emotion of in turn , represented by are updated as follows:
(1)
(2)
(3)
(4)
(5)
Furthermore, if the role of is “hero” and it has an item agent that has the role corresponding to the 's role, its
owner is changed from to . In addition, and are updated as follows:
(6)
(7)
2. If is set at the same position of item agents that is not belonged to any character agents in the field, an item
agent is randomly selected from them and is defined as . 's owner is changed to and the emotion of in
turn , represented by is updated as follows:

(8)
3. If the health condition of in turn , represented by is below threshold and have item agents whose role is
“food”, an item agent is randomly selected from them and is defined as ℓ. loses ℓ and the health condition of
in turn , represented by is updated as follows:
(9)
After that, the health condition of ℓ, represented by and position in the field are reset.
4. After performed the action of interaction, the action number is updated to .
Update and end determination of the ABS The friendship values, health conditions, and influence values of all
agents and all scenes are updated. Here, A set of character agents is defined as and , , a set of all
agents is defined as and , and a set of all agents and all scenes is defined as and . Update
equations are shown below:
(10)
(11)
(12)
(13)
The objective of this update is representing that the importance of character's emotions, items and scenes in the
story is changed by the time course. In addition, turn t is updated to t = t + 1.
Finally, if a character agent whose role is “hero” is set at the same position of a scene whose role is “goal”, the ABS
is halted. If not, it returns to the agent selection.
3.2. Log Data Generation (Step 2): Two types of log data, namely action logs and status logs are generated via
the ABS implemented by 3.1. The details of these log types are explained as follows:
Action log: Action log information is saved if each agent performs a movement or an interaction in the ABS. The
data of action log is {turn, main agent, target agent, position of main agent, action type, action number, the
change amount of each parameter caused by the action, item that is exchanged between agents by the action}.
Status log: The status logs are saved at the start of each turn and at the end of the last turn in the ABS. These
comprise the turn and all parameters of all agents and scenes.
3.3 Story Model Generation (Step 3): The story model is generated from action logs and status logs. The
algorithm of story model generation is as follows:
1. A character agent whose role is “hero” is defined as . In addition, a set of scenes whose role is “event” is
defined as . The status log of the first turn is defined as .
2. Start scene is generated using .
3. A set of the status log of the last turn when is set at the same position of scene is defined as .
4. The status log of minimum turn in is defined as and this minimum turn is defined as .
5. A set of action log of maximum turn in action log that have action types corresponding to is defined as
.
6. The action log of minimum turn in is defined as and this minimum turn is defined as .
7. A scene is generated using , , and the status log of turn .

Table 1. Correspondence of scene elements to the log data
Scene elements Log data
SID No data
Scene name Scene and action type
Object Agent
Place Scene
Time No data
Relation Agent and scene
Type Agent’s type
OID Agent’s Id
Name Agent’s name
Age Agent’s age
Sex Agent’s sex
Personality Agent’s personality
Emotion Agent’s emotion
Posture Scene
Look Agent’s emotion and health condition
Condition Agent’s health condition
Position Scene and agent’s role
Action Action type
Action number Action number
Causal Relation No data
Result Agent’s parameters
Sentiment Friendship value and familiarity value
Identification Agent’s role
8. is updated to . If , it returns to 6.
10. The end scene is generated using each agent's status logs of the last turn. Finally, the set of scenes generated in
2, 7 and 10 is defined as the story model.
The start scene is only one type. All story models have the same start scene. However, there are four types of end
scene, namely simple-end, happy-end, bad-end, and unexpected-end. The end scene is selected from these
prepared scenes in accordance with character agent's parameters such as friendship value, familiarity value, and
position in the field.
Each scene is generated by selecting their elements from choices. The selection of them is in accordance with data
of action logs and status logs corresponding to them. However, there are some elements that do not correspond to
the log data. Table 1 shows the correspondence of scene elements to the log data.
3.4. Picture Book Generation (Step 4): In this paper, a page of the picture book is generated from a scene of the
story model. It is constructed a picture and sentences. The algorithm of picture book generation is as follows:
1. The number of scenes of the story model generated by 3.3 is defined as and the -th scene in the story
model is defined as . Here, denotes the start scene and denotes the end scene. Set
.
2. Suitable picture parts and sentence templates are selected by referring to ’s elements from the prepared
image and sentence template sets.
3. A picture is generated by arranging selected picture parts by human according to . Sentences are generated
by combining selected sentence templates and specific elements such as name or emotion of . -th page of
picture book is generated by combining the picture and sentences.
5. The picture book that is pages is generated by the proposed method.

Table 2. Experimental conditions
(80, 80)
3
2
4
[-100.0, 100.0]
[-100.0, 100.0]
[-1.0, 1.0]
[0.0, 1.0]
[0.0, 1.0]
5
10
Table 3. Correspondence of OID to object attributes
OID Name Role Sex Age
1 Little Red Riding Hood “Hero” Woman 10
2 Wolf “Enemy” Man 8
3 Hunter “Ally” Man 30
4 meat “Food” null null
5 gun “Weapon” null null
IV. COMPUTER EXPERIMENTS
To confirm the effectiveness of the proposed method, we carried out user experiment. In the proposed method, we
have to prepare the outline of the generated story in advance. In this experiment, the outline of the story generated
by the proposed method was prepared manually referring to “Little Red Riding Hood”. It was defined as follows:
Simple-end: The hero arrives her grandmother's house without encountering the enemy.
Happy-end: The hero and the ally exterminate the enemy, and they arrive hero's grandmother's house and eat
dinner with her.
Bad-end: The hero is eaten by the enemy in the way to her grandmother's house.
Unexpected-end: The enemy apologizes to the hero and becomes good friend with her, and they arrive her
grandmother's house and eat dinner with her.
In the proposed method, we implement the GUI system in order for users to set the user parameters. The proposed
method can generate over 20000 kinds of picture books.
4.1. Experimental Method: We asked annotators to generate 5 picture books by changing the simulation seed in
the proposed method. Table 2 and Table 3 show the experimental conditions and the correspondence of object
index called OID to object attributes. Experiments are carried out by 10 annotators. They are university and
graduate students. 5 of them are men and the rest are women. After generating picture books, we asked annotators
to evaluate them with a 4-point rating of 4 (good) – 1 (bad) from 6 viewpoints: story, unexpectedness, differences
from the original story, quality, preference, and variety. We confirmed that annotators knew the story of “Little Red
Riding Hood” before this experiment.
4.2. Experimental Results: Fig. 3 shows the example of picture book generated by user experiment. We call this
“ ”. Tables 4 - 6 show examples of the story model of . Fig. 3 and Tables 4 - 6 are generated in
Japanese and translated into English. Tables 4 - 6 correspond to Fig. 3. For example, the columns whose objects are
O-2 or R-2 in table 5 and table 6 correspond to the second page of picture book in Fig. 3. We can understand that
the story of Fig. 4 is that Little Red Riding Hood is eaten by the wolf in the way to her grandmother's house. The
backgrounds, characters or items are defined by the story model in Fig. 3. The look, personality, emotion and

Table 4. Example of ’s story model (Scene)
SID Scene name Objects Place Time Relation
1 Start O-1 Start Morning R-1
2 Acquisition of meat O-2 In the forest Noon R-2
3 Encounter with Wolf O-3 In the forest Noon R-3
4 Encounter with Hunter O-4 In the forest Noon R-4
5 Attack on Wolf O-5 In the forest Noon R-5
6 Bad-end O-6 In the forest Night R-6
Table 5. Example of ’s story model (Object)
Objects O-1 O-2 O-2 O-3 ……
OID 1 1 4 1 ……
Type Character Character Item Character ……
Name Little Red Riding Hood Little Red Riding Hood Meat Little Red Riding Hood ……
Age 10 10 null 10 ……
Sex Woman Woman null Woman ……
Personality Curious Curious null Curious ……
Emotion Normal Normal null Normal ……
Posture Standing Standing null Standing ……
Look Normal Normal null Normal ……
Condition Good Good Good Good ……
Position Right Left Right Left ……
Table 6. Example of ’s story model (Relation)
RID R-1 R-2 R-2 R-2 R-2 ……
OID-OID 1-1 1-1 1-4 4-1 4-4 ……
Action null null Pick up -Pick up null ……
Action number null null 1 1 null ……
Causal relation null null Chain Chain null ……
Result null null Becomes owner -Becomes owner null ……
Sentiment null null null null null ……
Identification Myself Myself “Hero” “Food Myself ……
Table 7. Annotators’ evaluation for generated picture books
Viewpoints Mean Standard deviation
Story 3.00 0.96
Unexpectedness 2.98 0.93
Differences from original story 3.48 0.67
Quality 2.66 1.01
Preference 2.50 1.04
Variety 2.90 0.94
position of the objects are also defined by the story model. Therefore, these show that the proposed method can
generate coherent picture books.
Next, Table 7 shows the evaluations of annotators. In Table 7, the means of evaluations about story and differences
from the original story are high as 3.00 and 2.98. Considering the median of evaluations is 2.50, we can understand
that the proposed method can generate different stories from the original story. On the other hand, the mean of
evaluations about quality are not high as 2.66 and the standard deviation is over 1.00. In addition, the mean of
evaluations about variety is high as 2.90. These results suggest that the proposed method generate stories that are
too unexpected to represent as picture book because the ABS depends on its own randomness.

Figure 3. generated picture book

V. CONCLUSIONS
In this study, we proposed a novel semi-automatic picture book generation method based on story model and ABS
as a generation that does not depend on representation mediums. The computational experiments are carried out to
confirm the effectiveness of the proposed method. The results of the computational experiments are listed below.
 The proposed method can generate appropriate picture books.
 The proposed method can generate coherent, difference stories from the original story.
In future work, we plan to optimize the settings of the ABS by using evolutionary computation [10] [11] or
machine learning [12] [13]. This will allow us to generate picture books based on human preferences and to
quantitatively evaluate the story. In addition, current ABS propose a solution where all agents select and perform
actions according to their emotions, their personalities and so on. However, these elements have a weak influence
while a strong influence is given by random numbers. Therefore, the implementation of a new ABS, where actions,
emotions, personalities and roles would have a strong influence for each agent's actions is also an important future
work.
VI. ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant, Grant-in-Aid for Scientific Research(C), 26330282.
REFERENCES
[1] G. Prince, A dictionary of narratology (University of Nebraska Press, 2003).
[2] Turner S. R, The creative process: a computer model of storytelling and creativity (Psychology Press,
1994).
[3] Rafael Pérez y Pérez and Mike Sharples, Mexica: a computer model of a cognitive account of creative
writing, Journal of Experimental & Theoretical Artificial Intelligence, Vol. 13, No. 2, 2001, 119-139.
[4] Pablo Gervás, Belén Díaz-Agudo, Federico Peinado, and Raquel Hervás, Story plot generation based on
CBR, Knowledge-Based Systems, Vol. 18, No. 45, 2005, 235-242.
[5] J. R. Meehan, The metanovel: writing stories by Computer (Garland Publishing, 1980).
[6] Ruck Thawonmas and Tomonori Shuda, Comic layout for automatic comic generation from game log, in
Ciancarini, P., Nakatsu, R., Rauterberg, M., Roccetti, M. (Eds.), New frontiers for entertainment
computing (Springer US, Boston, MA, 2008.) 105-115.
[7] Miki Ueno, Naoki Mori, and Keinosuke Matsumoto, 2-scene, comic creating system based on the
distribution of picture state transition, in Sigeru Omatu, Hugues Bersini, Juan M. Corchado, Sara
Rodríguez, Paweł Pawlewski, Edgardo Bucciarelli (Eds.), Distributed Computing and Artificial
Intelligence, 11th International Conference (Springer International Publishing, Cham, 2014) 459-467.
[8] Paul Davidsson, Multi agent based simulation: beyond social simulation, in Scott Moss, Paul Davidsson
(Eds.), Multi-Agent-Based Simulation (Springer Berlin Heidelberg, Berlin, Heidelberg, 2001) 97-107.
[9] Charles M. Macal and Michael J. North, Tutorial on agent-based modeling and simulation, In Proc. of the
37th Conference on Winter Simulation, WSC '05, 2005, 2-15.
[10] B. W. Goldman and W. F. Punch, Parameter-less population pyramid, In Proc. of the 2014 Conference on
Genetic and Evolutionary Computation, GECCO'14, (New York, NY, USA, 2014. ACM) 785-792.
[11] N. Hansen, The cma evolution strategy: a comparing review, in Lozano, J.A., Larrañaga, P., Inza, I.,
Bengoetxea, E. (Eds.), Towards a new evolutionary computation (Springer, 2006) 75-102.
[12] Richard S. Sutton and Andrew G. Barto, Introduction to reinforcement learning, (MIT Press, Cambridge,
MA, USA, 1st edition, 1998).
[13] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, Playing
atari with deep reinforcement learning, ArXiv, e-prints, 2013.

BIOGRAPHIES
Kiyohito Fukuda is a doctor’s course student in the department of Computer Science and Intelligent Systems at
Osaka Prefecture University. He received his B.S. and M.S. degrees in engineering from Osaka Prefecture
University in 2013 and 2015 respectively. His research interests include natural language processing, and artificial
intelligence.
Naoki Mori is an Associate Professor in the Department of Computer Science and Intelligent Systems at Osaka
Prefecture University. He received his B.E. degree in 1992 from Department of Physics, Kyoto University, Kyoto,
Japan, the M.E. degree in 1994 from Department of Nuclear Engineering, Kyoto University, and the Ph.D. degree
in 1999 from the Department of Electrical Engineering, Kyoto University. He was a Visiting Fellow at the
University of New South Wales at Australian Defence Force Academy from October 2003 to September 2004. His
research interests include evolutionary computation and machine learning.
Keinosuke Matsumoto is an Emeritus Professor at Osaka Prefecture University. He received his B.S. and M.S.
degrees in mechanics from Kyoto University in 1976 and 1978 respectively. He worked for Mitsubishi Electric
Corporation as a researcher from 1978 to 1996. He received his Ph.D. in electrical engineering from Kyoto
University for his work on a knowledge-based approach to power system restoration. His research interests include
software engineering, object-oriented technologies, and intelligent systems.

Semi-automatic Picture Book Generation based on Story Model and Agent-based Simulation

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Semi-automatic Picture Book Generation based on Story Model and Agent-based Simulation

Semelhante a Semi-automatic Picture Book Generation based on Story Model and Agent-based Simulation (20)

Mais de IJMREMJournal

Mais de IJMREMJournal (20)

Último

Último (20)

Semi-automatic Picture Book Generation based on Story Model and Agent-based Simulation