Social media provides information, in the form of images, that is valuable to a vast set of human activities, including salvage and rescue in the case of crisis situations (such as accidents, explosions, and fire). However, these services produce images in a rate that is impossible for human beings to absorb and analyze; thus, it is a requirement to have methods for automatic analysis. However, despite the multiple works on image analysis, there are no studies on the specific topic of fire detection over social media. To fill this gap, this work describes the use and the evaluation of an ample set of content-based image retrieval and classification techniques in the task of fire detection. In our intent, we (1) built a ground-truth set of annotated images regarding fire occurrence; (2) engineered the Fast-Fire Detection and Retrieval ($\FFDnR$) architecture to combine configurations of feature extractors and distance functions to work with instance-based learning; and (3) evaluated 36 image descriptors in the task of fire detection. Our results demonstrated that, for fire detection, the best image descriptors concerning efficacy (F-measure, Precision-Recall, and ROC) and processing efficiency (wall-clock time) are achieved with MPEG-7 feature extractors Color Structure and Scalable Color, and with distance functions City-Block and Euclidean. Our work shall provide basis for further developments regarding monitoring of images from social media.

1. Marcos Vinícius Naves Bedo (speaker)
bedo@icmc.usp.br
Gustavo Blanco, Willian Oliveira, Mirela Cazzolato,
Alceu Costa, José F. Rodrigues Jr.,
Agma Traina, Caetano Traina Jr.
Techniques for effective and efficient fire
detection from social media images
Full paper at: http://www.icmc.usp.br/pessoas/junio

2. Outline
 Introduction
 Background
 The Fast-Fire Detection Method
 Experiments
 Conclusions

3. Outline
 Introduction
 Rescuer Project
 Fire Detection Module
 Background
 The Fast-Fire Detection Method
 Experiments
 Conclusions

4. Rescuer Project
 The RESCUER project is a BR-EU consortium
aiming at developing solutions to improve the
decision-making process in disaster
situations:
 Industrial plants;
 Densely populated area;
 Crowded events;
 Project details: http://www.rescuer-project.org/

5. A smartphone user
sends textual data
about the situation.
The user may also upload
multimedia content such as
photo and video
Multimedia
data are
automatically
analyzed
http://www.rescuer-project.org/

6. A smartphone user
sends textual data
about the situation.
The user may also upload
multimedia content such as
photo and video
Multimedia
data are
automatically
analyzed
http://www.rescuer-project.org/http://www.rescuer-project.org/

7. A smartphone user
sends textual data
about the situation.
The user may also upload
multimedia content such as
photo and video
Multimedia
data are
automatically
analyzed
http://www.rescuer-project.org/http://www.rescuer-project.org/

8. A smartphone user
sends textual data
about the situation.
The user may also upload
multimedia content such as
photo and video
Multimedia
data are
automatically
analyzed
http://www.rescuer-project.org/http://www.rescuer-project.org/

9. Image Analysis
• Fire detection
– Presence of fire images
located near an
emergency scenario

10. Image Analysis
• Fire detection
– Presence of fire images
located near an
emergency scenario

11. Image Analysis
• Fire detection
– Presence of fire images
located near an
emergency scenario
Identify fire presence in images
arriving from the Flickr social network

12. Problem Definition
• Problem Definition
– Given an image previously updated to
a social media service, return 'true' if
there is fire or 'false' otherwise.

13. Outline
 Introduction
 Background
 The Fast-Fire Detection Method
 Experiments
 Conclusions

14. Outline
 Introduction
 Background
 Feature Extractor Methods
 Evaluation Functions
 Instance-Based Learning
 The Fast-Fire Detection Method
 Experiments
 Conclusions

15. Fire Detection Module
• Feature Extractor Methods
– MPEG-7: designed to represent color,
texture and shape
– Standardize representation for color
images

16. MPEG7 - Color Extractor Methods
Spatial Correlation Number of Features Color Space
Color Layout Yes 16 YCbCr

17. MPEG7 - Color Extractor Methods
Spatial Correlation Number of Features Color Space
Color Layout Yes 16 YCbCr
Color Structure Yes 128 HMMD

18. MPEG7 - Color Extractor Methods
• Hue
• Saturation
• Value
Haar transform
Spatial Correlation Number of Features Color Space
Color Layout Yes 16 YCbCr
Color Structure Yes 128 HMMD
Scalable Color No 256 HSV

19. MPEG7 - Color Extractor Methods
Spatial Correlation Number of Features Color Space
Color Layout Yes 16 YCbCr
Color Structure Yes 128 HMMD
Scalable Color No 256 HSV
Color
Temperature
No 1 XYZ

20. MPEG7 - Texture Methods
- Local Count
- Global Count
Count Approach Number of Features
Edge Histogram Yes 150

21. MPEG7 - Texture Methods
Count Approach Number of Features
Edge Histogram Yes 150
Texture-Browsing No 12

22. Evaluation Functions
Evaluation Function Distance Function Acronym
City-Block Yes CB
Euclidean Yes EU
Chebyshev Yes CH
Canberra Yes CA
• We employed six evaluation functions
as possibles setting to the
classification task

23. Evaluation Functions
• We employed six evaluation functions
as possibles setting to the
classification task
Evaluation Function Distance Function Acronym
City-Block Yes CB
Euclidean Yes EU
Chebyshev Yes CH
Canberra Yes CA
Kullback-Leibler No KU
Jeffrey Divergence No JF

24. Image Descriptors
• Image Descriptor
– An image descriptor is a pair <feature
extractor method, evaluation function>
– By using the previous evaluation
functions and feature extractor methods,
36 can be arranged.
– Image descriptors define the search
space.
24

25. Instance-Based Learning
• Assumption: Elements of the same class
belong to the same neighborhood
Iq

26. Instance-Based Learning
• Assumption: Elements of the same class
belong to the same neighborhood
Iq

27. Instance-Based Learning
• Assumption: Elements of the same class
belong to the same neighborhood
Iq

28. Instance-Based Learning
The Iq is labeled
according to its k
nearest neighbors.
Iq

29. Outline
 Introduction
 Background
 The Fast-Fire Detection Method
 Experiments
 Conclusions

30. Outline
 Introduction
 Background
 The Fast-Fire Detection Method
 Architecture
 Fire-Flickr Dataset
 Experiments
 Conclusions

31. Fire Detection Module
Image

32. Fire Detection Module
0.7 0.4 0.1 0.9 0.2 ...
Data representation
through a Feature
Extractor Method
Image

33. Fire Detection Module
0.7 0.4 0.1 0.9 0.2 ...
Data representation
through a Feature
Extractor Method
Image
Classifier
Knowledge
Database

34. Fire Detection Module
0.7 0.4 0.1 0.9 0.2 ...
Data representation
through a Feature
Extractor Method
Image
Data classification –
may require an
Evaluation Function
Classifier
Knowledge
Database

35. The FFireDt Method
• FFireDT: Our proposal
– Setting: Image Descriptor
– Set of modules to perform image
analysis:
• Feature Extractor Module
• Evaluation Functions Module
• IBL classifier module

36. The FFireDt Method

37. The FFireDt Method

38. The FFireDt Method

39. Fire-Flickr Dataset
• Downloaded 5,962 images from Flickr
API
– Textual descriptors as ‘fire car accident’,
‘criminal fire’, ‘house burning’, etc.
– 7 subjects (non-blinded) evaluated the
images as containing or not traces of fire
• Average disagreement 7.2%
– 1,000 images with and without fire

40. Fire-Flickr Dataset
{fire}
{not fire}
• Dataset avaliable at:
www.gbdi.icmc.usp.br
– Including the extractors and functions

41. Outline
 Introduction
 Background
 The Fast-Fire Detection Method
 Experiments
 Conclusions

42. Outline
 Introduction
 Background
 The Fast-Fire Detection Method
 Experiments
 F-measure
 Precision x Recall and ROC curves
 Performance Evaluation
 Conclusions

43. Experiments
• Metrics to evaluate FFireDt
– Test to evaluate F-Measure
– Precision vs. Recall curves
– ROC curves
• Processing Performance
– Image Descriptors
• ‘Cost x Benefit’ Analysis

44. Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
City-Block
Jeffrey Divergence
Canberra
Kullback Leibler
Euclidean
Chebyshev
Results
• F-Measure using all possible settings
for FFireDt
F-Measure -> Higher is better

45. Results
• F-Measure using all possible settings
for FFireDt
0.847
Image Descriptor <Color Layout, Euclidean>
Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
City-Block
Jeffrey Divergence
Canberra
Kullback Leibler
Euclidean
Chebyshev

46. Results
• F-Measure using all possible settings
for FFireDt
Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
City-Block
Jeffrey Divergence
Canberra
Kullback Leibler
Euclidean
Chebyshev
0.843
Image Descriptor <Scalable Color, City-Block>

47. Results
• F-Measure using all possible settings
for FFireDt
Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
City-Block
Jeffrey Divergence
Canberra
Kullback Leibler
Euclidean
Chebyshev
0.866
Image Descriptor <Color Structure, Jeffrey>

48. Results
• F-Measure using all possible settings
for FFireDt
Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
City-Block
Jeffrey Divergence
Canberra
Kullback Leibler
Euclidean
Chebyshev
0.800
Image Descriptor <Color Temperature, Canberra>

49. Results
• F-Measure using all possible settings
for FFireDt
Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
City-Block
Jeffrey Divergence
Canberra
Kullback Leibler
Euclidean
Chebyshev
0.815
Image Descriptor <Edge Histogram, Jeffrey>

50. Results
• F-Measure using all possible settings
for FFireDt
Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
City-Block
Jeffrey Divergence
Canberra
Kullback Leibler
Euclidean
Chebyshev
0.766
Image Descriptor <Texture Browsing, City-Block>

51. Results
• The top-6 image descriptors grouped by
feature extractor methods were:
– ID1: Color Strucuture and Jeffrey Divergence
– ID2: Color Layout and Euclidean
– ID3: Scalable Color and City-Block
– ID4: Edge Histogram and Jeffrey Divergence
– ID5: Color Temperature and Canberra
– ID6: Texture Browsing and City-Block

52. Results
• The Precision-Recall curves show that
ID1, ID2 and ID3 achieved a better
behavior than others

53. Results
• The Precision-Recall curves show that
ID1, ID2 and ID3 achieved a better
behavior than others

54. Results
• The Precision-Recall curves show that
ID1, ID2 and ID3 achieved a better
behavior than others
– We discard the bottom-3 candidates

55. Results
• We checked the ROC curves for ID1, ID2
and ID3

56. Results
• We checked the ROC curves for ID1, ID2
and ID3

57. Results
• We checked the ROC curves for ID1, ID2
and ID3

58. Results
• We checked the ROC curves for ID1, ID2
and ID3
The choice becomes a
matter of
performance!

59. Results
• Processing Time
– Feature Extractor Method

60. Results
• Processing Time
– Feature Extractor Method

61. Results
• Processing Time
– Evaluation Function costs

62. Results
• Processing Time
– Evaluation Function costs

63. Results
• Performance Analysis (cost vs. benefit)
– Feature Extractor Methods
• Color Structure and Scalable Color
– Evaluation Functions
• City Block, Euclidean, and Chebyshev as
Evaluation Functions

64. Results
• Performance Analysis (cost vs. benefit)
– Color Structure and Scalable Color
– City Block,Euclidean, and Chebyshev
0.853
Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
City-Block
Jeffrey Divergence
Canberra
Kullback Leibler
Euclidean
Chebyshev

65. Outline
 Introduction
 Background
 The Fast-Fire Detection Method
 Experiments
 Conclusions

66. Conclusions
• We designed a new approach to fire
detection: FFireDT
• FFireDT has achieved a precision closer
to human annotation in fire detection
– Experiments show the precision and
computational cost
– Determine the most suitable Image
Descriptor as FFireDT setting

67. Thank you for your attention!
Techniques for effective and efficient fire
detection from social media images
Marcos Vinícius Naves Bedo (speaker)
bedo@icmc.usp.br
Gustavo Blanco, Willian Oliveira, Mirela Cazzolato,
Alceu Costa, José F. Rodrigues Jr.,
Agma Traina, Caetano Traina Jr.

68. Results
• FFireDt using Instance Based Learning
vs. other classifiers
Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
FFireDT
Naive-Bayes
RandomForest
J48
0.8
0.7
0.6
0.5
0.4
0.3
0.9

69. Color
Layout
Scalable
Color
Color
Structure
Color
Temperature
Edge
Histogram
Texture
Browsing
FFireDT
Naive-Bayes
RandomForest
J48
0.8
0.7
0.6
0.5
0.4
0.3
0.9
Results
• FFireDt using Instance Based Learning
vs. other classifiers