SlideShare uma empresa Scribd logo
1 de 62
Baixar para ler offline
Avv. Giovanni Battista Gallus
LL.M. Ph.d. ISO 27001 Lead Auditor
gallus@array.eu - gallus@gm-lex.eu
The laws of robotics and autonomous vehicles may be
much more than three, but don't panic... yet.
@gbgallusAvv. Giovanni Battista Gallus
●
A Robolaw for all robots?
“Hard” laws and “soft” laws
Overview of the Eu Resolution
on robots
Civil liabilities
Autonomous veihicles
Ethical issues
@gbgallusAvv. Giovanni Battista Gallus
●
The 2017 Vienna Biennale
@gbgallusAvv. Giovanni Battista Gallus
●
From 1980
Paramount Pictures (1980)
© Paramount Pictures (1980)
@gbgallusAvv. Giovanni Battista Gallus
●
To 2017
@gbgallusAvv. Giovanni Battista Gallus
●
From Avatar
@gbgallusAvv. Giovanni Battista Gallus
●
To reality
@gbgallusAvv. Giovanni Battista Gallus
●
From the Jetsons
© Hanna-Barbera
@gbgallusAvv. Giovanni Battista Gallus
●
To our homes and nursing
homes
@gbgallusAvv. Giovanni Battista Gallus
●
Industry 4.0
By ChristophRoser at http://www.allaboutlean.com AllAboutLean.com
Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=47640595
@gbgallusAvv. Giovanni Battista Gallus
●
All men robots are not
created equal
“The term robot is a-technical and encompasses a wide range of
applications that have very little in common. For this very reason,
it is impossible to develop a unitary body of rules applicable to
all kinds of robotic applications, rather different rules should
apply to different classes of devices”
A. Bertolini
Kitmondo PPM
David Stillman
@gbgallusAvv. Giovanni Battista Gallus
●
“Smart Robot” definition
The acquisition of autonomy through sensors and/or
by exchanging data with its environment (inter-
connectivity) and the trading and analysing of those
data;
self-learning from experience and by interaction
(optional criterion);
at least a minor physical support;
the adaptation of its behaviour and actions to the
environment;
absence of life in the biological sense
European Parliament resolution of 16 February 2017
@gbgallusAvv. Giovanni Battista Gallus
●
Question time!
Which among the
following legal
statuses is closer,
at the present
time, to the one
applicable to
smart robots?
@gbgallusAvv. Giovanni Battista Gallus
●
Digital person?
https://en.wikipedia.org/w/index.php?curid=12543502
@gbgallusAvv. Giovanni Battista Gallus
●
Slave?
@gbgallusAvv. Giovanni Battista Gallus
●
Pet/animal?
@gbgallusAvv. Giovanni Battista Gallus
●
Children/minor?
@gbgallusAvv. Giovanni Battista Gallus
●
Software?
@gbgallusAvv. Giovanni Battista Gallus
●
Tool (machine – car –
manufactured object)?
@gbgallusAvv. Giovanni Battista Gallus
●
Hard laws
Directive 85/374/EEC on liability for defective products
Machinery Directive 2006/42/EC
Council Directive 89/391/EEC of 12 June 1989on safety and health at work
Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April
2017 on medical devices
EU Regulation 2016/679 (GDPR)
Future EU Regulation on Drones (Notice of Proposed Amendment 2017-05 (A)-
Introduction of a regulatory framework for the operation of drones)
Convention on damage caused by foreign aircraft to third parties on the surface, 7
October 1952
Several Conventions regarding international civil liabilities (nuclear power;
dangerous and noxious substances; oil pollution etc)
The hardest part: all national laws regulating civil and criminal liability for
harmful acts
@gbgallusAvv. Giovanni Battista Gallus
●
Soft Laws
European Parliament resolution of 16 February 2017 with
recommendations to the Commission on Civil Law Rules on
Robotics
Charter on Robotics
– CODE OF ETHICAL CONDUCT FOR ROBOTICS ENGINEERS
European Parliament’s resolution of 29 October 2015 on safe use
of remotely piloted aircraft systems
Opinion 1/2015 on Privacy and Data Protection Issues relating to
the Utilisation of Drones
2015 Riga Declaration on remotely piloted aircraft (drones)
All relevant technical standards (ISO/TC 299 -Robotics and others)
@gbgallusAvv. Giovanni Battista Gallus
●
@gbgallusAvv. Giovanni Battista Gallus
●
@gbgallusAvv. Giovanni Battista Gallus
●
We must accept liability
The trend towards automation
requires that those involved in
the development and
commercialisation of AI
applications build in security
and ethics at the outset,
thereby recognizing that they
must be prepared to accept
legal liability for the quality of
the technology they produce
@gbgallusAvv. Giovanni Battista Gallus
●
Control and verification
Safeguards and the possibility of human control
and verification need to be built into the
process of automated and algorithmic
decision-making
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations - general
Compulsory registration
EU Agency for Robotics and Artificial
Intelligence
Attention to emotional connection between
humans and robots
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations - general
“Kill switch”
(sorry, HAL 9000)
Cryteria
https://commons.wikimedia.org/w/index.php?
curid=11651154
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations –
research & innovation
Open science and responsible ethical innovation
Interoperability between systems, devices and
cloud services, based on security and privacy by
design
Open environment, from open standards and
innovative licensing models, to open platforms
and transparency, in order to avoid lock-in in
proprietary systems that restrain interoperability
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations – Ethical
principles I
Human safety, health and security; freedom,
privacy, integrity and dignity; self-
determination and non-discrimination;
personal data protection
Strict and efficient guiding ethical
framework for the development, design,
production, use and modification of robots
@gbgallusAvv. Giovanni Battista Gallus
●
Whose ethics rule?
The beginning of ‘policy’ for
robotics is that agents have to
act ethically right and
maximise good consequences.
The main value issues seem to
concern honesty, esp.
nondeception of users and
customers.
euRobotics Topics Group
Ethical, Legal and Socio-economic
Issues in Robotics
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations – Ethical
principles II
Transparency
it must always be possible to reduce the AI system´s computations to a form
comprehensible by humans;
advanced robots should be equipped with a ‘black box’ which records data on
every transaction carried out by the machine, including the logic that contributed
to its decisions
Guiding ethical framework should be based on the principles of
beneficence, non-maleficence, autonomy and justice, on the principles
and values enshrined in Article 2 of the Treaty on European Union and in
the Charter of Fundamental Rights
Special attention should be paid to robots that represent a significant
threat to confidentiality owing to their placement in traditionally
protected and private spheres
@gbgallusAvv. Giovanni Battista Gallus
●
And now...
X RATED MATERIAL
@gbgallusAvv. Giovanni Battista Gallus
●
Roboprostitution
© Warner Bros, 2001
© HBO 201
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations – IP &
flow of data
Intellectual Property
Compliance with the principles of GDPR
right to respect for private life and to the
protection of personal data
High level of security
– Responsibility of designers of robotics and AI to
develop products to be safe, secure and fit for
purpose
@gbgallusAvv. Giovanni Battista Gallus
●
Data protection & robots
https://www.flickr.com/photos/pallina60loon/
@gbgallusAvv. Giovanni Battista Gallus
●
Data protection by
design
Data protection by
default
Data Protection
Impact
assessment
EU Regulation 2016/679
Necessity &
proportionality
Data protection by
default
Transparency
Certification
mechanism
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations –
Standards safety and
security
International harmonisation of technical
standards
Lawful reverse-engineering and open
standards
Setting up of special technical committees,
such as ISO/TC 299 Robotics
Efficient strategy & monitoring mechanism for
testing robots in real-life scenarios
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations –
Autonomous means of
transport
Urgent need for global rules
Drivers reaction time (human
factor)
Drones
@gbgallusAvv. Giovanni Battista Gallus
●
“A key finding is that it took 2-3 seconds for participants to ‘takeover’
manual controls and resume active driving after short periods of
autonomous driving in urban environments”
http://www.venturer
-cars.com/trial-1-
results/
@gbgallusAvv. Giovanni Battista Gallus
●
@gbgallusAvv. Giovanni Battista Gallus
●
@gbgallusAvv. Giovanni Battista Gallus
●
On the other side of the
pond
http://cyberlaw.stanford.edu/wiki/index.php/Automated_Driving:_Legislative_and_Regulatory_Action
@gbgallusAvv. Giovanni Battista Gallus
●
@gbgallusAvv. Giovanni Battista Gallus
●
The (near) future
for drones
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations – Care &
medical Robots
Human factor is still fundamental
Proper training of surgeons
Vital to respect the principle of
the supervised autonomy of
robots
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations – Human
repair and enhancement
Cyber physical systems
Appropriately staffed committees on robot
ethics, in hospitals & health care institutions
Patching and updating of vital medical
applications (also by independent trusted
entities)
@gbgallusAvv. Giovanni Battista Gallus
●
Human enhancements &
biorobotic devices
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations – Liability I
European framework
Smooth joint human / robot activities
Legislative initiatives + soft laws
No limitation on damages not to property, on the sole
grounds that damage is caused by a non-human agent
Strict liability or risk management approach?
At least at the present stage the responsibility
must lie with a human and not a robot
@gbgallusAvv. Giovanni Battista Gallus
●
The blame game: the
monster or the creator?
We need to retain the principle
that humans are responsible for
their actions, while robots and
other machines are not
euRobotics Topics Group
Ethical, Legal and Socio-economic Issues
in Robotics
© Gruskoff/Venture Films 1974
@gbgallusAvv. Giovanni Battista Gallus
●
Recommendations – Liability II
Programming vs training
Liability should be proportional to the actual level of instructions given to
the robot and of its degree of autonomy, so that the greater a robot's
learning capability or autonomy, and the longer a robot's training, the
greater the responsibility of its trainer should be
Obligatory insurance scheme, taking into account all potential
responsibilities in the chain
Compensation fund for uninsured robots
Limited liability for the manufacturer, the programmer, the owner
or the user, if they contribute to a compensation fund, as well as if
they jointly take out insurance to guarantee compensation where
damage is caused by a robot
@gbgallusAvv. Giovanni Battista Gallus
●
The Jessica Rabbit principle,
applied to robotics
I'm not bad
I’m just
drawn build,
programmed and
trained
that way...
© Touchstone Pictures 1988
@gbgallusAvv. Giovanni Battista Gallus
●
The importance of training
@gbgallusAvv. Giovanni Battista Gallus
●
Asimov’s laws are not
enough (sorry, Isaac...)
A robot may not injure a human being or,
through inaction, allow a human being to come
to harm
A robot must obey the orders given by human
beings except where such orders would conflict
with the first law
A robot must protect its own existence as long
as such protection does not conflict with the
first or second laws
@gbgallusAvv. Giovanni Battista Gallus
●
Even if you change their
order...
https://xkcd.com/1613/
@gbgallusAvv. Giovanni Battista Gallus
●
The future of RoboLiability
The inadequacies of existing rules suggests to
radically replace a fault based rule with a risk-
management approach (based on absolute
liability rules), holding liable the party who is
better placed to minimize the cost and
acquire insurance
euRobotics Topics Group
Ethical, Legal and Socio-economic Issues in
Robotics
@gbgallusAvv. Giovanni Battista Gallus
●
And finally, RoboArt
Are we ready to accept the gift of the Evil
deity?
@gbgallusAvv. Giovanni Battista Gallus
●
Is a robot luckier than me?
Many thanks to Marco Scialdone for the idea
@gbgallusAvv. Giovanni Battista Gallus
●
In conclusion, we might ask
Marvin...
“I've calculated
your chance of
survival, but I
don't think you'll
like it”
By Chris Favero - USA (Marvin the Paranoid Android #cosplay #C2E2 2015)
via Wikimedia Commons
@gbgallusAvv. Giovanni Battista Gallus
●
The success of robots lies,
quite literally, in your hands
© Spyglass Entertainment, Universal Pictures, 2003
@gbgallusAvv. Giovanni Battista Gallus
●
@gbgallusAvv. Giovanni Battista Gallus
●
Questions?Questions?
@gbgallusAvv. Giovanni Battista Gallus
●
Avv. Giovanni Battista Gallus
Let’s continue the conversation
gallus@array.eu @gbgallus
Unless stated otherwise,
all texts are distributed
under a Creative Commons
Attribution – non
commercial – sharealike
3.0 Unported license

Mais conteúdo relacionado

Semelhante a Robotics law: Ada-Europe 2017 presentation

IRJET- Vehicle Security System using IoT Application
IRJET-  	  Vehicle Security System using IoT ApplicationIRJET-  	  Vehicle Security System using IoT Application
IRJET- Vehicle Security System using IoT ApplicationIRJET Journal
 
2019 - Colloque National Enseignants DUT GEII
2019 - Colloque National Enseignants DUT GEII2019 - Colloque National Enseignants DUT GEII
2019 - Colloque National Enseignants DUT GEIINicolas Lesconnec
 
DEDICAT 6G_HEXA-X_Workshop_on-6G The Dedicat6G use cases
DEDICAT 6G_HEXA-X_Workshop_on-6G The Dedicat6G use casesDEDICAT 6G_HEXA-X_Workshop_on-6G The Dedicat6G use cases
DEDICAT 6G_HEXA-X_Workshop_on-6G The Dedicat6G use casesDedicat6G
 
4_12_November_-_2.10pm_-_LIVIA_KECEROVA.pptx
4_12_November_-_2.10pm_-_LIVIA_KECEROVA.pptx4_12_November_-_2.10pm_-_LIVIA_KECEROVA.pptx
4_12_November_-_2.10pm_-_LIVIA_KECEROVA.pptxBotaQ2
 
Techprom presentation 1 6-18-9
Techprom presentation 1 6-18-9Techprom presentation 1 6-18-9
Techprom presentation 1 6-18-9Videotomsk
 
Automakers Changing Tactics On Reliability
Automakers Changing Tactics On ReliabilityAutomakers Changing Tactics On Reliability
Automakers Changing Tactics On ReliabilityAbacus Technologies
 
6G Training Course Part 3: 6G Use Cases & Applications
6G Training Course Part 3: 6G Use Cases & Applications6G Training Course Part 3: 6G Use Cases & Applications
6G Training Course Part 3: 6G Use Cases & Applications3G4G
 
Bluetooth Controlled Garbage Collection Robot Arm
Bluetooth Controlled Garbage Collection Robot ArmBluetooth Controlled Garbage Collection Robot Arm
Bluetooth Controlled Garbage Collection Robot ArmIRJET Journal
 
Making a Better World with Technology Innovations
Making a Better World with Technology InnovationsMaking a Better World with Technology Innovations
Making a Better World with Technology InnovationsImesh Gunaratne
 
MiDoMet Smart metering system
MiDoMet Smart metering systemMiDoMet Smart metering system
MiDoMet Smart metering systemMiDo Srl
 
Cwin16 tls-iot approach-applied_in_the_plm_domain
Cwin16 tls-iot approach-applied_in_the_plm_domainCwin16 tls-iot approach-applied_in_the_plm_domain
Cwin16 tls-iot approach-applied_in_the_plm_domainCapgemini
 
Summarized paper next generation advanced driver assistance systems towards t...
Summarized paper next generation advanced driver assistance systems towards t...Summarized paper next generation advanced driver assistance systems towards t...
Summarized paper next generation advanced driver assistance systems towards t...Ilona Anna Cieslik
 
Techprom presentation 1 6-18-10
Techprom presentation 1 6-18-10Techprom presentation 1 6-18-10
Techprom presentation 1 6-18-10Videotomsk
 
PROJECT LIGTAS PRESENTATION.pptx
PROJECT LIGTAS PRESENTATION.pptxPROJECT LIGTAS PRESENTATION.pptx
PROJECT LIGTAS PRESENTATION.pptxJessaSiares
 
Techprom presentation 1 6-18-8
Techprom presentation 1 6-18-8Techprom presentation 1 6-18-8
Techprom presentation 1 6-18-8Videotomsk
 
IOT SMOKE DETECTION SYSTEM USING ARDUINO
IOT SMOKE DETECTION SYSTEM USING ARDUINOIOT SMOKE DETECTION SYSTEM USING ARDUINO
IOT SMOKE DETECTION SYSTEM USING ARDUINOIRJET Journal
 
A Survey on Vehicle Tracking System using IoT
A Survey on Vehicle Tracking System using IoTA Survey on Vehicle Tracking System using IoT
A Survey on Vehicle Tracking System using IoTIRJET Journal
 
IOT Innovation Ecosystem - Smart Cities Summit 2018 - Algiers
IOT Innovation Ecosystem - Smart Cities Summit 2018 - AlgiersIOT Innovation Ecosystem - Smart Cities Summit 2018 - Algiers
IOT Innovation Ecosystem - Smart Cities Summit 2018 - AlgiersSmart Algiers
 

Semelhante a Robotics law: Ada-Europe 2017 presentation (20)

IRJET- Vehicle Security System using IoT Application
IRJET-  	  Vehicle Security System using IoT ApplicationIRJET-  	  Vehicle Security System using IoT Application
IRJET- Vehicle Security System using IoT Application
 
2019 - Colloque National Enseignants DUT GEII
2019 - Colloque National Enseignants DUT GEII2019 - Colloque National Enseignants DUT GEII
2019 - Colloque National Enseignants DUT GEII
 
DEDICAT 6G_HEXA-X_Workshop_on-6G The Dedicat6G use cases
DEDICAT 6G_HEXA-X_Workshop_on-6G The Dedicat6G use casesDEDICAT 6G_HEXA-X_Workshop_on-6G The Dedicat6G use cases
DEDICAT 6G_HEXA-X_Workshop_on-6G The Dedicat6G use cases
 
4_12_November_-_2.10pm_-_LIVIA_KECEROVA.pptx
4_12_November_-_2.10pm_-_LIVIA_KECEROVA.pptx4_12_November_-_2.10pm_-_LIVIA_KECEROVA.pptx
4_12_November_-_2.10pm_-_LIVIA_KECEROVA.pptx
 
Techprom presentation 1 6-18-9
Techprom presentation 1 6-18-9Techprom presentation 1 6-18-9
Techprom presentation 1 6-18-9
 
Automakers Changing Tactics On Reliability
Automakers Changing Tactics On ReliabilityAutomakers Changing Tactics On Reliability
Automakers Changing Tactics On Reliability
 
6G Training Course Part 3: 6G Use Cases & Applications
6G Training Course Part 3: 6G Use Cases & Applications6G Training Course Part 3: 6G Use Cases & Applications
6G Training Course Part 3: 6G Use Cases & Applications
 
Bluetooth Controlled Garbage Collection Robot Arm
Bluetooth Controlled Garbage Collection Robot ArmBluetooth Controlled Garbage Collection Robot Arm
Bluetooth Controlled Garbage Collection Robot Arm
 
Making a Better World with Technology Innovations
Making a Better World with Technology InnovationsMaking a Better World with Technology Innovations
Making a Better World with Technology Innovations
 
MiDoMet Smart metering system
MiDoMet Smart metering systemMiDoMet Smart metering system
MiDoMet Smart metering system
 
ART20204396.pdf
ART20204396.pdfART20204396.pdf
ART20204396.pdf
 
Cwin16 tls-iot approach-applied_in_the_plm_domain
Cwin16 tls-iot approach-applied_in_the_plm_domainCwin16 tls-iot approach-applied_in_the_plm_domain
Cwin16 tls-iot approach-applied_in_the_plm_domain
 
Summarized paper next generation advanced driver assistance systems towards t...
Summarized paper next generation advanced driver assistance systems towards t...Summarized paper next generation advanced driver assistance systems towards t...
Summarized paper next generation advanced driver assistance systems towards t...
 
Techprom presentation 1 6-18-10
Techprom presentation 1 6-18-10Techprom presentation 1 6-18-10
Techprom presentation 1 6-18-10
 
5G Enablers and Use Cases, an European Pespective
5G Enablers and Use Cases, an European Pespective5G Enablers and Use Cases, an European Pespective
5G Enablers and Use Cases, an European Pespective
 
PROJECT LIGTAS PRESENTATION.pptx
PROJECT LIGTAS PRESENTATION.pptxPROJECT LIGTAS PRESENTATION.pptx
PROJECT LIGTAS PRESENTATION.pptx
 
Techprom presentation 1 6-18-8
Techprom presentation 1 6-18-8Techprom presentation 1 6-18-8
Techprom presentation 1 6-18-8
 
IOT SMOKE DETECTION SYSTEM USING ARDUINO
IOT SMOKE DETECTION SYSTEM USING ARDUINOIOT SMOKE DETECTION SYSTEM USING ARDUINO
IOT SMOKE DETECTION SYSTEM USING ARDUINO
 
A Survey on Vehicle Tracking System using IoT
A Survey on Vehicle Tracking System using IoTA Survey on Vehicle Tracking System using IoT
A Survey on Vehicle Tracking System using IoT
 
IOT Innovation Ecosystem - Smart Cities Summit 2018 - Algiers
IOT Innovation Ecosystem - Smart Cities Summit 2018 - AlgiersIOT Innovation Ecosystem - Smart Cities Summit 2018 - Algiers
IOT Innovation Ecosystem - Smart Cities Summit 2018 - Algiers
 

Mais de Giovanni Battista Gallus

Presentazione sul Drone Journalism - Master Giornalismo di Torino
Presentazione sul Drone Journalism - Master Giornalismo di TorinoPresentazione sul Drone Journalism - Master Giornalismo di Torino
Presentazione sul Drone Journalism - Master Giornalismo di TorinoGiovanni Battista Gallus
 
Free/open source software and drones Security and privacy issues
Free/open source software and drones Security and privacy issuesFree/open source software and drones Security and privacy issues
Free/open source software and drones Security and privacy issuesGiovanni Battista Gallus
 
Trojans and criminal investigation: is it possibile to find a balance with fu...
Trojans and criminal investigation: is it possibile to find a balance with fu...Trojans and criminal investigation: is it possibile to find a balance with fu...
Trojans and criminal investigation: is it possibile to find a balance with fu...Giovanni Battista Gallus
 
Free as in freedom. The importance of FOSS in the surveillance era
Free as in freedom. The importance of FOSS in the surveillance eraFree as in freedom. The importance of FOSS in the surveillance era
Free as in freedom. The importance of FOSS in the surveillance eraGiovanni Battista Gallus
 
I droni e la legge - presentazione SINNOVA15 - gallus
I droni e la legge - presentazione SINNOVA15 - gallusI droni e la legge - presentazione SINNOVA15 - gallus
I droni e la legge - presentazione SINNOVA15 - gallusGiovanni Battista Gallus
 
Presentazione all'Open Data Day 2014 - Giovanni Battista Gallus
Presentazione all'Open Data Day 2014 - Giovanni Battista GallusPresentazione all'Open Data Day 2014 - Giovanni Battista Gallus
Presentazione all'Open Data Day 2014 - Giovanni Battista GallusGiovanni Battista Gallus
 
Presentazione Eprivacy Winter Edition - Bocconi - http://e-privacy.winstonsmi...
Presentazione Eprivacy Winter Edition - Bocconi - http://e-privacy.winstonsmi...Presentazione Eprivacy Winter Edition - Bocconi - http://e-privacy.winstonsmi...
Presentazione Eprivacy Winter Edition - Bocconi - http://e-privacy.winstonsmi...Giovanni Battista Gallus
 
Presentazione sulla computer forensics, dopo la L. 48/2008
Presentazione sulla computer forensics, dopo la L. 48/2008Presentazione sulla computer forensics, dopo la L. 48/2008
Presentazione sulla computer forensics, dopo la L. 48/2008Giovanni Battista Gallus
 

Mais de Giovanni Battista Gallus (12)

Presentazione sul Drone Journalism - Master Giornalismo di Torino
Presentazione sul Drone Journalism - Master Giornalismo di TorinoPresentazione sul Drone Journalism - Master Giornalismo di Torino
Presentazione sul Drone Journalism - Master Giornalismo di Torino
 
Free/open source software and drones Security and privacy issues
Free/open source software and drones Security and privacy issuesFree/open source software and drones Security and privacy issues
Free/open source software and drones Security and privacy issues
 
Gallus presentazione droni_cortina
Gallus presentazione droni_cortinaGallus presentazione droni_cortina
Gallus presentazione droni_cortina
 
Trojans and criminal investigation: is it possibile to find a balance with fu...
Trojans and criminal investigation: is it possibile to find a balance with fu...Trojans and criminal investigation: is it possibile to find a balance with fu...
Trojans and criminal investigation: is it possibile to find a balance with fu...
 
Free as in freedom. The importance of FOSS in the surveillance era
Free as in freedom. The importance of FOSS in the surveillance eraFree as in freedom. The importance of FOSS in the surveillance era
Free as in freedom. The importance of FOSS in the surveillance era
 
I droni e la legge - presentazione SINNOVA15 - gallus
I droni e la legge - presentazione SINNOVA15 - gallusI droni e la legge - presentazione SINNOVA15 - gallus
I droni e la legge - presentazione SINNOVA15 - gallus
 
Ijf14 tutela fonti_anonime_gallus
Ijf14 tutela fonti_anonime_gallusIjf14 tutela fonti_anonime_gallus
Ijf14 tutela fonti_anonime_gallus
 
Presentazione all'Open Data Day 2014 - Giovanni Battista Gallus
Presentazione all'Open Data Day 2014 - Giovanni Battista GallusPresentazione all'Open Data Day 2014 - Giovanni Battista Gallus
Presentazione all'Open Data Day 2014 - Giovanni Battista Gallus
 
Presentazione Eprivacy Winter Edition - Bocconi - http://e-privacy.winstonsmi...
Presentazione Eprivacy Winter Edition - Bocconi - http://e-privacy.winstonsmi...Presentazione Eprivacy Winter Edition - Bocconi - http://e-privacy.winstonsmi...
Presentazione Eprivacy Winter Edition - Bocconi - http://e-privacy.winstonsmi...
 
Presentazione_linuxday
Presentazione_linuxdayPresentazione_linuxday
Presentazione_linuxday
 
Gallus oristano 25.2.12
Gallus oristano 25.2.12Gallus oristano 25.2.12
Gallus oristano 25.2.12
 
Presentazione sulla computer forensics, dopo la L. 48/2008
Presentazione sulla computer forensics, dopo la L. 48/2008Presentazione sulla computer forensics, dopo la L. 48/2008
Presentazione sulla computer forensics, dopo la L. 48/2008
 

Último

如何办理(Curtin毕业证书)科廷科技大学毕业证学位证书
如何办理(Curtin毕业证书)科廷科技大学毕业证学位证书如何办理(Curtin毕业证书)科廷科技大学毕业证学位证书
如何办理(Curtin毕业证书)科廷科技大学毕业证学位证书SD DS
 
如何办理威斯康星大学密尔沃基分校毕业证学位证书
 如何办理威斯康星大学密尔沃基分校毕业证学位证书 如何办理威斯康星大学密尔沃基分校毕业证学位证书
如何办理威斯康星大学密尔沃基分校毕业证学位证书Fir sss
 
如何办理佛蒙特大学毕业证学位证书
 如何办理佛蒙特大学毕业证学位证书 如何办理佛蒙特大学毕业证学位证书
如何办理佛蒙特大学毕业证学位证书Fir sss
 
如何办理(GWU毕业证书)乔治华盛顿大学毕业证学位证书
如何办理(GWU毕业证书)乔治华盛顿大学毕业证学位证书如何办理(GWU毕业证书)乔治华盛顿大学毕业证学位证书
如何办理(GWU毕业证书)乔治华盛顿大学毕业证学位证书SD DS
 
POLICE ACT, 1861 the details about police system.pptx
POLICE ACT, 1861 the details about police system.pptxPOLICE ACT, 1861 the details about police system.pptx
POLICE ACT, 1861 the details about police system.pptxAbhishekchatterjee248859
 
如何办理纽约州立大学石溪分校毕业证学位证书
 如何办理纽约州立大学石溪分校毕业证学位证书 如何办理纽约州立大学石溪分校毕业证学位证书
如何办理纽约州立大学石溪分校毕业证学位证书Fir sss
 
如何办理(Rice毕业证书)莱斯大学毕业证学位证书
如何办理(Rice毕业证书)莱斯大学毕业证学位证书如何办理(Rice毕业证书)莱斯大学毕业证学位证书
如何办理(Rice毕业证书)莱斯大学毕业证学位证书SD DS
 
Difference between LLP, Partnership, and Company
Difference between LLP, Partnership, and CompanyDifference between LLP, Partnership, and Company
Difference between LLP, Partnership, and Companyaneesashraf6
 
Test Identification Parade & Dying Declaration.pptx
Test Identification Parade & Dying Declaration.pptxTest Identification Parade & Dying Declaration.pptx
Test Identification Parade & Dying Declaration.pptxsrikarna235
 
Group 2 Marlaw Definition of Bill of Lading .pptx
Group 2 Marlaw Definition of Bill of Lading .pptxGroup 2 Marlaw Definition of Bill of Lading .pptx
Group 2 Marlaw Definition of Bill of Lading .pptxjohnpazperpetua10
 
Special Accounting Areas - Hire purchase agreement
Special Accounting Areas - Hire purchase agreementSpecial Accounting Areas - Hire purchase agreement
Special Accounting Areas - Hire purchase agreementShubhiSharma858417
 
The Patents Act 1970 Notes For College .pptx
The Patents Act 1970 Notes For College .pptxThe Patents Act 1970 Notes For College .pptx
The Patents Act 1970 Notes For College .pptxAdityasinhRana4
 
Rights of under-trial Prisoners in India
Rights of under-trial Prisoners in IndiaRights of under-trial Prisoners in India
Rights of under-trial Prisoners in IndiaAbheet Mangleek
 
Good Governance Practices for protection of Human Rights (Discuss Transparen...
Good Governance Practices for protection  of Human Rights (Discuss Transparen...Good Governance Practices for protection  of Human Rights (Discuss Transparen...
Good Governance Practices for protection of Human Rights (Discuss Transparen...shubhuc963
 
如何办理(UoM毕业证书)曼彻斯特大学毕业证学位证书
如何办理(UoM毕业证书)曼彻斯特大学毕业证学位证书如何办理(UoM毕业证书)曼彻斯特大学毕业证学位证书
如何办理(UoM毕业证书)曼彻斯特大学毕业证学位证书srst S
 
如何办理(UNK毕业证书)内布拉斯加大学卡尼尔分校毕业证学位证书
如何办理(UNK毕业证书)内布拉斯加大学卡尼尔分校毕业证学位证书如何办理(UNK毕业证书)内布拉斯加大学卡尼尔分校毕业证学位证书
如何办理(UNK毕业证书)内布拉斯加大学卡尼尔分校毕业证学位证书SD DS
 
定制(BU文凭证书)美国波士顿大学毕业证成绩单原版一比一
定制(BU文凭证书)美国波士顿大学毕业证成绩单原版一比一定制(BU文凭证书)美国波士顿大学毕业证成绩单原版一比一
定制(BU文凭证书)美国波士顿大学毕业证成绩单原版一比一st Las
 
昆士兰科技大学毕业证学位证成绩单-补办步骤澳洲毕业证书
昆士兰科技大学毕业证学位证成绩单-补办步骤澳洲毕业证书昆士兰科技大学毕业证学位证成绩单-补办步骤澳洲毕业证书
昆士兰科技大学毕业证学位证成绩单-补办步骤澳洲毕业证书1k98h0e1
 
Key Factors That Influence Property Tax Rates
Key Factors That Influence Property Tax RatesKey Factors That Influence Property Tax Rates
Key Factors That Influence Property Tax RatesHome Tax Saver
 

Último (20)

如何办理(Curtin毕业证书)科廷科技大学毕业证学位证书
如何办理(Curtin毕业证书)科廷科技大学毕业证学位证书如何办理(Curtin毕业证书)科廷科技大学毕业证学位证书
如何办理(Curtin毕业证书)科廷科技大学毕业证学位证书
 
如何办理威斯康星大学密尔沃基分校毕业证学位证书
 如何办理威斯康星大学密尔沃基分校毕业证学位证书 如何办理威斯康星大学密尔沃基分校毕业证学位证书
如何办理威斯康星大学密尔沃基分校毕业证学位证书
 
如何办理佛蒙特大学毕业证学位证书
 如何办理佛蒙特大学毕业证学位证书 如何办理佛蒙特大学毕业证学位证书
如何办理佛蒙特大学毕业证学位证书
 
如何办理(GWU毕业证书)乔治华盛顿大学毕业证学位证书
如何办理(GWU毕业证书)乔治华盛顿大学毕业证学位证书如何办理(GWU毕业证书)乔治华盛顿大学毕业证学位证书
如何办理(GWU毕业证书)乔治华盛顿大学毕业证学位证书
 
POLICE ACT, 1861 the details about police system.pptx
POLICE ACT, 1861 the details about police system.pptxPOLICE ACT, 1861 the details about police system.pptx
POLICE ACT, 1861 the details about police system.pptx
 
如何办理纽约州立大学石溪分校毕业证学位证书
 如何办理纽约州立大学石溪分校毕业证学位证书 如何办理纽约州立大学石溪分校毕业证学位证书
如何办理纽约州立大学石溪分校毕业证学位证书
 
如何办理(Rice毕业证书)莱斯大学毕业证学位证书
如何办理(Rice毕业证书)莱斯大学毕业证学位证书如何办理(Rice毕业证书)莱斯大学毕业证学位证书
如何办理(Rice毕业证书)莱斯大学毕业证学位证书
 
Difference between LLP, Partnership, and Company
Difference between LLP, Partnership, and CompanyDifference between LLP, Partnership, and Company
Difference between LLP, Partnership, and Company
 
Test Identification Parade & Dying Declaration.pptx
Test Identification Parade & Dying Declaration.pptxTest Identification Parade & Dying Declaration.pptx
Test Identification Parade & Dying Declaration.pptx
 
Group 2 Marlaw Definition of Bill of Lading .pptx
Group 2 Marlaw Definition of Bill of Lading .pptxGroup 2 Marlaw Definition of Bill of Lading .pptx
Group 2 Marlaw Definition of Bill of Lading .pptx
 
Special Accounting Areas - Hire purchase agreement
Special Accounting Areas - Hire purchase agreementSpecial Accounting Areas - Hire purchase agreement
Special Accounting Areas - Hire purchase agreement
 
The Patents Act 1970 Notes For College .pptx
The Patents Act 1970 Notes For College .pptxThe Patents Act 1970 Notes For College .pptx
The Patents Act 1970 Notes For College .pptx
 
Rights of under-trial Prisoners in India
Rights of under-trial Prisoners in IndiaRights of under-trial Prisoners in India
Rights of under-trial Prisoners in India
 
Good Governance Practices for protection of Human Rights (Discuss Transparen...
Good Governance Practices for protection  of Human Rights (Discuss Transparen...Good Governance Practices for protection  of Human Rights (Discuss Transparen...
Good Governance Practices for protection of Human Rights (Discuss Transparen...
 
如何办理(UoM毕业证书)曼彻斯特大学毕业证学位证书
如何办理(UoM毕业证书)曼彻斯特大学毕业证学位证书如何办理(UoM毕业证书)曼彻斯特大学毕业证学位证书
如何办理(UoM毕业证书)曼彻斯特大学毕业证学位证书
 
如何办理(UNK毕业证书)内布拉斯加大学卡尼尔分校毕业证学位证书
如何办理(UNK毕业证书)内布拉斯加大学卡尼尔分校毕业证学位证书如何办理(UNK毕业证书)内布拉斯加大学卡尼尔分校毕业证学位证书
如何办理(UNK毕业证书)内布拉斯加大学卡尼尔分校毕业证学位证书
 
young Call Girls in Pusa Road🔝 9953330565 🔝 escort Service
young Call Girls in  Pusa Road🔝 9953330565 🔝 escort Serviceyoung Call Girls in  Pusa Road🔝 9953330565 🔝 escort Service
young Call Girls in Pusa Road🔝 9953330565 🔝 escort Service
 
定制(BU文凭证书)美国波士顿大学毕业证成绩单原版一比一
定制(BU文凭证书)美国波士顿大学毕业证成绩单原版一比一定制(BU文凭证书)美国波士顿大学毕业证成绩单原版一比一
定制(BU文凭证书)美国波士顿大学毕业证成绩单原版一比一
 
昆士兰科技大学毕业证学位证成绩单-补办步骤澳洲毕业证书
昆士兰科技大学毕业证学位证成绩单-补办步骤澳洲毕业证书昆士兰科技大学毕业证学位证成绩单-补办步骤澳洲毕业证书
昆士兰科技大学毕业证学位证成绩单-补办步骤澳洲毕业证书
 
Key Factors That Influence Property Tax Rates
Key Factors That Influence Property Tax RatesKey Factors That Influence Property Tax Rates
Key Factors That Influence Property Tax Rates
 

Robotics law: Ada-Europe 2017 presentation

  • 1. Avv. Giovanni Battista Gallus LL.M. Ph.d. ISO 27001 Lead Auditor gallus@array.eu - gallus@gm-lex.eu The laws of robotics and autonomous vehicles may be much more than three, but don't panic... yet.
  • 2. @gbgallusAvv. Giovanni Battista Gallus ● A Robolaw for all robots? “Hard” laws and “soft” laws Overview of the Eu Resolution on robots Civil liabilities Autonomous veihicles Ethical issues
  • 3. @gbgallusAvv. Giovanni Battista Gallus ● The 2017 Vienna Biennale
  • 4. @gbgallusAvv. Giovanni Battista Gallus ● From 1980 Paramount Pictures (1980) © Paramount Pictures (1980)
  • 5. @gbgallusAvv. Giovanni Battista Gallus ● To 2017
  • 6. @gbgallusAvv. Giovanni Battista Gallus ● From Avatar
  • 7. @gbgallusAvv. Giovanni Battista Gallus ● To reality
  • 8. @gbgallusAvv. Giovanni Battista Gallus ● From the Jetsons © Hanna-Barbera
  • 9. @gbgallusAvv. Giovanni Battista Gallus ● To our homes and nursing homes
  • 10. @gbgallusAvv. Giovanni Battista Gallus ● Industry 4.0 By ChristophRoser at http://www.allaboutlean.com AllAboutLean.com Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=47640595
  • 11. @gbgallusAvv. Giovanni Battista Gallus ● All men robots are not created equal “The term robot is a-technical and encompasses a wide range of applications that have very little in common. For this very reason, it is impossible to develop a unitary body of rules applicable to all kinds of robotic applications, rather different rules should apply to different classes of devices” A. Bertolini Kitmondo PPM David Stillman
  • 12. @gbgallusAvv. Giovanni Battista Gallus ● “Smart Robot” definition The acquisition of autonomy through sensors and/or by exchanging data with its environment (inter- connectivity) and the trading and analysing of those data; self-learning from experience and by interaction (optional criterion); at least a minor physical support; the adaptation of its behaviour and actions to the environment; absence of life in the biological sense European Parliament resolution of 16 February 2017
  • 13. @gbgallusAvv. Giovanni Battista Gallus ● Question time! Which among the following legal statuses is closer, at the present time, to the one applicable to smart robots?
  • 14. @gbgallusAvv. Giovanni Battista Gallus ● Digital person? https://en.wikipedia.org/w/index.php?curid=12543502
  • 15. @gbgallusAvv. Giovanni Battista Gallus ● Slave?
  • 16. @gbgallusAvv. Giovanni Battista Gallus ● Pet/animal?
  • 17. @gbgallusAvv. Giovanni Battista Gallus ● Children/minor?
  • 18. @gbgallusAvv. Giovanni Battista Gallus ● Software?
  • 19. @gbgallusAvv. Giovanni Battista Gallus ● Tool (machine – car – manufactured object)?
  • 20. @gbgallusAvv. Giovanni Battista Gallus ● Hard laws Directive 85/374/EEC on liability for defective products Machinery Directive 2006/42/EC Council Directive 89/391/EEC of 12 June 1989on safety and health at work Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices EU Regulation 2016/679 (GDPR) Future EU Regulation on Drones (Notice of Proposed Amendment 2017-05 (A)- Introduction of a regulatory framework for the operation of drones) Convention on damage caused by foreign aircraft to third parties on the surface, 7 October 1952 Several Conventions regarding international civil liabilities (nuclear power; dangerous and noxious substances; oil pollution etc) The hardest part: all national laws regulating civil and criminal liability for harmful acts
  • 21. @gbgallusAvv. Giovanni Battista Gallus ● Soft Laws European Parliament resolution of 16 February 2017 with recommendations to the Commission on Civil Law Rules on Robotics Charter on Robotics – CODE OF ETHICAL CONDUCT FOR ROBOTICS ENGINEERS European Parliament’s resolution of 29 October 2015 on safe use of remotely piloted aircraft systems Opinion 1/2015 on Privacy and Data Protection Issues relating to the Utilisation of Drones 2015 Riga Declaration on remotely piloted aircraft (drones) All relevant technical standards (ISO/TC 299 -Robotics and others)
  • 24. @gbgallusAvv. Giovanni Battista Gallus ● We must accept liability The trend towards automation requires that those involved in the development and commercialisation of AI applications build in security and ethics at the outset, thereby recognizing that they must be prepared to accept legal liability for the quality of the technology they produce
  • 25. @gbgallusAvv. Giovanni Battista Gallus ● Control and verification Safeguards and the possibility of human control and verification need to be built into the process of automated and algorithmic decision-making
  • 26. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations - general Compulsory registration EU Agency for Robotics and Artificial Intelligence Attention to emotional connection between humans and robots
  • 27. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations - general “Kill switch” (sorry, HAL 9000) Cryteria https://commons.wikimedia.org/w/index.php? curid=11651154
  • 28. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – research & innovation Open science and responsible ethical innovation Interoperability between systems, devices and cloud services, based on security and privacy by design Open environment, from open standards and innovative licensing models, to open platforms and transparency, in order to avoid lock-in in proprietary systems that restrain interoperability
  • 29. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – Ethical principles I Human safety, health and security; freedom, privacy, integrity and dignity; self- determination and non-discrimination; personal data protection Strict and efficient guiding ethical framework for the development, design, production, use and modification of robots
  • 30. @gbgallusAvv. Giovanni Battista Gallus ● Whose ethics rule? The beginning of ‘policy’ for robotics is that agents have to act ethically right and maximise good consequences. The main value issues seem to concern honesty, esp. nondeception of users and customers. euRobotics Topics Group Ethical, Legal and Socio-economic Issues in Robotics
  • 31. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – Ethical principles II Transparency it must always be possible to reduce the AI system´s computations to a form comprehensible by humans; advanced robots should be equipped with a ‘black box’ which records data on every transaction carried out by the machine, including the logic that contributed to its decisions Guiding ethical framework should be based on the principles of beneficence, non-maleficence, autonomy and justice, on the principles and values enshrined in Article 2 of the Treaty on European Union and in the Charter of Fundamental Rights Special attention should be paid to robots that represent a significant threat to confidentiality owing to their placement in traditionally protected and private spheres
  • 32. @gbgallusAvv. Giovanni Battista Gallus ● And now... X RATED MATERIAL
  • 33. @gbgallusAvv. Giovanni Battista Gallus ● Roboprostitution © Warner Bros, 2001 © HBO 201
  • 34. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – IP & flow of data Intellectual Property Compliance with the principles of GDPR right to respect for private life and to the protection of personal data High level of security – Responsibility of designers of robotics and AI to develop products to be safe, secure and fit for purpose
  • 35. @gbgallusAvv. Giovanni Battista Gallus ● Data protection & robots https://www.flickr.com/photos/pallina60loon/
  • 36. @gbgallusAvv. Giovanni Battista Gallus ● Data protection by design Data protection by default Data Protection Impact assessment EU Regulation 2016/679 Necessity & proportionality Data protection by default Transparency Certification mechanism
  • 37. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – Standards safety and security International harmonisation of technical standards Lawful reverse-engineering and open standards Setting up of special technical committees, such as ISO/TC 299 Robotics Efficient strategy & monitoring mechanism for testing robots in real-life scenarios
  • 38. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – Autonomous means of transport Urgent need for global rules Drivers reaction time (human factor) Drones
  • 39. @gbgallusAvv. Giovanni Battista Gallus ● “A key finding is that it took 2-3 seconds for participants to ‘takeover’ manual controls and resume active driving after short periods of autonomous driving in urban environments” http://www.venturer -cars.com/trial-1- results/
  • 42. @gbgallusAvv. Giovanni Battista Gallus ● On the other side of the pond http://cyberlaw.stanford.edu/wiki/index.php/Automated_Driving:_Legislative_and_Regulatory_Action
  • 44. @gbgallusAvv. Giovanni Battista Gallus ● The (near) future for drones
  • 45. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – Care & medical Robots Human factor is still fundamental Proper training of surgeons Vital to respect the principle of the supervised autonomy of robots
  • 46. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – Human repair and enhancement Cyber physical systems Appropriately staffed committees on robot ethics, in hospitals & health care institutions Patching and updating of vital medical applications (also by independent trusted entities)
  • 47. @gbgallusAvv. Giovanni Battista Gallus ● Human enhancements & biorobotic devices
  • 48. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – Liability I European framework Smooth joint human / robot activities Legislative initiatives + soft laws No limitation on damages not to property, on the sole grounds that damage is caused by a non-human agent Strict liability or risk management approach? At least at the present stage the responsibility must lie with a human and not a robot
  • 49. @gbgallusAvv. Giovanni Battista Gallus ● The blame game: the monster or the creator? We need to retain the principle that humans are responsible for their actions, while robots and other machines are not euRobotics Topics Group Ethical, Legal and Socio-economic Issues in Robotics © Gruskoff/Venture Films 1974
  • 50. @gbgallusAvv. Giovanni Battista Gallus ● Recommendations – Liability II Programming vs training Liability should be proportional to the actual level of instructions given to the robot and of its degree of autonomy, so that the greater a robot's learning capability or autonomy, and the longer a robot's training, the greater the responsibility of its trainer should be Obligatory insurance scheme, taking into account all potential responsibilities in the chain Compensation fund for uninsured robots Limited liability for the manufacturer, the programmer, the owner or the user, if they contribute to a compensation fund, as well as if they jointly take out insurance to guarantee compensation where damage is caused by a robot
  • 51. @gbgallusAvv. Giovanni Battista Gallus ● The Jessica Rabbit principle, applied to robotics I'm not bad I’m just drawn build, programmed and trained that way... © Touchstone Pictures 1988
  • 52. @gbgallusAvv. Giovanni Battista Gallus ● The importance of training
  • 53. @gbgallusAvv. Giovanni Battista Gallus ● Asimov’s laws are not enough (sorry, Isaac...) A robot may not injure a human being or, through inaction, allow a human being to come to harm A robot must obey the orders given by human beings except where such orders would conflict with the first law A robot must protect its own existence as long as such protection does not conflict with the first or second laws
  • 54. @gbgallusAvv. Giovanni Battista Gallus ● Even if you change their order... https://xkcd.com/1613/
  • 55. @gbgallusAvv. Giovanni Battista Gallus ● The future of RoboLiability The inadequacies of existing rules suggests to radically replace a fault based rule with a risk- management approach (based on absolute liability rules), holding liable the party who is better placed to minimize the cost and acquire insurance euRobotics Topics Group Ethical, Legal and Socio-economic Issues in Robotics
  • 56. @gbgallusAvv. Giovanni Battista Gallus ● And finally, RoboArt Are we ready to accept the gift of the Evil deity?
  • 57. @gbgallusAvv. Giovanni Battista Gallus ● Is a robot luckier than me? Many thanks to Marco Scialdone for the idea
  • 58. @gbgallusAvv. Giovanni Battista Gallus ● In conclusion, we might ask Marvin... “I've calculated your chance of survival, but I don't think you'll like it” By Chris Favero - USA (Marvin the Paranoid Android #cosplay #C2E2 2015) via Wikimedia Commons
  • 59. @gbgallusAvv. Giovanni Battista Gallus ● The success of robots lies, quite literally, in your hands © Spyglass Entertainment, Universal Pictures, 2003
  • 61. @gbgallusAvv. Giovanni Battista Gallus ● Questions?Questions?
  • 62. @gbgallusAvv. Giovanni Battista Gallus ● Avv. Giovanni Battista Gallus Let’s continue the conversation gallus@array.eu @gbgallus Unless stated otherwise, all texts are distributed under a Creative Commons Attribution – non commercial – sharealike 3.0 Unported license