Understanding of light sensing organs in biology creates opportunities for the development of novel optic systems that cannot be available with existing technologies. The insect's eyes, i.e., compound eyes, are particularly notable for their exceptional interesting optical characteristics, such as wide fields of view and infinite depth-of-field. While the construction of man-made imaging systems with these characteristics is of interest due to potential for applications in micro air vehicles (MVAs) and clinical endoscopes, currently available devices offer only limited capabilities due to their use of compound lens systems in planar geometries. In this presentation, I discuss a complete set of materials, design layouts and integration schemes for digital cameras that mimic fully hemispherical compound eyes. Certain of the concepts extend recent advances in ‘stretchable electronics’ that provide previously unavailable options in design. I also discuss another interesting hierarchical micro- and nanostructures that can be found in eyes of night-active insects such as moth and mosquito. I present research trends on fabrication methods, optical characteristics, and various applications for artificial micro-/nanostructures that resemble ‘moth eye’ structure.
Presentation on how to chat with PDF using ChatGPT code interpreter
Biomimetics : Compound eyes
1. Biomimetics : Compound eyes
Young Min Song
Assistant Professor
Department of Electronic Engineering
Pusan National University
http://sites.google.com/site/youngminsong81
1
2. A Future for Electronics: Stretchy, Curvy, Bio-Integrated
Past
Current
Future
Industrial
Personal
Bio-Integr. / Bio-Insp.
PNAS 106, 10875 (2009).
Science 327, 1603 (2010).
8. Artificial (camera) vs. biological (human eye) imaging
• Small field of view, high resolution imaging
• Complex multi-component lens systems to achieve
focal imaging plane with small aberrations
• Planar CCD detectors
Double Gauss focusing lens
CCD detector
Light receptors
(hemispherical)
• High field of view, high resolution imaging
lens
• Simple lens system
• Curved (hemispherical) detectors (retina)
9. Imaging With a Single Lens
Planar Camera
Ray Tracing
40
20
lens
0
-20
-40
- Planar (commecial camera)
- Hemispherical (human eye)
- Parabola (ideal)
-60
-40
-20
0
Distance (mm)
20
40
10. Mimicking the human eye
form hemispherical PDMS transfer element
compressed
interconnect
~1 cm
~1 cm
radially stretch PDMS
form Si focal plane array
and release from underlying
wafer substrate
adhesive
cure adhesive; flop over substrate
compressible
interconnect
integrate optics &
interconnect to control
electronics to complete
the device
transfer focal plane array onto PDMS
Si device island
(photodetector
& pn diode)
hemispherical focal plane array
Nature 454, 748 (2008)
10
11. Mimicking the human eye
Eyeball camera mounted on PCB
Hemispherical detector
1 cm
1 cm
With single lens
Image
5
5 mm
10
12
0
5
5
0
5
Nature 454, 748 (2008)
(axis scale: mm)
Others: Hawk eye, zooming, etc.
11
13. Research Trends
Europe – CURVACE (Curved Artificial Compound Eyes)
: 2009-2013, Collaborative project (EPFL, ISF Fraunhofer, etc. )
the Future and Emerging Technologies (FET) programme within the
Seventh Framework Programme for Research of the European
Commission, under FETOpen grant number: 237940
Japan – TOMBO (thin observation
modules by bound optics)
: 2000-present, Osaka Univ., etc.
US – UCB, UIUC, Harvard Univ.,
Ohio Univ., etc.
: 2000~present, Optic components/systems
Science (2006)
13
14. Compound Eye Camera
Compound Eye
Challenge
Microlens
Rhabdom
Ommatidium
Optic Nerve
Screening
pigment
Requirement – Full set of microlens/photoreceptor units
with hemispherical geometry
14
15. Approach – Stretchable Optical/Electrical Subsystem
Optical subsystem
Elastomeric
microlens array
Combine,
stretch
Electrical subsystem
Stretchable
photodiode array
Hemispherical Compound
eye camera
Y. M. Song et al., Nature 497, 95 (2013) 15
17. Polymeric Microlens Arrays
Mechanical modeling
PDMS
membrane
FEM
Strain (%)
50
25
0
Optical design
Aluminum mold
Target FOV ~160° ∆Φ = 11°, ∆φ = 9.7°
L0
r = 0.4 mm, dpost = 0.8 mm, L0 = 0.92 mm
r
f
dpost
d
Mechanical design
h
t
f = 1.35 mm, h = 0.4 mm, t = 0.55 mm
d = 0.16 mm
18. Electrical Subsystem (Photodiode/Blocking Diode)
2nd
1st
Encapsulation
metal layer
2nd PI layer
metal layer
1st PI layer
N+ doped
Imaging pixel
P+ doped
N+ doped
Blocking diode
Photodiode
200 μm
19. Integration of Optical/Electrical Subsystem
Microlens array
Integrated form of lens/pixel arrays
(flat state)
Photodetector array
5 mm
19
25. Imaging with Wide Field of View
Object movement
Left (- 50°)
Center (0°)
Right (50°)
Laser spot illumination
0°
20°
40°
60°
80°
20°
40°
60°
80°
x z
y
Y. M. Song et al., Nature 497, 95 (2013) 25
26. Depth of field experiment
Camera
40°
- 40°
DA = 12 mm
DB = 12 mm
DA = 12 mm
DB = 22 mm
DA = 12 mm
DB = 32 mm
Y. M. Song et al., Nature 497, 95 (2013) 26
27. Applications and future works
Surveillance,
Military, etc.
http://paulmader.blogspot.com/
Novel imaging systems
- Apposition type
- Superposition type (refractive, reflective, neural)
- Polarization, color, etc.
27
33. Parabola shape SWSs
Approach – Lens-like shape transfer
Interference
lithography
PR patterns
Reflowed PR
patterns
Parabola-shaped
SWS
Photoresist
Substrate
Y. M. Song et al., Small 6, 984 (2010)
Period : 300nm
33
35. Optical device applications
Grating equation (reflection)
sin r , m
m
n
sin i
Photovoltaic devices
n = 1.0
Λ≈ λ
-1
Absorbing
materials
m = +1
n ~ 3.5
Back reflector
- Higher order diffraction
- Reflection minima
θr,m : m-th order reflected diffraction angle
θi : incidence angle
m : diffraction order
λ : incident wavelength
Λ : grating period
n : refractive index of incident medium
Light emitting
diodes/materials
Transparent
glasses/materials
n = 1.0
Λ≈ λ
-1
m = +1
0
Active medium
n ~ 1.5
n ~ 3.5
- Higher order diffraction
- Total internal reflection
Multiple internal reflection
35
36. Optical device applications
Transparent
glasses/materials
Light emitting
diodes/materials
Photovoltaic devices
800
2
Height
o
12.71%
400
13.31%
13.92%
300
1
0
-1
14.52%
200
100 nm,
300 nm,
500 nm,
99
12.10%
500
Z (um)
Height (nm)
i = 20
o
Transmittance (%)
Cell eff.
11.50%
600
100
i = 0
Cell efficiency
Transmittance
700
98
200 nm
400 nm
flat surface
97
96
95
94
93
92
91
100
100 200 300 400 500 600 700 800
Period
Period (nm)
-2
-0.5
0.0
0.5
-0.5
0.0
0.5
90
300
400
X (um)
Y. M. Song et al., Appl. Phys.
Lett. 97, 093110 (2010)
Y. M. Song et al., Opt.
Express 19, A157(2011)
600
700
Wavelength
Bare
glass
Y. M. Song et al., Opt. Lett.
35, 276 (2010)
Y. M. Song et al., Sol. Mat.
101, 73 (2012)
500
800
Wavelength (nm)
Oneside
SWS
Bothside
SWS
Y. M. Song et al., Opt.
Express 18, 13063 (2010)
K. Choi et al., Adv. Mater.
(2010)
Y. M. Song et al., ‘Antireflective nanostructures for optical device applications’
36
37. Thank you!
Nature
Bio-inspiration ‘Beyond biology’
Contact Information
Young Min Song
ysong@pusan.ac.kr
051-510-3120, 010-2992-8182
http://sites.google.com/site/youngminsong81
37
Notas do Editor
Future: mount/laminate electronics stretchy, curvy interface.
Tissue like form of electronics
- Epidermis for health monitoring
Brain machine interface
Inspiration from biology
As you know, recently, curved or flexible electronic devices are very big issues in electronics market. And, some of these products such as galaxy round or g-flex are now commercially available. On the other hand, in research fields, stretchable electronics is of great importance for various future applications. One of the examples is biomedical applications. If you want to attach the electronic devices to our body or to implant inside our body, then the device should be stretchable, because our body and organ has curvy and stretchy surfaces. So, many different types of stretchable devices, such as stretchable LEDs, batteries, and other electronic sensors are already reported.
As you know, recently, curved or flexible electronic devices are very big issues in electronics market. And, some of these products such as galaxy round or g-flex are now commercially available. On the other hand, in research fields, stretchable electronics is of great importance for various future applications. One of the examples is biomedical applications. If you want to attach the electronic devices to our body or to implant inside our body, then the device should be stretchable, because our body and organ has curvy and stretchy surfaces. So, many different types of stretchable devices, such as stretchable LEDs, batteries, and other electronic sensors are already reported.
As you know, recently, curved or flexible electronic devices are very big issues in electronics market. And, some of these products such as galaxy round or g-flex are now commercially available. On the other hand, in research fields, stretchable electronics is of great importance for various future applications. One of the examples is biomedical applications. If you want to attach the electronic devices to our body or to implant inside our body, then the device should be stretchable, because our body and organ has curvy and stretchy surfaces. So, many different types of stretchable devices, such as stretchable LEDs, batteries, and other electronic sensors are already reported.
In animal kingdom, there are around ten different types of eye and they can be largely categorized by two: one is the camera-type eye, which can be easily found in human, mamalian, bird, and fishes. It consists of single lens and the photo-receptor arrays at the backside of the eyeball.
The other one is compound eye, which consists of bundles of microlenses. And these can be easily found in insect’s eye or in crustaceans’ eye like shrimp or lobster something like that. And, more generally, most of the arthropods (in Korean, 절지동물) has this type of eyes, so sometimes it is called Arthropods eye. Because the arthropods possess 80% of all the existing animal species in the world, the study of this type of eye optics is pretty much valuable to develop new type of imaging systems and optic components.
In the previous slide, I mentioned that there are two major types of eye. In this camera type eye, it has a single lens and the photo-receptor arrays, as known as retina at the backside of the eyeball. On the other hand, in compound eye, it consists of an array of individual imaging unit, (in the biological system), as known as an ommatidium. It consists of microlens and photocensitive part known as rhabdom, and these units are distributed out over the surface of hemisphere. When you build an eye with this type of design, It allow the creature that simultaneously look in all directions at once. And it also provide the interesting capability (for imaging system) such as high sensitivity to motion and infinite depth of field.
So, our goal was to make the man-made version of this insect’s eye and to explore these interesting imaging characteristics.
In the previous slide, I mentioned that there are two major types of eye. In this camera type eye, it has a single lens and the photo-receptor arrays, as known as retina at the backside of the eyeball. On the other hand, in compound eye, it consists of an array of individual imaging unit, (in the biological system), as known as an ommatidium. It consists of microlens and photocensitive part known as rhabdom, and these units are distributed out over the surface of hemisphere. When you build an eye with this type of design, It allow the creature that simultaneously look in all directions at once. And it also provide the interesting capability (for imaging system) such as high sensitivity to motion and infinite depth of field.
So, our goal was to make the man-made version of this insect’s eye and to explore these interesting imaging characteristics.
So, Our team also introduced stretchable electronics for this compound eye systems.
As you can see here, first we prepared two different subsystems. In optical subsystem, it consists of elastomeric microlens arrays on a thin base membrane. This can be simply made by molding of polymeric materials like a PDMS. Electrical subsystem is very thin and stretchable silicon photodetector arrays, which is fabricated on a silicon on insulator wafer and it is transferred to thin polymeric membrane. And then, these two systems are bonded each other with a precise alignment process, after that, by using some mechanical tools, this integrated system can be deformed from flat to full hemispherical shape like this.
Interesting point is that, in the electrical subsystem, each imaging pixel is interconnected with adjacent pixels by using serpentine shaped metal lines (not a straight line). So, during the deformation process, this metal lines are stretched out like a spring which enables very large deformation of this system without any electrical and mechanical failure.
(So, the basic structure is here, but, for the proper operation, we need to consider some important factors in terms of optics, electronics as well as mechanics.)
(For the proper operation, we need to carefully design the geometries of optical system.) In optical design, there are two important parameters. First one is inter-ommatidial angle that can be defined by the angle between adjecent ommatidia. The other one is acceptance angle of each ommatidium, which corresponds to the focal length of microlens and the diameter of photodetector. The basic rule is that the acceptance angle should be smaller than the inter-ommatidial angle to prevent the overlapping of the visual field of each ommatidium. If not, the captured image will be pretty much blurred. So, to satisfy this condition,we adjusted several parameters, such as initial length between each ommatidia, radius of curvature of this hemisphere. Focal length of this microlens and lateral dimension of photodetector is also important to determine the acceptance angle. (Refractive index of this polymeric material is also important.)
So, based on this rule, we made a polymeric microlens arrays. This is the picture of the aluminum mold and the resulting PDMS microlens arrays. this aluminum mold with concave-shaped structures were made simply by micromachining.
In our current version, we aimed 160 degree field of view, in this case, inter-ommatidial angle is 11 degree, and the acceptance angle is around 10 degree, which satisfy the condition that I mentioned in a previous slide. And the corresponding design parameters are listed here.
Now we should prepare the electrical subsystem.
This figure shows an exploded schematic illustration of a single unit cell, which consists of thin doped silicon membrane, metal interconnection lines and polymeric layers. Each cell involves single photodiode as well as a blocking diode for preventing the electrical crosstalk from the adjacent pixels. And you can see here, each pixel is interconnected with other devices both in row and column with serpentine shaped metal lines encapsulated with polymeric materials.
Because we used thin silicon membrane of a silicon on insulator wafer, the total thickness of this electrical subsystem is only few micrometers, which allows very large deformation. I am not gonna go into details on fabrication, but if you have question, please ask me after this presentation.
Alright, now we have an optical and electrical subsystem. So, by using a transfer printing method with a precise alignment process, this photodetector array can be transferred to the backside of this microlens arrays. This is the integrated form of these two systems in a flat state. This is transparent while this color is black and brown, because it contains only very thin silicon membrane. (so there are no light absorptive medium at the background).
You can see here, each imaging pixel is positioned at the bottom center of the corresponding microlenses.
For the hemispherical deformation, the integrated system is sealed in a customized fluidic chamber that consists of inlet/outlet like this. So, if we fill the water inside the chamber and close the input port and extract the water from this output port, then due to the pressure change, this flat membrane deformed to hemispherical form like this. After that we can hold this shape by using some hemispherical supporting rod made of pdms (with some adhesive).
This picture shows compound eye camera that we made. It involves 180 ommatidia with full hemispherical geometry which enable an extremely wide field of view.
And as you can see here, some of the metal lines are largely stretched.
For the imaging test, we need to consider several other things. First of all, in a real insect’s eye, they have black pigments between each ommatidia to prevent the stray light from the adjacent ommatidia. (In a biological term, it is called screening pigment) So, we prepared similar stuffs, that is black matrix with perforated hole arrays and we stretched this sheet and wrapped on our compound eye camera. And also, we used black supporting rod instead of clear one to prevent the back scattered light (at these areas). And we used thin film contact pads to connect our camera to a printed circuit board. And, finally, this pcb is connected to a control box with a ribbon cable to record a data set for image construction.
Alright so, I want to briefly introduce the operating principle of our camera. This is central portion of our camera and this is an object with ‘+’ shaped line art pattern. As you know, each microlens produces a small image of the object at specific positions, but only if a portion of these image overlaps with the active area of photodiode, then it generates photocurrent like this. Because the current version of our camera has only limited number of ommatidia, this camera produce a sampled image like that. In order to improve the resolution, we used a scanning method with 10 x 10 moving steps, in x and y direction, as a result, we could obtain this kind of image. (Actually, all of these are simulation results and I will show you the experimental results later.)
Here is the measurement setup. This is our camera which is mounted on a rotation stage for scanning process. The light is illuminated from here with some specific mask pattern and it is projected to our camera.
These are the representative output images captured by our compound eye camera. These two are experimental results and these are modeling results. And these are original line art patterns that we used as a mask. As I mentioned before, In order to improve the resolution, I used a scanning method with 10 x 10 moving steps, in x and y direction, as a result, We could obtain this kind of image. And, the resulting images are rendered on a hemispherical plane and as you can see here, the shape and size of these images are quite well matched with the modeling results. Actually, this modeling is conducted by ray tracing method with some of basic parameters that we calculated in the previous slide.
(But, still there are some loss of resolution at several parts, which is caused by the parasitic scatting within the camera.)
One of the powerful mode of compound eye is the wide field of view. These images show an example of this characteristics. When you put the object at center, left and right with a 50 degree rotation angle, and take a picture, then you can see that all three resulting images show comparable clarity, without anomalous aberrations. And this laser spot illumination also shows the uniformity in sizes, shapes, and positions of these spots over the entire viewing angles. (from zero to 80 degrees)
Another important feature is infinite depth-of-field (due to the optic geometry), which means that everything is in focus independent the distance away from the camera.
For this experiment, We prepared two object with different patterns (triangle and circle) at different positions like this. And, if this object is fixed and this object moves away from the camera, then as expected, these circular pattern size decreases whereas triangle has same size. But, interesting point is that this image is still in focus.
So, again, we can say that this type of camera can simultaneously take the pictures of multiple objects at widely different angular positions and even different distances, without any tuning of optic systems.
Our current camera is inspired by this apposition compound eyes which can be found in daylight insects. On the other hand, in some night active insects, the structures are little bit different with this and each photorecepter receives light from so many microlenses. This type of compound eye is more sensitive to light, while it has lower image resolution. This is really interesting point. And, another important point is that it has nanostructures on the surface of microlens arrays.
And, second challenge is that how we can apply these nanostructure on various optical devices. And what is the design rule for the specific applications. Based on the grating equation again, the m-th order diffraction angle is strongly related to the refractive index of incident medium. This is very important factor to determine the design rules. For example, in general, we know that smaller period is good for broadband antireflection. But, in photovoltaic device or photodetector, it is not.
In case of solar cell or other light absorbing medium, by controlling the period, we can arbitarilly induce the first order transmitted diffractions while minimizing the surface reflection. In this case, this higher order diffraction will increase the light path length thereby will increase the absorption efficiency. This is quite different with the conventional grating structure, because they has high surface reflection.
This kind of interesting effects can also be found in LED and transparent glasses. In case of LEDs, the grating period should be much smaller than the optical wavelength, because the light is coming out from inside to outside and the refractive index of these materials are very high. In case of transparent glasses, the light is coming from outside, but it experiences multiple internal reflection, these structure should has smaller period than optical wavelength.
So based on the theoretical modeling and expectation, I fabricated solar cells, LEDs, and highly transparent glasses, with well designed nanostructures. And all of these experimental results showed remarkable enhancements in optical efficiency. I don’t wanna go into details, but I wanna mentioned that this interesting nanostructure can be applied to any type of optoelectronic devices and optic components.