The document summarizes two approaches to implementing foveated imaging in CMOS image sensors: (1) A pyramidal architecture with multiple rings of pixels having different integration times, allowing for dynamic range enhancement. (2) A universal multiresolution sensor using a 3T pixel design that allows pixels to be grouped and averaged, enabling adaptive resolution. Both designs aim to mimic the human retina and improve efficiency over traditional sensors. The pyramidal and multiresolution sensors were fabricated in 0.18um CMOS technology and are being tested for applications like video conferencing and industrial inspection.

1. Foveated Architectures for
CMOS Image Sensors
Fayçal Saffih1, Richard Hornsey2
1
Integrated Camera Group, Electrical & Computer Engineering
Department, University of Waterloo, ON, Canada
2
Centre for Vision Research, Department of Computer Science,
York University, Toronto, Ontario, Canada
Electronic Imaging 2004, San Jose

2. Outline
Introduction & Motivation
Motivation Implementation
Pyramidal CMOS Image Sensor
Applications
Universal Multiresolution CMOS Image Sensor
Applications
Final Conclusion.
Electronic Imaging 2004, San Jose

3. Introduction & Motivation
CMOS image sensor development levels:
 Device level: Optical charge devices such as photodiodes.
 Goal: Dynamic range enhancement, Fill factor increase…etc.
 Circuit Level: Amplifiers, buffers, ADC’s…etc.
 Goal: Efficient charge transfer, high speed acquisition, noise filtering…etc
 System Level: Signal processing blocks such as motion detection, image
processing blocks, …etc.
 Goal: Image information extraction (motion, segmentation…etc), performance
enhancement (data reduction, multiresolution, foveated vision…etc).
Motivation
 Imitation of biological visual systems such as Human Visual
System.
 Efficient Data Reduction.
 Efficient Image Information Transfer.
 Minimization of Power Consumption.
 Emphasize on the importance of architecture design in CMOS Imagers.
Philosophy: Image acquisition is a Vision needs vision
Electronic Imaging 2004, San Jose

4. Motivation Implementation
System level implementation
 The pixel structure is the 3T Active Pixel Sensor (APS).
 The acquisition system is a non-classical (non-
orthogonal) architecture.
 Solution: Pyramidal CMOS image sensor
Device level implementation
 The pixel structure is a non-classical (3T active pixel
sensor).
 The acquisition system is a classical orthogonal
architecture.
 Solution: Universal Multiresolution CMOS Image Sensor
Electronic Imaging 2004, San Jose

5. System Level approach
Pyramidal
CMOS Image Sensor
Electronic Imaging 2004, San Jose

6. Classical Acquisition System
The orthogonal acquisition
architecture along with raster
scanning suffers from the
following issues:
 1D sampling architecture.
 Unique integration time
 Different sampling speed
between horizontal and vertical
axis of the image. This leads to
anisotropic distribution of
motion blur, being higher in the
vertical axis than in the
horizontal.
Electronic Imaging 2004, San Jose

7. Human Fovea Architecture
Dynamic Range is higher in cones than Circular symmetry of the fovea photocells
rods. (rods and cones) has lead to symmetrical yet
HVS system DR (~106) is much larger co-centric image sampling.
than the photoreceptor DR (~102).
Electronic Imaging 2004, San Jose

8. Pyramidal Acquisition
Architecture
Electronic Imaging 2004, San Jose

9. Pyramidal Acquisition
Architecture
Floorplan of the imager is
composed of square
pixels (16µmx16µm)
orthogonally compacted.
Reset and select signals
are shared among each
ring
The output buses are
diagonals to dump the ring
output into 8 CDS block at
the base of the pyramid
Electronic Imaging 2004, San Jose

10. Bouncing Scanning Scheme
Estimated Integration time of a
ring r for inward Scanning:
 r +1 
Trin = 2  ∑ iT s + ( R − r )Tspl  + rT s
i = R 
 i → i −1
 

Estimated Integration time of a
ring r for outward Scanning:
 r −1 
Trout = 2  ∑ iTs + ( r − 1)Tspl  + rTs
 i =1 
 i → i +1
 

Electronic Imaging 2004, San Jose

11. Ring Integration Time
Estimated of all rings’ integration time for inward (ISc) and Rings’ RMS of inward (ISc) and bounced scanning
bounced scanning (BSc) at 40Fps. (BSc) at 8Fps under light intensity of 43.33uW
(current testing)
Electronic Imaging 2004, San Jose

12. Dynamic Range Fovea
Each ring in the pyramidal imager The resulting dynamic range
will have two different integration enhancement is made with two
time. Fusing their two output will acquired scenes. Thus, this type
enhance the ring’s dynamic range of dynamic range enhancement is
by:20log(Tint1/Tint2), where Tint1 ≥Tint2 called, intrascene dynamic range.
Electronic Imaging 2004, San Jose

13. Experimental Results
Inward and Bounced Images from
the Pyramid Imager at 14lux@29Fps
Fused Image of the inward Pyramidal CMOS Image Sensor
and bounced images Layout
Electronic Imaging 2004, San Jose

14. Applications
Applications that need optimal data transfer such as:
 Video-Phone
 Internet cameras
Applications that need foveated vision:
 IndustrialInspection
 Surveillance Cameras
 Low Vision Enhancement
 Consumers Cameras
Electronic Imaging 2004, San Jose

15. Disadvantages
Complex data structure for image re-construction
Mismatch between sampling capacitors between the
different pyramid clusters may create some undesired
artifacts.
The central diagonal in the pyramid is sampled by
capacitors with twice the capacity of their neighbor pixels.
 Although the above are real problems in our CMOS imager, they
are not limitations as they are all solvable problems.
Electronic Imaging 2004, San Jose

16. Conclusion
New architecture (hardware) and scanning scheme
(software) for CMOS imagers have been suggested.
The benefits of the new approach in image sampling are
due to:
 The 2D nature of the sampling rings
 Centricity of the sampling units has lead to foveated dynamic
range enhancement topology.
The difference in sampling architectures in CMOS
imagers greatly impacts their performances.
Electronic Imaging 2004, San Jose

17. Device Level approach
Universal
Multiresolution
CMOS Image Sensor
Electronic Imaging 2004, San Jose

18. Multiresolution CMOS Image Sensor
Motivation
 Only small parts of image are regions of interest.
Regions of low interest need to be sub-sampled or
averaged.
 With increasing image resolution the ultimate limitation
is image transfer especially for video broadcasting.
 Minimizing of power consumption requires a
minimization of power consumption of the imager OR
minimization of “data of interest” to be readout
 Programmability and expandability are the most
important feature of such an architecture in order for it
to span the largest field of applications:
Universality
Electronic Imaging 2004, San Jose

19. The Kernelling
 Just two clock cycles are enough for the kernel averaging: Fast Kernelling
Electronic Imaging 2004, San Jose

20. Building Block: The Pixel
Electronic Imaging 2004, San Jose

21. Multiresolution Decoder
for Row reset and select
CDS Block
CDS Block
CDS Block
CDS Block
CDS Block
CDS Block
CDS Block
CDS Block
CDS Block
The Architecture
CDS Block
CDS Block
Multiresolution Decoder for
CDS Block
CDS Block
Decoder for Column Select of CDS Block
CDS Block
Correlated Double Sampling (CDS) CDS Block
CDS Block
Row-Average Support & Column-Average
CDS Block
Multiresolution Decoder for
Row-Average and Sampling
Electronic Imaging 2004, San Jose
Image
Output

22. Application: Programmability
Fundamental Foveated Multiresolution Random Foveated Multiresolution
Multi-Foveated Image Sampling
Electronic Imaging 2004, San Jose

23. Application: Programmability
Fundamental Foveated Multiresolution Fundamental Foveated Multiresolution
With horizontally-rectangular kernels With vertically-rectangular kernels
Effect of kernel topology on spatial filtering
Electronic Imaging 2004, San Jose

24. Applications
Multiple Object Tracking
Industrial inspection
Image processing prototyping
Remote image acquisition requiring minimal
image transfer bandwidth.
Electronic Imaging 2004, San Jose

25. Final Conclusion
Two different approaches for implementing
foveated imaging have been suggested.
The two imagers were fabricated in dual voltage
1P6M standard 0.18µm CMOS technology. The
two imagers under test.
Parallelism, programmability and expandability
were the keys behind the proposed CMOS
image sensors’ architectures
Electronic Imaging 2004, San Jose

26. Acknowledgement
The authors are grateful to:
 Canadian Microelectronics Corporation.
 NSERC Canada.
 Betacom Inc.
Thank you !
Electronic Imaging 2004, San Jose