1. Computer Vision for Microscopes -1- Modelling Research Centre, 16th Dec 2005
Computer Vision for Microscopes
Microsystem & Machine Vision Laboratory
16th Dec 2005
MMVL http://vision.eng.shu.ac.uk/mmvl/ (vision/www)
MRC http://www.shu.ac.uk/research/meri/modelling_rc.html
MERI http://www.shu.ac.uk/research/meri/
SHU http://www.shu.ac.uk/
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
2. Computer Vision for Microscopes -2- Modelling Research Centre, 16th Dec 2005
MiCRoN
European Union IST project
Uppsala, Lausanne, St. Ingbert, Athens, Pisa, Barcelona, Karlsruhe
project goals
• Manipulate µm-sized
objects
• Closed-loop control of
robots
• 3D object recognition
and tracking
image acquisition
• micro-camera
http://wwwipr.ira.uka.de/~micron/ • microscope
http://www.cordis.lu/ist/
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
3. Computer Vision for Microscopes -3- Modelling Research Centre, 16th Dec 2005
MiCRoN
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
4. Computer Vision for Microscopes -4- Modelling Research Centre, 16th Dec 2005
MiCRoN hardware (i)
camera
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
5. Computer Vision for Microscopes -5- Modelling Research Centre, 16th Dec 2005
test hardware (i)
test environment
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
6. Computer Vision for Microscopes -6- Modelling Research Centre, 16th Dec 2005
MiCRoN hardware (ii)
first pictures
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
7. Computer Vision for Microscopes -7- Modelling Research Centre, 16th Dec 2005
test hardware (ii)
microscope environment
camera sensor actors
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
8. Computer Vision for Microscopes -8- Modelling Research Centre, 16th Dec 2005
software
Mimas computer vision toolkit
open source computer vision library
deliver solution ⇐ restrict vision domain
⇐ develop reusable toolset
• Sensor data: V4L, gstreamer-interface
• Segmentation: LSI-filters, morphology, disparity-estimator, DFT,
image-processing
• Feature-extraction: edges, corners, ...
• Feature-matching: optic flow, SVD-correspondence, correlation, champfer
matching, PGH, fast POL
• pose-estimation: particle filter, hough transform
http://vision.eng.shu.ac.uk/mediawiki/index.php/Mimas
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
9. Computer Vision for Microscopes -9- Modelling Research Centre, 16th Dec 2005
Canny-like edge detector
video edges
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
10. Computer Vision for Microscopes - 10 - Modelling Research Centre, 16th Dec 2005
geometric hashing
1988, Lamdan & Wolfson
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
11. Computer Vision for Microscopes - 11 - Modelling Research Centre, 16th Dec 2005
geometric hashing
preprocessing recognition
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
12. Computer Vision for Microscopes - 12 - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 2 DOF
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
13. Computer Vision for Microscopes - 13 - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 2 DOF
t1
t2 = s − m 1
VM ( , m)
1
0
s
0 5 5
1
3
m
m
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
14. Computer Vision for Microscopes - 14 (i) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
15. Computer Vision for Microscopes - 14 (ii) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features hash table
pose (3 DOF)
misses hits
80
120
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
16. Computer Vision for Microscopes - 14 (iii) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features hash table
pose (3 DOF)
misses hits
80 10
120 190
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
17. Computer Vision for Microscopes - 14 (iv) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features hash table
pose (3 DOF)
misses hits
80 10 50
120 190 150
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
18. Computer Vision for Microscopes - 14 (v) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features hash table
pose (3 DOF)
misses hits
80 10 50 200∗
120 190 150 0∗
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
19. Computer Vision for Microscopes - 15 - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
test on artificial sequence test on pipette
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
20. Computer Vision for Microscopes - 16 - Modelling Research Centre, 16th Dec 2005
geometric hashing
focus stacks as model database
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
21. Computer Vision for Microscopes - 17 - Modelling Research Centre, 16th Dec 2005
geometric hashing
3-D/4 DOF
test on povray sequence (time lapse)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
22. Computer Vision for Microscopes - 18 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
2001/2004, Greenspan, Shang & Jasiobedzki
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
23. Computer Vision for Microscopes - 19 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
2001/2004, Greenspan, Shang & Jasiobedzki
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
24. Computer Vision for Microscopes - 20 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
inverse transformation
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
25. Computer Vision for Microscopes - 21 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
houghspace for relative shifts
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
26. Computer Vision for Microscopes - 22 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
focus stacks for 4 DOF
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
27. Computer Vision for Microscopes - 23 - Modelling Research Centre, 16th Dec 2005
software architecture
application layers
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
28. Computer Vision for Microscopes - 24 - Modelling Research Centre, 16th Dec 2005
software architecture
UML static structure
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
29. Computer Vision for Microscopes - 25 - Modelling Research Centre, 16th Dec 2005
software architecture
UML collaboration diagram
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
30. Computer Vision for Microscopes - 26 - Modelling Research Centre, 16th Dec 2005
command-line tool
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
31. Computer Vision for Microscopes - 27 - Modelling Research Centre, 16th Dec 2005
tests
syringe-chip
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
32. Computer Vision for Microscopes - 28 - Modelling Research Centre, 16th Dec 2005
tests
gripper
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
33. Computer Vision for Microscopes - 29 - Modelling Research Centre, 16th Dec 2005
tests
gripper2
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
34. Computer Vision for Microscopes - 30 - Modelling Research Centre, 16th Dec 2005
tests
full automation
video video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
35. Computer Vision for Microscopes - 31 - Modelling Research Centre, 16th Dec 2005
MiCRoN setup (i)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
36. Computer Vision for Microscopes - 32 - Modelling Research Centre, 16th Dec 2005
MiCRoN setup (ii)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
37. Computer Vision for Microscopes - 33 - Modelling Research Centre, 16th Dec 2005
MiCRoN setup (iii)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
38. Computer Vision for Microscopes - 34 - Modelling Research Centre, 16th Dec 2005
MiCRoN setup (iv)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
39. Computer Vision for Microscopes - 35 - Modelling Research Centre, 16th Dec 2005
MiCRoN
assembly mockup
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
40. Computer Vision for Microscopes - 36 - Modelling Research Centre, 16th Dec 2005
MiCRoN
assembly mockup
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
41. Computer Vision for Microscopes - 37 - Modelling Research Centre, 16th Dec 2005
demonstration
macro-scale demonstration
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
42. Computer Vision for Microscopes - 38 - Modelling Research Centre, 16th Dec 2005
acknowledgements
• Jon Travis: technical suggestions, administrative support
• Fabio Caparrelli: camera drive/electronics/driver software, electronics setup,
PI driver software, management
• Balasundram Amavasai: proposal, computer vision suggestions, mimas
long-term strategy
• Arul Selvan: students supervision, telecentric optics, shift estimation
• Manuel Boissenin: realtime tracking of multiple micro-objects, parallelisation
• Jan Wedekind: software architecture and integration, realtime recognition of
multiple micro-objects
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
43. Computer Vision for Microscopes - 39 - Modelling Research Centre, 16th Dec 2005
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
44. Computer Vision for Microscopes - 40 - Modelling Research Centre, 16th Dec 2005
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
45. Computer Vision for Microscopes - 41 - Modelling Research Centre, 16th Dec 2005
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
46. Computer Vision for Microscopes - 42 - Modelling Research Centre, 16th Dec 2005
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory
47. Computer Vision for Microscopes - 43 - Modelling Research Centre, 16th Dec 2005
Happy Christmas!
http://vision.eng.shu.ac.uk/mediawiki/
Upcoming: MiCRoN public final report
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory