1. Multimedia Systems - Class Project
Group 2
Spatio-Temporal Invariant Points in Videos
Members:
Priyatham Bollimpalli – 10010148
Pydi Peddigari Venkat Sai – 10010149
PVS Dileep – 10010180

2. The objective here is to find the spatio-temporal invariant points in a given input video. We
implement the following models on a set of contiguous frames of a video, called a scene.
We divide the problem into three cases, one with the background being fixed in a video and
the entire scene is not dynamic, second, background fixed and the entire scene is
reasonably dynamic, and final one with the background moving and the objects are also
moving. We examine those cases below:
Case1: When the Background is fixed and the entire scene is not dynamic
In this case, the background in the scene is fixed across several frames, while the foreground
objects can keep moving across the whole video, but they do not occupy the entire frame of
video with their movements, i.e., only some parts of the frame would be having movement,
while a decent part of the frame would remain static. The following procedure is followed to
detect the spatio-temporal interest points in the scene:
 Every scene is a collection of several frames. In this instance, we would consider a
scene of a video in which the background is constant, and in the foreground, there is
a ball which quickly moves underneath a wooden block which is constant in the
video. These are just a four of the several frames present in the video:
 The difference of the first two frames of the image would be computed. In the
resulted difference image, we would have several ranges of pixel values possible.

3. Hence, we would keep a grey threshold of say 0.2, and put all the locations of the
frame which are above that grey threshold level, to pixel value of that of white. We
also have an image which we will keep writing through across several iterations of
this method(we refer to that image as Rframe). Now, in the previously resulted
frame of difference, we find all the locations where the pixel value is white, and fill
out all those corresponding locations on the Rframe to black.
 We keep on repeating this process of finding the difference between two successive
frames of the scene, thresholding the difference to get some pixel values which are
white, and filling out all the locations on Rframe where there is a white pixel to
black.
For example, the difference of the frames at several instants are as follows:
 Then, the final Rframe produced will be as follows:
The portions which are black in this image would depict those points where a
temporal invariance is not possible, as the objects keep moving in those areas. The
portions which are white would indicate those points which are constant throughout
the period of the video. These would be our points of interest for application of SIFT
on those points.

4.  On the final go, we would apply SIFT on the original frames of the scene, and find all
those points resulted from SIFT, and consider such a point among them to be our
interest point in this case only if that point is among the white portions of the
Rframe, i.e., we would only want the interest points which coincide with the white
portions of the Rframe.
Hence, the resulted interest points finally would form our Spatio temporal invariant
points in this case.
The above algorithm is also run on other videos as follows:
 Some of the several frames in the original video :
 Some among the resulted differences in the frames at several instants are:

5.  Then the Rframe resulted is as follows:
 Now, we find the interest points at those locations, which are among the whie
portions of the Rframe, since, only those would be the temporal invariant parts of
the image. Hence, the interest points of the scene resulted at some of the several
instants are as follows:

6. So, the above points marked with green would represent the spatio-temporal
interest points of the scene, at some of the instants among the whole.
 It is run on another video as follows:
 The Rframe resulted is :
 Interest points resulted at several frames are :

7. Case2: When the Background is fixed and the entire scene is reasonably
dynamic
In this case, the background in the scene is fixed across several frames, while the foreground
objects can keep moving across the whole video, and they do occupy the most of the frame
of video with their reasonable movements. In this case, the following procedure is followed:
 Consider a scene as follows:
 Now, if Case-1 was used here, the Rframe resulted would be as follows:

8. Hence, if case-1 is used here, we can observe that most of the region is blacked out
since the objects motion is present almost over the entire image, and hence, we lose
some of the possible interest points.
 Hence, we adopt the following method now. This method would use automatic
detection and motion-based tracking of moving objects in a video. This problem can
be seen as:
o detecting moving objects in each frame
o associating the detections corresponding to the same object over time
 The association of detections to the same object is based solely on motion. The
motion of each track is estimated by a Kalman filter. The filter is used to predict the
track's location in each frame, and determine the likelihood of each detection being
assigned to each track.
 In any given frame, some detections may be assigned to tracks, while other
detections and tracks may remain unassigned. The assigned tracks are updated using
the corresponding detections. The unassigned tracks are marked invisible. Each track
keeps count of the number of consecutive frames, where it remained unassigned. If
the count exceeds a specified threshold, the example assumes that the object left
the field of view and it deletes the track.
 So in the process, a frame is read, objects are detected with their centroids and
bounding boxes, and motion segmentation using the foreground detector. Next, the
detections are assigned to tracks. Then, the assigned tracks would be update and the
unassigned tracks would be updated by marking them invisible, and the lost tracks
would be deleted.
 The following are the results of the tracked objects at some of the instants of the
scene:

9.  Now, we apply SIFT on the tracked objects at all instants of the video. Then, the
following would be the interest points produced at several instants of the video:
 Now, to gather further more interest points from the video, we can also combine the
interest points generated from case-1, and hence, the resulted interest points would
be:

10. So this would capture all the interest points possible, combining case-1 and case-2
Case3: When the background is moving and the objects are also moving
When the camera as well as well objects are moving, tracking the objects is a challenging
issue. There will be rarely anything invariant even in one particular scene. This is still an
open research problem and some heuristic methods are successful. The video has a moving
car with a moving camera. Some of the frames are given below.

11. One heuristic method which we found on the web to solve it is given below.
Note that segmenting the car frame differentiation (background subtraction) won’t work
because the camera is also moving henceforth the background is also moving. Hence normal
prediction algorithms would fail in this case. To tackle this issue, Optical flow is used where
the scene and the car have different directions of flow. In the below figures, the red points
denote the optical flow of the background and the green points denote the points having an
optical flow opposite to the red points. Note that the green points are able to track the car
present in the entire video and hence locate the green points which are spatio-temporally
invariant.
Now, combination of the points obtained here with those obtained in case-1 and case-2 (it is
less likely that any points would be there), gives the total points possible.

12. Conclusion & Future Work:
In this report we have tried to perform three kinds of techniques on any given scene in a
video – when the background is almost stationary and the scene is not dynamic, when the
background is stationary and the scene is dynamic and the background and the scene
bother are dynamic. Combination of points obtained from all the three methods gives
maximum possible spatio-temporal invariant points.
There is lot of scope for future work in this area and we wish to pursue it further. The
following are the issues involved.
 In case-1, the level at which thresholding is done defines the extent/degree to which
motion of the object is considered. Lower the threshold, greater the impact of
motion. This entirely depends on the video i.e. if a video has lot of illumination and
contrast changes between the frames, then the difference of the frames would give
many false contours. In this case considering higher value of threshoding is desirable.
In some other cases the frame rate would be too high due to which almost negligible
amount of motion would be captured between the frames. In this case lower value
for thresholding is desirable. Hence developing an optimal threshold value
automatically by taking the video quality, frame rate into account is one area of
future work.
 In case-3, the above example gives a possible approach to solution. It works because
the camera is also continuously moving along with the car and getting the points
with opposite optical flow works. But if the camera remains stationary for some time
and moves again suddenly, we need a separate system to first track the background
movement and that in combination of foreground motion can be used to find out
the points on the object. Many object tracking methods exist and many are still
pursed since this is a very active area of research. It is likely that exploring into this
area would give a generic algorithm for obtaining an object and the points which are
consistent on it throughout the scene.
 Case-2 considers the background to be stationary and the motion of the objects to
be uniform. Modifying the parameters for the detection, assignment, and deletion
steps of the trackers according to the video may be done. The tracking in this
example was solely based on motion with the assumption that all objects move in a
straight line with constant speed. When the motion of an object significantly
deviates from this model, the example may produce tracking errors.
The likelihood of tracking errors can be reduced by using a more complex motion
model, such as constant acceleration, or by using multiple Kalman filters for every
object. Also, you can incorporate other cues for associating detections over time,
such as size, shape, and colour.
References
1. http://www.mathworks.in/help/vision/examples/motion-based-multiple-object-
tracking.html
2. http://www.youtube.com/watch?v=MOaKnCSejXQ