First published online April 17, 2009
Journal of Experimental Biology 212, 1307-1323 (2009)
Published by The Company of Biologists 2009
doi: 10.1242/jeb.025379
Wing and body motion during flight initiation in Drosophila revealed by automated visual tracking
Ebraheem I. Fontaine1,*,
Francisco Zabala2,
Michael H. Dickinson2 and
Joel W. Burdick1
1 Mechanical Engineering, California Institute of Technology, Pasadena, CA
91125, USA
2 Bioengineering, California Institute of Technology, Pasadena, CA 91125,
USA
View larger version (26K):
[in this window]
[in a new window]
|
Fig. 1. Experimental set-up for capturing 3D high-speed sequences of take-off. (A)
Arrangement of high-speed cameras and LED panels for back lighting. Individual
flies emerged from inside a Pasteur pipette. To elicit escape responses, a
stop was removed that released a black disk which fell toward the fly along a
brass rod. (B) Images of Drosophila synchronously captured from three
camera views. The high-speed video offers no strong visual features except the
silhouette. Even with three camera views, the complex wing beat motion is
difficult to capture due to low observability of the wings at certain postures
and motion out of the camera's depth of field.
|
|
View larger version (23K):
[in this window]
[in a new window]
|
Fig. 2. Segmentation procedure for Drosophila images. (A) Typical image of
Drosophila during flight initiation. (B) Image segmented into body
(green) and appendage (yellow) pixels. (C) Histogram of pixel intensities
(0–255) from A fitted with the sum of two Gaussians. The local minimum
of the Gaussian sum is chosen as the threshold to classify body and appendage
pixels. (D) Histogram of fly pixels calculated from background subtraction in
over 200 frames across three different camera views. The characteristic
bimodal shape is due to the opaque nature of the fly's body cuticle (lower
intensity peak) versus the more translucent appendages (higher
intensity peak).
|
|
View larger version (26K):
[in this window]
[in a new window]
|
Fig. 3. Generative model of fly body. (A) Triangle mesh of Drosophila
calculated from multiple calibrated images [courtesy W. Dickson
(Dickson et al., 2006 )]. (B)
Complete generative model constructed from the data points shown in A. The
model consists of three shape primitives: the body, head and wing. The
generative modeling approach offers a more compact representation of the shape
and motion of the fly than its triangle mesh counterpart. (C–E) Method
for constructing components of the body shape primitive. (C) The centerline
C(u) is a 3D B-spline curve with five control points (only
three of them are visible in the axes). The curve of the centerline lies
completely in the x–z plane. The width profile,
Rb(u), is revolved around C(u)
using an elliptical cross-section where the lateral direction is 20% wider
than the dorsal–ventral direction. (D) Complete head model of the fly
constructed identically to the method described for C using a different
profile curve and the x-axis as the centerline. (E) Outline profile
of the fly wing model constructed from a closed planar B-spline curve with 20
control points. For other definitions see `Table of abbreviations' in
text.
|
|
View larger version (17K):
[in this window]
[in a new window]
|
Fig. 4. Geometric generative model of Drosophila. Following aeronautical
convention, the rotations about the x-, y- and z-axes are
defined as roll, pitch and yaw, respectively. The downward pointing
z-axis is chosen so that positive pitch angles correspond to pitching
upwards. The model's kinematic chain includes a coordinate transformation from
the left wing frame to the body frame [given by (Qlw,
Tbw)] and a transformation from the body-fixed to the
world-fixed frame F, denoted by (Qb, T).
Analogous transformations exist for the right wing.
|
|
View larger version (28K):
[in this window]
[in a new window]
|
Fig. 5. Procedure for initializing automated tracker. (A) We used customized
software for manual digitizations of Drosophila body kinematics from
Card and Dickinson (Card and Dickinson,
2008). Points were clicked at the head, tail wing joint and wing
tip in multiple camera views to manually fit a geometric model to the images.
The manually estimated body pose was then used as an initial guess for the
automated algorithm. (B) At the initial frame, the profile of the body was
refined, while holding the pose parameters fixed, to more closely match the
actual shape of the fly by minimizing the error described in `Model
registration' (Materials and methods).
|
|
View larger version (26K):
[in this window]
[in a new window]
|
Fig. 6. Predictive component of tracker. (A) Motion model used to predict the
location of the fly in the next frame, pk+1,
given estimates from the previous frame, pk. Here, the
displayed motion during the upstroke of the wingbeat is exaggerated to
illustrate the concept. (B) Rotational motion of Drosophila left wing
motion during take-off (120 out of 380 samples shown). Motion is parameterized
by four quaternions which vary smoothly with time. The query of m=5
previously calculated poses is matched with position 106 of the prior
database. The relative motion to position 107 is used to calculate the
prediction.
|
|
View larger version (17K):
[in this window]
[in a new window]
|
Fig. 8. Implementation of roll constraint. Because the roll angle of the body is
unobservable from silhouette data in the images, a symmetry constraint within
the transverse plane of the body must be incorporated. (A) Unconstrained
estimate of the fly's pose; (top) projection of the model vectors into the
transverse plane, (bottom) 3D pose with transverse plane illustrated in gray.
This body configuration is highly unlikely given the biomechanics of
Drosophila. (B) Constrained estimate after rotating the body by angle
and updating joint angle vectors, Q.
|
|
View larger version (95K):
[in this window]
[in a new window]
|
Fig. 9. Performance metrics of tracker compared to human digitizer. Only body
kinematics are compared because human-tracked wing motion is unavailable.
(A,B) Two frames where a large discrepancy in roll angle was observed between
the human estimate and the algorithm. From visual inspection, the human
estimate in A appears more accurate than the algorithm's estimate, while in B
the algorithm appears to provide a better estimate and more accurate roll
angle. (C) Time trace of entire video sequence with frames A and B indicated.
Tracker values are solid lines, data from human are shown as open circles. (D)
Root mean square (r.m.s.) deviations between the human estimates and our
tracker for body orientation and translation. Each bar represents a separate
video sequence. The roll angle shows the greatest deviation, as expected due
to the symmetrical nature of the fly's body. Video sequence from C has the
largest deviation amongst all videos.
|
|
View larger version (96K):
[in this window]
[in a new window]
|
Fig. 12. Examples of gross errors in tracking algorithm. (A) During some escape
maneuvers, the fly's wing can undergo large deformations (shown in Aii) that
are not captured by our current rigid body model. In other camera views (Ai),
this deformation is not apparent. (B) Despite this large error, the algorithm
does not lose track and is able to continue successful estimation. (C) Another
failure mode of the tracking algorithm. The fly as seen in Ci is facing
towards the camera during an upstroke. The left wing (top) is in the proper
configuration, but the right wing (bottom) is flipped in the wrong orientation
(pronation instead of supination).
|
|
View larger version (55K):
[in this window]
[in a new window]
|
Fig. 13. Example of voluntary take-off. (A) 3D trajectory of fly during take-off
sequence. Wing kinematics for stroke cycles at the beginning, middle and end
of the sequence are shown to the right. The right wing is indicated in red,
the left in blue. (B) Time history of angles describing wing and body
kinematics throughout the take-off sequence. The wing angles were defined
relative to a plane through the wing hinges that is inclined 62 deg. from the
body axis (see Ai), which is the position of the mean stroke plane in hovering
flies. Sequence is shown in supplementary material Movie 1. See text for
details.
|
|
View larger version (67K):
[in this window]
[in a new window]
|
Fig. 17. Wing kinematics during voluntary take-offs. Each set of traces shows data
from eight sequences. Raw data are shown in black; averages are shown in color
for the right (red) and left (blue) wings. The standard deviation envelopes
are plotted as light red and blue areas in each trace. To align the different
flight sequences, the time axis of each trace was normalized so that the first
three strokes took the same amount of time. Averages and standard deviation
envelopes are shown only over this three-stroke interval.
|
|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati 
Twitter What's this?
© The Company of Biologists Ltd 2009
and detected edge locations
shown in red and yellow, respectively.
The edge locations are used to reconstruct the projection ray corresponding to
that point on the image silhouette. (B) The distance,
||x2||, from a point
x to a line L=(n, m) represented in Plüker
coordinates is calculated using the relation
||x2||=||xxn–m||.
This distance provides the error vector, x2, that is
minimized to make the model points match the projection rays of the image
silhouette. (C) In order to fit the geometric model to the images from
multiple camera views, we reconstruct the projection rays from the image
silhouette in each of the three camera views. The intersection of the
projection rays from each camera and the model are displayed individually for
illustration. We fit the model by minimizing the point to line distance across
all projection rays in all cameras.
, 
and 
) plots the kinematic angles for the
right (red) and left (blue) wing. Automatically tracked data are shown in
color; manual-digitized data are shown in black. The r.m.s. errors are 3.3
deg. (
