QUICK SEARCH:   [advanced] Author: Keyword(s):
Home     Help     Feedback     Subscriptions     Archive     Search     Table of Contents

First published online April 17, 2009
Journal of Experimental Biology 212, 1324-1335 (2009)
Published by The Company of Biologists 2009
doi: 10.1242/jeb.025502

This Article Summary Figures Only Full Text (PDF) Supplementary Material Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Ristroph, L. Articles by Cohen, I. Search for Related Content PubMed Articles by Ristroph, L. Articles by Cohen, I. Social Bookmarking                         What's this?

Automated hull reconstruction motion tracking (HRMT) applied to sideways maneuvers of free-flying insects

Leif Ristroph^1,*, Gordon J. Berman¹, Attila J. Bergou¹, Z. Jane Wang² and Itai Cohen¹

¹ Department of Physics, Cornell University, Ithaca, NY 14853, USA
² Department of Theoretical and Applied Mechanics, Cornell University, Ithaca, NY 14853, USA

^* Author for correspondence (e-mail: lgr24{at}cornell.edu)

Accepted 17 January 2009

	Summary

TOP Summary INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION APPENDIX References

 
Flying insects perform aerial maneuvers through slight manipulations of their wing motions. Because such manipulations in wing kinematics are subtle, a reliable method is needed to properly discern consistent kinematic strategies used by the insect from inconsistent variations and measurement error. Here, we introduce a novel automated method that accurately extracts full, 3D body and wing kinematics from high-resolution films of free-flying insects. This method combines visual hull reconstruction, principal components analysis, and geometric information about the insect to recover time series data of positions and orientations. The technique has small, well-characterized errors of under 3 pixels for positions and 5 deg. for orientations. To show its utility, we apply this motion tracking to the flight of fruit flies, Drosophila melanogaster. We find that fruit flies generate sideways forces during some maneuvers and that strong lateral acceleration is associated with differences between the left and right wing angles of attack. Remarkably, this asymmetry can be induced by simply altering the relative timing of flips between the right and left wings, and we observe that fruit flies employ timing differences as high as 10% of a wing beat period while accelerating sideways at 40% g.

Key words: insect flight, motion tracking, aerodynamics, wing kinematics measurement, fruit fly

	INTRODUCTION

TOP Summary INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION APPENDIX References

 
The swimming of fish and the flight of insects are impressive feats of nature. These locomotor displays are the collective result of genetic, evolutionary, neurological, sensorial and biomechanical influences and, when quantified, provide a window into the inner workings of these animals. Behavioral studies generally follow one of two approaches (Stephens et al., 2008). In the first, organisms are put in artificial environments and given a small number of choices in response to a stimulus, sacrificing richness of behavior for added experimental control. In the second, the animal is observed `in the wild', in which case a broad spectrum of behavior can be observed but not easily characterized due to limited measurement capabilities. For investigations of insect flight, this dichotomy is exemplified by studies of wing motions in tethered and wild insects. Tethered flight allows detailed measurement of kinematics but does not allow the studying of maneuvers. Field studies investigate more complex behaviors but with limited measurement and control capabilities. What is missing is an approach that would marry the beneficial aspects of these methods and would allow recording of complex behaviors while giving full access to the locomotor metrics. Such an approach would significantly increase the range of behaviors that can be investigated quantitatively.

State-of-the-art approaches to capturing the motion of locomoting animals involve high-speed, 3D videography combined with digitization of the captured sequences (Lauder and Madden, 2008). Most current techniques for extracting 3D body and wing kinematics of flying animals rely on manual motion tracking. One approach involves positioning a computer model of an organism so that it overlays the image of the filmed organism (Fry et al., 2003; Fry et al., 2005; Liu and Sun, 2008). Another method requires tracking the position of representative marker features on the organism through time (Jensen, 1956; Nachtigall, 1966; Zanker, 1990; Hedrick et al., 2002; Wang et al., 2003; Combes and Daniel, 2003; Russell, 2004; Hedrick and Daniel, 2006). Unfortunately, these techniques demand significant human input, resulting in poorly characterized or uncharacterized errors, limited throughput and red-eyed researchers. More automated methods, similar to those developed for motion tracking of cockroaches and fish (Revzen et al., 2005; Fontaine et al., 2007), require the development of morphologically appropriate wing and body models when applied to flying animals (Fontaine, 2008; Fontaine et al., 2009). Thus, there remains a need for accurate, automated and versatile methods that do not require morphological inputs.

The study of insect flight in particular stands to benefit from high throughput and accurate tracking techniques. Asymmetries in flight kinematics appear to be quite subtle, even for wing motions that bring about extreme maneuvers. For instance, it has been reported that fruit flies execute rapid changes in yaw, or saccades, by inducing differences between the amplitude of the left and right wings of about 5 deg. and shifting the stroke plane by about 2 deg. (Fry et al., 2003). Further exploration of the myriad maneuvers performed by insects will require large data sets that allow for identification of slight kinematic manipulations. In addition to addressing maneuverability, such data would offer insight into the roles of aerodynamics, efficiency, control and stability in insect flight (Wang, 2005).

Here, we outline a novel approach to the motion capture of flying insects. Rather than restricting the flight behavior, we film the rich free-flight repertoire of insects. This, by necessity, sacrifices much of our control over the maneuvers the insects perform. However, by automating our apparatus and recording many such events, we can identify common strategies used in similar maneuvers. Most importantly, our experimental arrangement is designed to yield films that contain time-resolved, 3D information about the motion of the insect body and wings during flight. In this work, we focus on a novel motion tracking technique we term Hull Reconstruction Motion Tracking (HRMT). We demonstrate that subtle, yet statistically significant, differences in flight modes can be clearly discerned using this method. More specifically, we examine sideways flight of fruit flies, and show that the generation of lateral acceleration is associated with changes in the timing of the rapid flipping of the wings. Overall, this approach is a key step toward a quantitative description of the rich flight behavior of insects.

	MATERIALS AND METHODS

TOP Summary INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION APPENDIX References

 
Generally, an analysis of insect flight requires a method for recording the flight events and a method for recovering the flight kinematics. Here, we describe techniques that allow for the automation of both high-speed videography and motion tracking.

High-speed, 3D videography
We have assembled an automated, versatile system for capturing many video sequences of flying insects. The apparatus is composed of three high-speed cameras focused on a cubical filming volume contained within a large Plexiglas flight chamber (Fig. 1A). The cameras are orthogonally arranged using precision rails mounted on an optical table. We use Phantom v7.1 CMOS digital cameras (Vision Research, Wayne, NJ, USA) that are sensitive to visible light. We find that filming at 8000 frames s^–1 at a resolution of 512 pixelsx512 pixels is a suitable compromise in temporal and spatial resolution. At this rate, we capture about 30–35 wing orientations per wing stroke of the fruit fly (Drosophila melanogaster), which beats its wings approximately 250 times per second. The cameras are event triggered, as described below, and are synchronized electronically. In our experiments, the cameras automatically save the images on an internal memory buffer. Once a recording sequence is finished, the cameras dump the images onto an external computer hard drive and then become available for recording more flight events.

View larger version (11K): [in this window] [in a new window]   Fig. 1. An experimental assembly for filming free-flying insects. (A) Three orthogonal cameras C aim toward a focal volume in a flight chamber FC, magnifying the image with bellows B and zoom lens L. Opposite each camera is a film slide projector P that illuminates the chamber. (B) The cameras are triggered to begin filming when crossed laser beams are broken by the flying insect. A laser L emits a beam that diverges at a beam-splitter BS and is re-routed by mirrors M to intersect through the flight chamber. Beam expanders BE inflate the beam to the size of the focal volume, and photodiodes PD detect the beam breakage. Simultaneous breakage of the beams initiates filming via a modified Schmitt trigger switching circuit (not shown).

Because fruit flies are small, measuring about 3 mm in body length, we magnify with an optical bellows (Nikon PB-5, Nikon USA, Melville, NY, USA) and a zoom lens (Nikon Macro-Nikkor, 28–105 mm) attached to each camera, as shown in Fig. 1A. The bellows can be expanded or contracted to achieve varying magnification and thus accommodate different-sized filming volumes. For D. melanogaster, typical cubical volumes described in this paper measure 1.5 cm in side length. This filming arrangement insures that perspective distortion between the near and far portions of the chamber is less than 5%.

Achieving crisp images of fruit flies in flight requires short exposure times (<30 µs), high magnification and large depth-of-field (high f-stop) values. These requirements all reduce the light available for filming. To avoid heating the filming volume, we use three slide projectors (Kodak Ektagraphic series, Kodak, Rochester, NY, USA) that provide infrared-filtered intense white light. Each lamp is directed toward its opposing camera, as in Fig. 1A. Thus, our films consist of silhouettes or shadows of the flying insect (Fig. 2A).

View larger version (38K): [in this window] [in a new window]   Fig. 2. Aligned silhouettes are rendered by image processing and registration. (A) The three orthogonal cameras provide images of a fruit fly in flight (top). To obtain silhouettes from these raw images, a background picture is subtracted and the resulting image is thresholded (bottom). (B) Because the cameras are not perfectly aligned, the pixel coordinates in different views may not correspond to the same spatial coordinate. In order to register the images, we form a minimal bounding rectangle around the shadow in each view and then shift and scale the images such that the rectangle corners are consistent between views.

The beam expanders in our assembly allow us to match the triggering volume to the filming volume, thereby maximizing the number of captured flight sequences. This versatility also accommodates the filming of insects of varying sizes. We generally expand the beam to 1–2 cm in diameter. Since the fly body area is of the order of 2 mm², our circuit is designed to reliably trigger on beam intensity disturbances of only a few per cent.

In a typical experiment, we release between one and 20 flies in the filming chamber. When interested in flight statistics, we release up to 20 flies and film for up to 3 h. In these experiments we obtain up to 10 events per hour. The flight chamber measures 13 cm on each side, so the flies are more than 20 body lengths from the nearest wall, indicating that the walls have negligible influence on the aerodynamics.

Automated tracking of flight kinematics
In order to analyze the vast amount of data collected with our apparatus, we have developed a method for automatically extracting the wing and body positions from flight films. This method is accurate, fast, model independent and broadly applicable. Our tracking algorithm neatly divides into four steps: image processing and registration, hull reconstruction, `dissection' of the hull reconstruction into a body and two wings, and extraction of positions and orientations. We implement all stages using custom-written MATLAB code (available at http://cohengroup.ccmr.cornell.edu/).

Our hull reconstruction method requires crisp silhouettes of the flies and accurate registration of the pixels in the images. To achieve registration, we first precisely align the cameras by fine adjustment of translation stage mounts. This procedure positions the center of each camera view to within a few pixels of a common point in space and also establishes the global, orthogonal coordinate system employed throughout this work. The procedure achieves equal magnification to better than 1%, as measured by imaging a ruled microscope slide that also determines the pixel-to-distance conversion. Next, we use the images from each flight movie itself to more precisely adjust the alignment and magnification. To obtain image silhouettes, we first subtract a background image from each picture. The resulting image is thresholded so that the insect shadow appears black on an otherwise white background (Fig. 2A, bottom). In order to calibrate the pixels so that their coordinates are aligned and properly scaled, we use a registration algorithm. We first enclose the silhouette from each view in the minimal bounding rectangle (Fig. 2B). We then scale and translate the images so that the pixel coordinates of the bounding rectangle corners match. For example, to register the pixels along the vertical direction, we shift and scale images from one of the horizontal cameras such that its vertical coordinate is consistent with images from the second, reference horizontal camera. We vertically shift the image from the first view such that the top of its bounding rectangle has the same vertical coordinate value as that of the reference view. Then, we scale the image from the first view such that the bottom of its bounding rectangle has the same vertical coordinate as that of the reference view. The same procedure is used to register the other image coordinates. Typically we find that the images need to be scaled by less than 1% and shifted by about 5 pixels to achieve registration. To insure consistent registration for each movie, we find the average shift and scale values for the entire image sequence and apply these values to all images. The resulting thresholded and registered image sequences are fed into the hull reconstruction algorithm.

In the context of our experiments, the method of visual hull reconstruction (Baumgart, 1974) entails using the three sets of 2D silhouettes to construct a 3D shape. Specifically, our algorithm identifies volume pixels, or voxels, in 3D whose 2D projections map onto black pixels in all three images. More intuitively, this procedure is equivalent to the geometric exercise of placing the images on three adjacent sides of a rectangular prism and extending each shadow in a direction perpendicular to the image (Fig. 3A). Here, simple extension of each shadow is justified by the rather small perspective distortion. In this scenario, the hull volume corresponds to the intersection of the 3D extended shadows. An example of the resulting shape is shown Fig. 3B. This collection of voxels forms a convex volume that envelops the 3D shape corresponding to the real insect. We show that, by using three cameras to image the insect, we obtain a visual hull that is sufficiently close in shape to the real insect that wing and body positions and orientations can be extracted.

View larger version (23K): [in this window] [in a new window]   Fig. 3. Visual hull reconstruction forms a 3D shape that is consistent with the three silhouettes. Our implementation seeks 3D volume pixels, voxels, that project onto the silhouette in each view. Hull reconstruction is equivalent to the exercise of first forming extended 3D shadows from the silhouettes (A), and then finding the intersection in space of these extended shadows (B). The resulting object is the visual hull of the insect, the largest volume shape that is consistent with the three silhouettes. The hull data consist of an array of voxel coordinates.

Portions of the hull that correspond to the body, right wing and left wing form well-defined groups of voxels. To collect voxels that are near one another, we use a k-means clustering algorithm with a Euclidean distance metric (MATLAB, 2004; The Math Works Inc., Natick, MA, USA). We find that identifying four clusters (k=4) neatly isolates two separate groups of voxels corresponding to the left and right wings and two additional larger groups of voxels that correspond to the anterior and posterior of the insect body. These two larger clusters are merged to identify all the voxels corresponding to the body, and the smaller clusters correspond to the wings. In Fig. 4, the voxels corresponding to each of the body, right wing and left wing are shown in different colors in order to illustrate how well these groupings are identified.

View larger version (18K): [in this window] [in a new window]   Fig. 4. The body, right wing and left wing of the insect are identified by applying a clustering algorithm. The top view (A) and two side views (B and C), show that the right (red) and left (dark blue) wings are clearly distinguished from the body (light blue).

View larger version (24K): [in this window] [in a new window]   Fig. 5. The positions and orientations are extracted for each of the body, right wing and left wing. The centroid is defined to be the mean of the voxel coordinates for each respective grouping. (A,B) To identify the three Euler angles of the body, we define two vectors on the body. The first is the axial unit vector, Â, which is found by applying principal components analysis (PCA) to the body voxel coordinates and gives the yaw angle, , and the pitch angle, β. The second is the lateral unit vector, , that runs from the insect's right to left and is identified as the normal to the plane formed by the centroids of the head, thorax and abdomen clusters. (C) The roll angle, , is the angle between and the unit yaw vector, . (D) For each wing, the span vector, , is identified by PCA and gives the stroke angle, , and the stroke deviation angle, . (E) The chord vector, , is parallel to the longest diagonal of the parallelogram cross-section of the wing hull. (F) The wing pitch, , is the angle between and unit stroke vector, . For other definitions, see Table of abbreviations.

Because each wing is thin, rigid and often occluded in one camera view by the insect body, its visual hull resembles a parallelepiped whose long axis is parallel to the wing span vector, . To determine , we apply PCA to the wing hull voxels. This vector allows determination of the Euler angles for the stroke, , and deviation, (Fig. 5D). The hull cross-sections perpendicular to form parallelograms (Fig. 5E, right). The wing chord vector, , is parallel to the longer diagonal of the parallelograms. The third Euler angle for each wing is the pitch angle, , and is defined to be the angle between and the unit stroke vector, . To determine , hull voxels near the mid-span (within 2 voxel side lengths) are projected onto a plane normal to , and the chord is the vector connecting the two voxel projections having the greatest separation. The definitions of all wing orientation angles are detailed in Fig. 5F.

These procedures lead to a full kinematic description consisting of 18 coordinates: three centroid coordinates and three Euler angles each for the body, right wing and left wing. These coordinates are computed independently for each time step in the movie and checked visually for mistakes. Together, these techniques constitute a HRMT method for extracting 3D kinematics from several 2D views of a flying insect. While the data in this paper pertain to insect flight, this method is applicable with suitable modifications to a variety of 3D motion studies of other complex-shaped, moving objects in space.

Assessing errors of the HRMT method
Discerning subtle differences in flight modes requires clear knowledge of errors in the data recovery method. Such errors cannot be determined from the movies of insects alone, as the actual kinematics are not known beforehand. Instead, we estimate the measurement error by running HRMT on a computer-generated model insect and comparing the extracted positions and orientations of the body with those we impose. The model insect consists of five ellipsoids: three for the head, thorax and abdomen, and one for each wing (Fig. 6A). We orient the ellipsoids in a given configuration, use a ray-tracing algorithm to determine the three orthogonal shadows, and run our analysis routine to extract the positions and orientations of the body and wings. Compared with the model insect volume, the hull volume is larger and contains extra protrusions that vary in size and location for different insect orientations. These protrusions arise because of occluded regions that are blocked from the view of all three cameras. The protrusions ultimately cause errors in the recovered coordinates, and these errors depend on the orientation of the body and the positions and orientations of the wings. Thus, though validating HRMT using such simulated data does not account for image registration errors, it does account for occlusion defects, which appear to be the primary source of error for the method. However, determining the error dependence on all relevant coordinate variables is not feasible. Because fruit flies typically assume a limited set of orientations and use a typical wing stroke pattern for flight, we perform an analysis that determines errors for realistic insect configurations. Our synthetic data correspond to a fixed body and 34 wing configurations obtained by applying a manual tracking program to a single stroke from a movie of a hovering fruit fly. Our manual tracking software relies on overlaying images of a virtual fly and is similar to other implementations (Fry et al., 2003; Liu and Sun, 2008). We estimate errors for all wing positions within this stroke as well as the errors associated with viewing this stroke from different angles.

View larger version (49K): [in this window] [in a new window]   Fig. 6. A test of the automated tracking algorithm on a computer model of a flapping fly. (A) A morphologically appropriate model fly consists of five ellipsoids. Three ellipsoids form the head, thorax and abdomen of the body, and two flat plates represent the wings. The wings act as three degree-of-freedom hinges that rotate about a point on the surface of the thorax. (B) Measured flapping motions are imposed on the wings of this model fly, the shadows in each of three views are generated, and finally the tracking algorithm is run on these shadows. For this case, the body is held fixed at a typical orientation of (, β, )=(0, 59, 0) deg. (C) A comparison of the imposed body position (open circles) and the measured position (filled circles) for the centroid (x, y, z). A histogram of the residuals, measured value minus the actual value, is shown to the right for each coordinate. The reconstruction method measures the body centroid to within the voxel size of 2 pixels. (D) A similar comparison for the body orientation angles reveals an accurate recovery, with errors of a few degrees. (E) The right wing centroid is recovered to within 2 pixels. (F) The right wing orientation angles can be resolved to better than 5 deg. The left wing shows similar statistics.

To obtain measurement errors for a typical viewing configuration, we fix the virtual fly body in an orientation of (, β, )=(0, 59, 0) deg. and plot the imposed time series data (open circles) and measured values (filled circles) for body and wing positions and orientation angles (Fig. 6C–F). The errors for each variable are concisely displayed as a histogram of the residual, defined as the difference between the measured value and the imposed value. For both the body and wing centroids, errors are within the coarse-grained voxel size of 2 pixels. The body orientation is also accurately recovered, generally to within a few degrees. The wing orientation angles and associated residuals for the right wing are shown in Fig. 6F, and the errors for the left wing have similar statistics. Errors for the wing orientations are typically under 5 deg.

The time series and residual data of Fig. 6 reveal several features of the hull reconstruction method. Most of our measurements average over the voxels in the hull and thus result in subvoxel resolution. Also, the residuals are nearly always centered on zero, indicating that there are only small systematic deviations. Further, the residuals have standard deviations of less than 2 pixels in the positions and 4 deg. in the orientations.

Furthermore, we find that in nearly all the cases we have examined, the mean residuals remain under 3 pixels and under 5 deg., regardless of both wing position during the stroke and viewing configuration. To summarize the dependence on wing position during the stroke, we plot the residuals for , and as a function of stroke angle in the body frame of reference, _b, for 16 different viewing configurations (Fig. 7). The configurations range in from 0 to 45 deg., in β from 45 to 90 deg., and in from 0 to 60 deg. In total, this analysis comprises 544 different postures of the insect, and the use of a single wing stroke in the analysis is justified by the fact that the basic wing motion varies in subtle ways even during extreme maneuvers (Fry et al., 2003). The residuals show no obvious trend with _b and all have standard deviations of less than 5 deg.

View larger version (8K): [in this window] [in a new window]   Fig. 7. Errors in the wing angles are nearly independent of phase in the wing stroke. To arrive at the displayed mean and standard deviation of residuals, we orient the model fly, impose wing motions and measure errors in each wing orientation angle. Left and right wing residuals are similar, so we lump these data together. Residuals in each angle are plotted as a function of the imposed stroke angle. The stroke angle b is measured in the body frame such that b is approximately –90 deg. at the dorsal flip and b is approximately 50 deg. at the ventral flip.

To summarize the dependence on viewing configuration, we plot the residuals averaged over an entire stroke as a function of body orientation (, β, ) relative to the viewing configuration (Fig. 8). The residuals for the body and wing positions are all centered within two pixels of zero (Fig. 8A,C). With the exception of highly pitched (β 90 deg.) or highly rolled (>15 deg.) body orientations, the residuals for body and wing orientation angles are also centered within 2 deg. of zero. The increased errors at high β and are not expected to affect most aerodynamic analyses as the fluid force is generated almost entirely by the wings, whose motions are accurately resolved. Thus, because HRMT tracks the body, right wing and left wing independently in the lab frame of reference, the accuracy of coordinate extraction in any one component is independent of any other. Collectively, these results indicate that HRMT is an accurate method for the motion capture of flying insects.

View larger version (31K): [in this window] [in a new window]   Fig. 8. Errors in the recovered coordinates depend on the body orientation relative to the cameras. To reveal this dependence, we set the model fly of Fig. 6A in various orientations and measure the residuals in all coordinates. (A) Dependence of body position error on typical values of , β and . In each plot, the residuals of the three position variables x, y and z are plotted next to one another and are shaded differently. The errors show little dependence on orientation and are generally smaller than the voxel size of 2 pixels. (B) Errors in body orientation as a function of orientation. The body roll is more difficult to resolve than and β and becomes particularly error prone when the body is rolled considerably. As might be expected, heading is highly inaccurate when the insect is pitched up vertically near β=90 deg. (C) The right wing position is generally resolved to within 2 pixels. (D) The right wing orientation is accurate to within 3 deg. for most typical orientations of the body. For high pitch β and high roll , the wing pitch, , is not as well resolved. The left wing has similar error statistics.

	RESULTS

TOP Summary INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION APPENDIX References

Measurement of flight kinematics
In Fig. 9A,C, we show top views of trajectories of two maneuvers. In the first, a fly performs a `dodge' maneuver in which its yaw orientation remains nearly constant while it moves from one straight trajectory to another parallel trajectory (Fig. 9A). In the second trajectory, a fly performs a `sashay' maneuver in which it continuously reorients to face the inside of a turn (Fig. 9C). For this sashay, the body velocity is nearly perpendicular to the yaw direction of the insect. In both of these recorded maneuvers, we see that the insects undergo significant lateral acceleration; that is, the insects produce forces perpendicular to the body orientation in the xy-plane (Fig. 9B,D). See supplementary material for movies of these two maneuvers (Movies 1 and 2).

View larger version (13K): [in this window] [in a new window]   Fig. 9. Fruit flies undergo lateral acceleration during two maneuvers. Lateral acceleration is the horizontal component of acceleration that is perpendicular to the insect yaw direction. (A) Top view of a `dodge' maneuver. The fly yaw orientation is indicated by the black arrowheads, and the horizontal component of acceleration is shown as the red vectors. During the dodge, the insect moves from one forward trajectory to a nearly parallel forward trajectory. (B) To execute the dodge maneuver, the fly accelerates leftward and then rightward while moving forward. (C,D) In this `sashay' maneuver, the fly initially generates a large rightward acceleration that switches to become leftward near the end of the maneuver. Here, the lateral acceleration is as large as 0.4 g. Lateral acceleration is calculated from the body position and orientation data using a window-averaging method for differentiating noisy data (A.J.B., L.R., G.J.B., I.C. and Z.J.W., manuscript in preparation).

View larger version (54K): [in this window] [in a new window]   Fig. 10. Wing orientation angles for two maneuvers. In both cases, the lateral acceleration crosses through zero, and we display wing orientations near such a transition. For the dodge maneuver, the lateral acceleration is leftward before time t 0.075 s and rightward thereafter. For the sashay maneuver, the lateral acceleration is rightward before t 0.033 s and leftward thereafter. (A–C) The time course of b, and for the dodge. In order to facilitate comparison of the right and left wings, we have plotted the body frame stroke angle, b. (E–G) The time course of b, and for the sashay. In both maneuvers, the kinematic data reveal that the wing motion consists of a flipping motion of the wings superposed on the flapping back and forth. Asymmetries in the right (red) and left (blue) wing motions are associated with lateral acceleration. (D,G) These asymmetries lead to differences in the aerodynamic angle of attack, , the angle between the chord and the instantaneous wing velocity. This angle is calculated from the other wing orientation angles and has typical errors of 5–8 deg.

A lateral force generation mechanism
The generation of sideways forces can be rationalized by considering how differences in the motions of the right and left wings lead to asymmetric fluid forces. For example, in both maneuvers, when the fly generates rightward force, the left wing stroke deviation angle, _L, is greater than the right wing stroke deviation angle, _R. Likewise, for leftward accelerating flight, _R>L. These observations are consistent with the generation of lateral force by sideways tilting the wing stroke planes, in much the same way as a helicopter executes a banked turn. In essence, the lift force, which is normal to wing velocity, is redirected to have a horizontal component. To estimate the magnitude of the lateral acceleration from the redirected lift, we make the approximation that the vertical acceleration is about g and this is redirected by an angle |_R–_L|/2. For the dodge maneuver, this calculation reveals that the redirected lift force accounts for about 8% g, or about half of the lateral acceleration. For the sashay, a similar calculation shows that lift accounts for 30% g, or about 70% of the lateral acceleration. These estimates suggest that the mechanism of lateral force production is not entirely due to the redirected lift force on the wings. An additional mechanism for producing lateral forces may be associated with the consistent asymmetries in the wing angles of attack.

Generating lateral forces from asymmetries in can be understood by considering the time-lapsed top view images in Fig. 11A,B. In these images, the angle of attack is related to the projected area of each wing. As the wings are primarily moving in the horizontal plane, a large projected area in the top view corresponds to a low angle of attack and a small projected area is associated with a large angle of attack. The nearly horizontal, arc-like wing motion suggests that the drag forces, which act anti-parallel to wing velocity, have a significant lateral component. This is consistent with the fact that the wings sweep out a large arc in , and thus have a lateral component to their trajectories near stroke reversals. When the wings move symmetrically, these drag forces cancel out (Fig. 11A). For motions with asymmetric angles of attack near stroke reversal, the wing with the larger generates a larger drag force. This imbalance in drag forces induces a lateral acceleration (Fig. 11B).

View larger version (29K): [in this window] [in a new window]   Fig. 11. A drag-based mechanism of lateral force generation. Fruit flies primarily flap their wings back and forth with the upstroke and downstroke separated by rapid wing flips at the stroke reversal. (A) Four snapshots of the wing orientations near stroke reversal for flight with no lateral acceleration. When no lateral force is produced, the wing motions are nearly symmetrical between left and right wings. (B) When the insect is accelerating to its left, the right and left wings have different angles of attack, as evidenced by the different projected areas of the wings in this top view. (C) An idealized representation of the wing motion that generates leftward force. By selecting different angles of attack for the two wings near stroke reversal, asymmetric drag forces lead to a lateral force imbalance. (D) This asymmetry can be simply actuated by having the left wing rotate prior to the right, consistent with the timing difference observed in the angle of attack data for laterally accelerating fruit flies.

A schematic representation of the asymmetric wing motion is shown in Fig. 11C. The bottom image shows the fly at the beginning of a downstroke. As the wings begin to move forward, the projected area of the right wing is smaller indicating that _R is greater than _L. This asymmetry results in a net drag force that points to the left. Similarly, leftward drag forces are induced near the end of the downstroke where _L>_R, at the start of the upstroke where _R>_L, and at the end of the upstroke where _L>_R. Remarkably, as the schematic diagram in Fig. 11D shows, this seemingly complicated sequence of events can be generated simply by having identical curves for _L and _R that are shifted in time. In fact, we do observe timing differences in the measured curves for _L and _R for both the dodge and sashay maneuvers (Fig. 10D,H). These observations indicate that such time shifts in wing rotation are important for lateral force generation.

To quantify this idea, we determine the time shift by calculating the correlation integral I(t)=⁰_Tdt_R(t)_L(t–t) over a wing beat period, T, and choosing the t that maximizes I(t). We plot lateral acceleration a versus the normalized t/T in Fig. 12 and find that these variables are strongly correlated and that larger time shifts correspond to more extreme lateral accelerations. Included in the plot are individual wing strokes from the dodge and sashay maneuvers discussed above, as well as kinematic data from three additional captured sequences of sideways flight. In total, over 70 wing strokes and 45,000 individual kinematic measurements were extracted. Remarkably, we find a strong overlap in the data for these maneuvers. This indicates that the timing difference between right and left wing rotation may be a general feature in the mechanism of lateral force generation of fruit flies.

View larger version (8K): [in this window] [in a new window]   Fig. 12. Lateral acceleration is correlated with the timing difference in the angle of attack of the right and left wings. Each point represents a single wing stroke during the dodge (open circles), sashay (black circles), and three additional sideways flight maneuvers (red, blue and green circles). The timing difference, t, is the shift in time between the right and left wing angles of attack, R and L, and has been normalized by the flapping period, T. The value of the lateral acceleration, a, is the average during each wing stroke and has been normalized by gravitational acceleration, g.

	DISCUSSION

TOP Summary INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION APPENDIX References

 
We introduce here a new method called Hull Reconstruction Motion Tracking (HRMT) for automated, fast and accurate extraction of kinematic data from films of flying insects. In particular, we show that with appropriate morphological considerations, three camera views of each flight event are sufficient for extracting the full wing and body kinematics. Our implementation of HRMT is a unique form of motion tracking that combines and builds on image registration, hull reconstruction, clustering and several geometric and analytical techniques. The main source of error associated with the technique arises from regions outside the fly that are included in the hull because they are blocked from all camera views. Despite these occluded regions, we find that when we test the accuracy of this method by running the algorithm on synthetic data, errors are very small, of the order of 1–3 pixels for centroid positions and 1–5 deg. for the orientation angles.

The HRMT system has many directions for future improvements. For example, increasing the number of viewing directions will increase accuracy. Currently the analysis does not make use of intensity differences that can be used to differentiate between various components of an object. The analysis algorithms can be sped up through optimization, exporting portions of the code to programming languages which are faster than MATLAB, and making the code parallel. Also, our implementation does not resolve the roll of the body well, a notoriously difficult task due to the symmetry of the insect body. To better resolve roll, HRMT may be supplemented with marker-based feature tracking or with the imposition of a morphologically appropriate body model (Fontaine, 2008). Further, our current implementation of HRMT uses a simple image registration procedure that takes advantage of the orthogonal filming arrangement and low distortion due to perspective. Calibration of images from more general camera orientations and larger distortions due to perspective would require the use of more general photogrammetric techniques, such as the Camera Calibration Toolbox available for MATLAB (MATLAB, 2004). The small errors associated with HRMT for coarse-grained reconstruction also suggest that our method will remain accurate for arrangements that would require such modifications to the registration procedure. Finally, our current implementation does not quantify wing deformations. For D. melanogaster such deformations are small. We estimate that the wing camber is largest at stroke reversal and measures about 15%. Such deformations, however, are known to be significantly more prominent in larger insects and are important for understanding aeroelasticity (Combes and Daniel, 2003). In order to adapt HRMT to aeroelastic studies, the implementation described in this paper could be combined with other photogrammetric techniques in order to better resolve such deformations (Walker et al., 2008).

Overall, however, HRMT offers several improvements over present motion-capture techniques. Automation eliminates the need for a researcher to manually perform motion tracking. This allows errors to be characterized in a reliable way, and we show that these errors are small and generally have no systematic dependence on relevant variables. Also, our implementation is fast, easy to apply, and not memory intensive; it can be run on a commonly available personal computer. This allows for rapid extraction of flight data and determination of statistically significant trends. Because the kinematic data are measured entirely in the lab frame of reference, the recovered coordinates are also directly suited to aerodynamic analyses such as computational fluid flow solvers or numerical force models. Finally, HRMT is versatile and may be readily modified for other locomotion studies in which the motion of many components is important.

To illustrate the utility of this technique, we use the HRMT method to perform a comprehensive analysis of sideways flight maneuvers of fruit flies. Because our automated filming apparatus was used to capture hundreds of free-flight movies, we were able to then select five films showing unambiguous sideways flight. The HRMT method was used to automatically recover 45,000 kinematic measurements for over 70 wing strokes. By having access to all of these data, we show that flies are able to generate lateral forces in a manner that takes advantage of the unique features of flapping flight. In particular, we show that sideways-flying insects induce differences in the right and left wing angles of attack near stroke reversal. Based on these data, we propose a model for generating lateral forces by accounting for unbalanced drag due to the difference between the wing angles of attack. These differences lead to a `drag ratcheting' mechanism in which drag force asymmetries give directed sideways motion. Our simplified model predicts that asymmetries in the drag forces can be generated by having identical curves for _L and _R that are shifted in time relative to one another. This mechanism is consistent with measurements in dynamically scaled flapping wing experiments showing that drag is extremely sensitive to the timing of wing rotation at stroke reversal (Dickinson et al., 1999). To test this model, we use the HRMT method to analyze many fruit fly wing strokes associated with different values of lateral acceleration. We find that there is a strong correlation between the measured lateral acceleration and the measured time shift between the curves for _R and _L (Fig. 12). These observations indicate that free-flying fruit flies alter wing rotation timing during maneuvers. This manipulation may be actively controlled by steering muscles (Dickinson et al., 1993) or passively influenced by fluid, inertial or elastic forces (Bergou et al., 2007). Future studies may elucidate the fluid force generation mechanism in more detail, perhaps using dynamically scaled experiments (e.g. Dickinson et al., 1999), fluid force models (Berman and Wang, 2007) or computational fluid dynamics algorithms (Xu and Wang, 2008). Irrespective of the detailed force mechanism, our free-flight data suggest manipulation of wing rotation timing is a robust way to control forces during flapping flight. Exotic aerial maneuvers might be implemented in flapping, flying robots using such simple actuation strategies.

	APPENDIX

TOP Summary INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION APPENDIX References

 
Comparison of HRMT with a manual tracking method
Because we present HRMT as an automated alternative to manual tracking techniques, it is important to compare the two approaches. To make this comparison, we first designed a graphical user interface program in MATLAB that requires the user to position a model insect so that its shadows overlie the movie images of the actual insect. Our manual tracking program is similar to that of other groups (Fry et al., 2003; Liu and Sun, 2008). We then performed manual tracking for over 200 frames of a flight sequence. The particular flight sequence captures the dodge maneuver that is discussed in detail above. We also ran HRMT on the same frames and computed the differences between the coordinates extracted by each method (Fig. A1). In general, we find a strong similarity in the two methods, with all mean differences in position coordinates being less than 8 pixels and all mean differences in angular coordinates being less than 5 deg. There are small, but systematic differences in the two methods. The differences are probably due in part to inaccuracy in the morphology and connectivity of the model insect needed for manual tracking. There may also be additional occlusion errors in the HRMT method due to morphological differences between the real and model flies. Nonetheless, the similarity of the results obtained by HRMT and by the manual tracking program suggests that both methods are capturing the key features of the wing and body motion.

View larger version (19K): [in this window] [in a new window]   Fig. A1. Comparison of coordinates tracked by the HRMT method and a manual method. Over 200 frames from the dodge sequence are tracked by both methods, and the differences in the measured coordinates are plotted as a histogram. Comparisons are displayed for the body centroid position (A), the body orientation (B), the right wing centroid position (C), and the right wing orientation (D). The left wing shows similar statistics to the right wing. The mean differences in position coordinates are as high as 8 pixels. With the exception of the roll angle, orientation angles recovered by the two approaches are similar, with no mean difference greater than 4 deg.

LIST OF ABBREVIATIONS

a

lateral acceleration

Â

body axis unit vector

Âxy

body axis projection onto xy-plane

wing chord unit vector

g

gravitational acceleration

HRMT

hull reconstruction motion tracking

body lateral unit vector

PCA

principal components analysis

wing span unit vector

xy

wing span projection onto xy-plane

t

time

t

right/left wing rotation timing difference

T

wing beat period

wing velocity unit vector

(x,

y, z) position of body or wing centroid

wing aerodynamic angle of attack

β

body pitch angle

body pitch angle unit vector

wing pitch angle

wing stroke deviation angle

wing stroke deviation unit vector

body roll angle

wing stroke angle

wing stroke unit vector

b

wing stroke angle in body frame

body yaw angle

body yaw unit vector

	Footnotes

 
Supplementary material available online at http://jeb.biologists.org/cgi/content/full/212/9/1324/DC1

We thank Li Zhang for pointing us toward important literature on visual hull reconstruction, Witat Fakcharoenphal for help with the image processing and filming, and Michael Dickinson for advice on the experimental protocol. All authors made significant contributions to the development of this technique. We are grateful for support from the Cornell NSF-IGERT program in Nonlinear Systems and the Packard Foundation.

	References

TOP Summary INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION APPENDIX References

 

Baumgart, B. G. (1974). Geometric modeling for computer vision. PhD thesis, Stanford University, Palo Alto, CA, USA.

Bergou, A. J., Xu, S. and Wang, Z. J. (2007). Passive wing pitch reversal in insect flight. J. Fluid Mech. 591,321 -337.

Berman, G. J. and Wang, Z. J. (2007). Energy minimizing kinematics in hovering insect flight. J. Fluid Mech. 582,153 -168.[CrossRef]

Cheung, G. K. M. (2003). Visual hull construction, alignment and refinement for human kinematic modeling, motion tracking and rendering. PhD thesis, Carnegie-Mellon University, Pittsburgh, PA, USA.

Combes, S. A. and Daniel, T. L. (2003). Into thin air: contributions of aerodynamic and inertial-elastic forces to wing bending in the hawkmoth Manduca sexta. J. Exp. Biol. 206,2999 -3006.[Abstract/Free Full Text]

Dickinson, M. H., Lehmann, F. O. and Goetz, K. G. (1993). The active control of wing rotation by Drosophila. J. Exp. Biol. 182,173 -189.[Abstract]

Dickinson, M. H., Lehmann, F. O. and Sane, S. (1999). Wing rotation and the aerodynamic basis of insect flight. Science 284,1954 -1960.[Abstract/Free Full Text]

Ellington, C. P. (1984). The aerodynamics of hovering insect flight. III. Kinematics. Philos. Trans. R. Soc. Lond. B Biol. Sci. 305,41 -78.[Abstract/Free Full Text]

Ennos, A. R. (1989). The kinematics and aerodynamics of the free flight of some Diptera. J. Exp. Biol. 142,49 -85.[Abstract/Free Full Text]

Federal Aviation Administration (FAA) (2001). Rotorcraft Flying Handbook. Newcastle, WA: Aviation Supplies and Academics.

Fontaine, E. (2008). Automated visual tracking for behavioral analysis of biological model organisms. PhD thesis, California Institute of Technology, Pasadena, CA, USA.

Fontaine, E., Lentink, D., Kranenbarg, S., Mueller, U. K., van Leeuwen, J. L., Barr, A. H. and Burdick, J. W. (2008). Automated visual tracking for studying the ontogeny of zebrafish swimming. J. Exp. Biol. 211,1305 -1316.[Abstract/Free Full Text]

Fontaine, E. I., Zabala, F., Dickinson, M. H. and Burdick, J. W. (2009). Wing and body motion during flight initiation in Drosophila revealed by automated visual tracking. J. Exp. Biol. 212,1307 -1323.[Abstract/Free Full Text]

Fry, S. N., Sayaman, R. and Dickinson, M. H. (2003). The aerodynamics of free-flight maneuvers of Drosophila. Science 300,495 -498.[Abstract/Free Full Text]

Fry, S. N., Sayaman, R. and Dickinson, M. H. (2005). The aerodynamics of hovering flight of Drosophila. J. Exp. Biol. 208,2303 -2318.[Abstract/Free Full Text]

Hedrick, T. L. and Daniel, T. L. (2006). Inverse problems in the flight control of the hawkmoth Manduca sexta. J. Exp. Biol. 209,3114 -3130.[Abstract/Free Full Text]

Hedrick, T. L., Tobalske, B. W. and Biewener, A. A. (2002). Estimates of circulation and gait change based on a three-dimensional kinematic analysis of flight in cockatiels (Nymphicus hollandicus) and ringed turtle-doves (Streptopelia risoria). J. Exp. Biol. 205,1389 -1409.[Abstract/Free Full Text]

Jensen, M. (1956). Biology and physics of locust flight. III. The aerodynamics of locust flight. Philos. Trans. R. Soc. Lond. B Biol. Sci. 239,511 -552.[Abstract/Free Full Text]

Lauder, G. V. and Madden, P. G. A. (2008). Advances in comparative physiology from high-speed imaging of animal and fluid motion. Annu. Rev. Physiol. 70,143 -163.[CrossRef][Medline]

Liu, Y. and Sun, M. (2008). Wing kinematics measurement and aerodynamics of hovering droneflies. J. Exp. Biol. 211,2014 -2025.[Abstract/Free Full Text]

Nachtigall, W. (1966). Die Kinematik der Schlagfliigelbewegungen von Dipteren. Methodische und Analytische Grundlagen zur Biophysik des Insektenflugs. Z. Vgl. Physiol. 52,155 -211.[CrossRef]

Newman, D. J. S. (1982). The functional wing morphology of some Odonata. PhD thesis, University of Exeter, Exeter, UK.

Revzen, S., Koditschek, D. E. and Full, R. J. (2005). Testing feedforward control models in rapid running insects using large perturbations. Integr. Comp. Biol. 45, 1061.

Russell, D. B. (2004). Numerical and experimental investigations into the aerodynamics of dragonfly flight. PhD thesis, Cornell University, Ithaca, NY, USA.

Stephens, G. J., Johnson-Kerner, B., Bialek, W. and Ryu, W. S. (2008). Dimensionality and dynamics in the behavior of C. elegans. PLoS Comput. Biol. 4,e1000028 .[CrossRef][Medline]

Walker, S. M., Thomas, A. L. R. and Taylor, G. K. (2009). Photogrammetric reconstruction of high-resolution surface topographies and deformable wing kinematics of tethered locusts and free-flying hoverflies. J. R. Soc. Interface 6, 351-366.[Abstract/Free Full Text]

Wang, H., Zeng, L., Liu, H. and Yin, C. (2003). Measuring wing kinematics, flight trajectory and body attitude during forward flight and turning maneuvers in dragonflies. J. Exp. Biol. 206,745 -757.[Abstract/Free Full Text]

Wang, Z. J. (2005). Dissecting insect flight. Annu. Rev. Fluid Mech. 37,183 -210.[CrossRef]

Xu, S. and Wang, Z. J. (2008). A 3D immersed interface method for fluid-solid interaction. Comput. Methods Appl. Mech. Engrg. 197,2068 -2086.[CrossRef]

Zanker, J. M. (1990). The wing beat of Drosophila melanogaster. I. Kinematics. Philos. Trans. R. Soc. Lond. B Biol. Sci. 327,1 -18.[Abstract/Free Full Text]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				E. I. Fontaine, F. Zabala, M. H. Dickinson, and J. W. Burdick Wing and body motion during flight initiation in Drosophila revealed by automated visual tracking J. Exp. Biol., May 1, 2009; 212(9): 1307 - 1323. [Abstract] [Full Text] [PDF]

This Article Summary Figures Only Full Text (PDF) Supplementary Material Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Ristroph, L. Articles by Cohen, I. Search for Related Content PubMed Articles by Ristroph, L. Articles by Cohen, I. Social Bookmarking                         What's this?