JEB desktop wallpaper calendar 2016

JEB desktop wallpaper calendar 2016

Summation of visual and mechanosensory feedback in Drosophila flight control
Alana Sherman, Michael H. Dickinson


The fruit fly Drosophila melanogaster relies on feedback from multiple sensory modalities to control flight maneuvers. Two sensory organs, the compound eyes and mechanosensory hindwings called halteres, are capable of encoding angular velocity of the body during flight. Although motor reflexes driven by the two modalities have been studied individually, little is known about how the two sensory feedback channels are integrated during flight. Using a specialized flight simulator we presented tethered flies with simultaneous visual and mechanosensory oscillations while measuring compensatory changes in stroke kinematics. By varying the relative amplitude, phase and axis of rotation of the visual and mechanical stimuli, we were able to determine the contribution of each sensory modality to the compensatory motor reflex. Our results show that over a wide range of experimental conditions sensory inputs from halteres and the visual system are combined in a weighted sum. Furthermore, the weighting structure places greater influence on feedback from the halteres than from the visual system.


When performing complex motor behaviors, animals often rely on feedback from several different sensory modalities. Although there has been extensive research on the sensory-motor reflexes involving a single sensory system (Dickinson, 1999; Fayyazuddin and Dickinson, 1999; Heide and Götz, 1996; Nalbach and Hengstenberg, 1994), less is known about the convergence of multiple modalities onto a common motor pathway. Insects, which rely on many sensory systems to control a small number of flight muscles, are excellent models for studying sensory fusion. For example, in flies, both the visual system and small gyroscopic hindwings called halteres can detect angular velocity of the body during flight (Nalbach, 1993; Pringle, 1948). Individually, either of these two sensory systems is capable of eliciting compensatory flight reflexes (Blondeau and Heisenberg, 1982; Dickinson, 1999; Sherman and Dickinson, 2003). How the fly integrates information from these two sensory systems to generate a coordinated response has yet to be determined.

A prominent feature of dipteran visual systems is a fast flicker fusion rate, which exceeds 250 Hz in the blowfly Calliphora (Autrum, 1958). By contrast, the compound eyes provide relatively crude spatial resolution. For instance, in Drosophila, the ommatidial spacing is approximately 5°, making it difficult for the flies to resolve small objects from a distance (Buchner, 1976). Each ommatidium contains a group of eight primary retinal photoreceptors that send axons into the brain (Hardie, 1985). The visual centers of the brain are composed of four optic ganglia: the lamina, the medulla, the lobula and the lobula plate (Strausfeld, 1976, 1984). Cells within each of the ganglia process visual motion in a hierarchy of progressing complexity. Due in large part to their relative accessibility to electrophysiology, the large identifiable cells of the lobula plate have been studied most extensively (Hausen, 1984). Physiological studies have revealed that certain lobula plate tangential cells are sensitive to the optic flow patterns resulting from rotation and translation of the fly in space (Krapp et al., 1998; Krapp and Hengstenberg, 1996). Behavioral studies have shown that full-field visual motion elicits compensatory responses, demonstrating a connection between the visual system and the flight motor (Blondeau and Heisenberg, 1982; Götz, 1964; Hengstenberg, 1991).

The flight motor also receives sensory feedback from the halteres. These tiny hindwings have been modified through evolution into sensory organs. The halteres, which beat anti-phase to the functional wings through a stroke plane of 180°, are positioned beneath the wings at a 30° angle from the transverse body axis (Fig. 1A; Drosophila: Dickinson, 1999; Calliphora: Nalbach, 1993). The base of the halteres is populated by hundreds of mechanoreceptors, consisting of campaniform sensilla and chordotonal organs, a subset of which are thought to encode Coriolis forces (Nalbach, 1993; Nalbach and Hengstenberg, 1994; Pringle, 1948). Coriolis forces, generated as the fly rotates in space, deflect the haltere from its beating plane. Because Coriolis forces are the cross product of the linear velocity of the haltere and the angular velocity of the body, the resultant strains measured by haltere mechanoreceptors should rise with rotational velocity. Haltere-mediated wingbeat responses have been correlated to rotational velocity, providing additional confirmation of their gyroscopic function (Dickinson, 1999).

Fig. 1.

(A) Cartoon of fly showing the compound eyes and mechanosensory halteres. (B) The flight simulator, which delivers visual and mechanosensory stimuli, is composed of a wrap-around light-emitting diode (LED) display mounted within a 3 degrees-of-freedom rotational gimbal. The fly is mounted in the center of the display, above a sensor that measures the left and right wingbeat amplitudes. (C) Moving striped patterns simulate the optic flow generated when the fly rotates along the roll and pitch axes.

Further evidence of the significance of visual and haltere feedback in flight control can be found by examining the interconnections between these modalities and the fly's motor systems. In Calliphora, motor neurons of neck muscles receive input from both visual and haltere afferents (Strausfeld and Seyan, 1985). The haltere-to-neck motor neuron connection is very fast with latencies from stimulation to motor neuron spiking of 2.5-3.0 ms (Calliphora: Sandeman and Markl, 1980). Haltere afferents also connect to the motor neuron of steering muscle B1 (mnb1) (Calliphora: Fayyazuddin and Dickinson, 1996; Drosophila: Trimarchi and Murphey, 1997), a muscle whose firing activity has been correlated with changes in wingbeat amplitude (Drosophila: Heide and Götz, 1996; Calliphora: Tu and Dickinson, 1996; Balint and Dickinson, 2001). Although a direct connection between visual afferents and the steering muscles has not yet been documented with intracellular recordings, in the flesh fly Neobellieria (=Sarcophaga) bullata, visual interneurons are dye coupled to the motor neurons of B1 and B2, providing anatomical support for such a connection (Gronenberg and Strausfeld, 1991). Visual afferents do, however, provide excitatory input to the muscles controlling the motion of the halteres, suggesting visual input can influence steering muscle activity indirectly (Calliphora: Chan et al., 1998).

Behavioral experiments conducted in a wide range of animals provide further insight into the mechanisms of sensory fusion. One aspect of multimodal integration, which has been the focus of previous work, is the role that each sensory modality plays in different phases of complex behaviors. In cockroaches, for example, visual and mechanosensory cues are involved in different stages of the escape reflex (Ye et al., 2003). Similarly, predatory fish rely on sensory feedback from the visual system, lateral line and, in some cases, electrosensory organs during feeding behavior. Although some or all of these components contribute to the overall success in prey capture, the relative importance of each feedback channel varies through different stages of feeding (Nelson et al., 2002; New, 2002; New and Kang, 2000). Locusts use feedback from many sensory modalities, including compound eyes, ocelli and wind-sensitive hairs, to orient themselves during flight (for a review, see Reichert, 1993). The ocellar system can inhibit the strong excitatory input from the compound eyes and wind hairs on descending interneurons if these modalities are providing feedback that is in conflict with the ocelli. This suggests that, in the presence of inconsistent measurements, the control system relies on the ocellar feedback (Reichert, 1993). Although these studies document interaction among certain sensory inputs during complex behaviors, they do not provide a quantitative measure of the relative contribution of each component when all sensory systems are intact. Studies in humans on the integration of visual and haptic cues during perception tasks have demonstrated that behaviors resulting from cue combinations correlate well with the output predicted by a maximum-likelihood estimator (Ernst and Banks, 2002; Hillis et al., 2002). In this paradigm, the relative contribution of each modality to the overall sensory estimate is directly related to the variance of its measurement, such that signals with lower variance are given more influence.

The goal of the present study is to characterize the integration of feedback from the halteres and visual system during compensatory flight maneuvers in fruit flies. Using a specialized flight simulator, we activate different sensory modalities both individually and concurrently while monitoring the animal's behavioral response. We systematically vary the relative phase, amplitude and rotational axis position of concurrent visual and mechanical oscillations to determine the contribution of each sensory modality. Our results show that the flight control system uses both sensory channels when available, such that the response to complementary concurrent stimuli is larger than the response elicited by exciting just one modality. The flight control system integrates these inputs in a manner that can be modeled by a weighted sum, in which haltere feedback is given preference over visual information.

Materials and methods

Animals and preparation

Data were collected from 2-4-day-old female fruit flies (Drosophila melanogaster Miegen). We used flies maintained in a laboratory colony started from a population of 200 wild-caught females. Prior to tethering, flies were anaesthetized on a 4°C peltier stage. A small piece of tungsten wire, diameter 0.1 mm, was glued between the head and the thorax. For a more detailed description of the tethering process, see Lehmann and Dickinson (1997). Each fly's head was fixed with glue to the thorax to prevent relative motion cues during experiments. Flies were given at least an hour to recover before being placed in the flight arena.

Flight simulator

A flight simulator composed of a cylindrical light-emitting diode (LED) display mounted within a 3 degrees-of-freedom rotational gimbal (for details, see Sherman and Dickinson, 2003) enabled us to present separate and concurrent visual and mechanosensory stimuli during tethered flight (Fig. 1B). The high spatial resolution (λmin=3.5°) wrap-around visual display was refreshed at approximately 1 kHz, well above the temporal resolution of the Drosophila visual system. Each tethered fly was mounted in the center of the display beneath an infrared LED and above two photocells that comprise part of a real time wingbeat analyzer (Heide and Götz, 1996; Lehmann and Dickinson, 1997). For each stroke cycle, the wingbeat analyzer measures the stroke amplitude for each wing, as well as the instantaneous wingbeat frequency.

Rotation experiments

The objective of this study was to test the flies' response to concurrent visual and mechanical stimuli. The two stimuli were presented as sinusoidal oscillations that varied by phase offset, amplitude or axis of rotation. In each case, our experimental protocol consisted of presenting each fly with a repeated series of three open-loop stimuli separated by 5 s of rest, an interlude that allowed their wingbeat amplitude and frequency to return to pre-stimulus levels. During these recovery periods, the flies were maintained under an optomotor closed-loop condition in which they controlled the position of a 14° wide dark stripe by adjusting their left and right wingbeat amplitudes (Götz, 1987). By switching to closed-loop conditions during the 'rest' periods, flies tended to respond more robustly during subsequent stimulus presentations. Each segment of stimulus presentation began with mechanical oscillation with no accompanying visual motion. This was followed by a visual oscillation during which a striped pattern on the display rotated around either the roll or pitch axis but with no mechanical oscillation (Fig. 1C). The final stimulus in each trial was a simultaneous presentation of visual and mechanical oscillations. Each stimulus presentation consisted of six sinusoidal oscillations.

The first set of experiments explored the effect of phase offset between visual and mechanical stimulation. For these experiments, the concurrent oscillations were always presented at the same frequency but with a phase offset, φ, which was systematically varied in different trials by increments of 45°. For each experiment, φ was selected at random from eight phase increments between 0° and 360°. The amplitude of both visual and mechanical oscillations was 30°. It should be noted that by defining both mechanical and visual rotations with the same convention (i.e. right-hand rule), a fly will experience a naturalistic combination of mechanical and visual stimuli when the two are presented with a phase offset of 180°. This is counter-intuitive but results from the fact that if a fly physically rotates to the left, the visual word will move across its retina to the left.

The second set of experiments measured how mechanical motion of varying amplitude influenced the contribution of visual feedback during simultaneous stimulation. The mechanical and visual stimuli were presented at the same frequency (1.2 Hz) but with the amplitude of the mechanical oscillations varied randomly between 5°, 10°, 20°, 30° and 40°. For all trials, the amplitude of the visual stimulus was fixed at 30°. Data were collected at each of the following phase offsets: 180°, 270° and 90°. All oscillations were about the pitch axis, since motor responses are generally more robust than for roll or yaw.

Our third experiment focused on the response to simultaneous rotations about two orthogonal axes. In this experiment, flies are presented with concurrent mechanical and visual rotations that have the same frequency and amplitude but differ in their axis of rotation. We measured the response to the following stimulus combinations: visual pitch/mechanical roll and visual roll/mechanical pitch.

Based on our results from previous frequency response experiments (Sherman and Dickinson, 2003), we selected one oscillation frequency for each experiment and axis such that the responses to visual and mechanical motion were of comparable strength. The selected frequency for the phase and amplitude experiments was 3.0 Hz for roll stimuli (ωpeak=565 deg. s-1) and 1.2 Hz for pitch (ωpeak=226 deg. s-1). For the orthogonal axes experiments, the oscillation frequency was 1.2 Hz for visual roll/mechanical pitch and 2.4 Hz for visual pitch/mechanical roll. Previous results showed that the haltere-mediated response to yaw was much weaker than the visually elicited yaw response (Sherman and Dickinson, 2003). For this reason, we focused on pitch and roll and did not investigate the interaction between visual and mechanical stimuli for rotations about the yaw body axis.

The data, which included left and right wingbeat amplitude, wingbeat frequency, the position of the visual pattern and the orientation of the gimbal, were digitized at 200 Hz and stored on computer. Signals were filtered digitally (zero phase delay) with a low pass cut-off of 20 Hz to remove any high frequency noise. To account for slight differences in the position of each fly relative to the wingbeat analyzer, we normalized the wingbeat data with respect to baseline variability as described previously (Sherman and Dickinson, 2003; Tammero and Dickinson, 2002). Thus, in these experiments the output of the wingbeat analyzer is used as a relative measure of behavioral responses, not as a precise measure of stroke amplitude. The output from the wingbeat analyzer is, however, linearly proportional to both stroke amplitude (Lehmann and Dickinson, 1997) and flight torque (Tammero et al., 2003). We interpret modulation of the sum of the left and right wingbeat amplitudes to represent the control of pitch, because a bilateral change in stroke amplitude creates moments about the pitch axis. Similarly, modulation in the difference between the left and right wingbeat amplitude represents the fly's attempt to adjust roll or yaw.

For these analyses, we only included data from flies that flew long enough to complete the multiple set of trials that were required for each experiment. The number of repetitions varied depending on the experiment: for phase experiments, flies completed at least five repetitions at each of the eight phase relationships; for amplitude experiments, flies completed minimally six repetitions at five amplitudes; and for multi-axis experiments, flies completed 20 repetitions for the one experimental condition. Multiple responses at a given experimental condition (phase offset, amplitude, axis) were time-averaged. A fast Fourier transform (FFT) algorithm was used to find the sine curve that best fit the averaged responses to mechanical rotation (Rm), visual rotation (Rv) and concurrent mechanical and visual rotation (Rm+v). The amplitude of the calculated sine fits is denoted ΔWBA in figures and legends. The averaged wingbeat amplitude signals are referred to as WBA. All data were analyzed using custom software written in MATLAB (Mathworks, Natick, MA, USA).


Flies respond to separate mechanical and visual oscillations about the roll axis with roughly sinusoidal changes in the difference of left and right wingbeat amplitude (Fig. 2i,ii). In free flight, flies experience a 180° phase offset between sensory stimuli as rotation of the body in space generates visual motion across the eye in the opposite direction. Flies presented with simultaneous mechanical and visual oscillations offset by a phase of 180° show responses that are considerably larger in magnitude than the responses to the two stimuli when presented separately (Fig. 2Aiii). For the fly whose data are presented in Fig. 2 under conditions that should approximate those of free flight, the two sensory motor reflexes appear to sum linearly. By experimentally shifting the phase offset between the visual and mechanical oscillations, we tested the relative weighting of the two sensory feedback channels in the compensatory reflex. The fly's response to a phase shift of 0°, which should represent conflicting visual and mechanosensory information, is quite small and indistinguishable from spontaneous background motor activity (Fig. 2Biii). Flies respond robustly at a phase offset of 270°, despite the unnatural lag between the sensory signals (Fig. 2Ciii). At this phase relationship, the sum of the responses to separate stimuli is approximately equal to the response to concurrent stimuli, although the fit is weaker than at a 180° phase offset.

Fig. 2.

Flies respond to mechanical, visual and concurrent oscillations with sinusoidal modulations of wingbeat amplitude. Rotational stimuli were applied around the roll axis. Each trace represents the time-averaged response to six trials for a single fly. (A) Concurrent visual and mechanical oscillations phase shifted by 180° elicit strong responses (Rm+v) that are approximately equal to the sum of the responses to each stimuli presented separately (Rm+Rv). (B) When visual and mechanical stimuli are presented with a 0° phase offset, flies display almost no response. (C) At a 270° phase offset, responses are intermediate in amplitude. L, left; R, right; φ, phase offset; θ, position. Wingbeat amplitude (WBA) is plotted in normalized units.

We examined the responses in wingbeat amplitude (left minus right for roll, left plus right for pitch) to simultaneous visual and mechanical oscillations at eight different phase offsets. The responses averaged across flies are plotted against the phase offset between the two sensory stimuli in Fig. 3 (A, pitch axis; B, roll axis). The response to concurrent stimuli is weakest at phases between +45° and -45°, conditions at which feedback from the visual system and halteres is most contradictory. The response to simultaneous 3 Hz visual and mechanical rotation about the roll axis peaks near a stimulus phase difference of approximately 150°. This 30° delay from the expected maximum stimulus phase of 180° might be explained by the intrinsic delay in visual motion processing, estimated to be approximately 30 ms in flies (Land and Collett, 1974). At the stimulus frequency of 3 Hz, this delay would cause a phase shift of 33°, which is comparable with the observed value. The same logic applied to the roll results, however, predicts a phase delay of 13°, whereas an advance of ∼20° was observed (Fig. 3B). Thus, although the intrinsic processing delays within the two modalities should influence these results, our measurements do not appear to offer adequate resolution to observe such subtle effects.

Fig. 3.

The magnitude of the wingbeat responses to concurrent oscillations about the pitch (A) and roll (B) axes is a function of the phase offset between mechanical and visual stimuli. (A) Pitch axis responses (mean ± s.e.m., N=10 flies). (B) Roll axis responses (mean± s.e.m., N=14 flies). The amplitude of the sine fits to wingbeat data (Rm+v; green circles) are plotted against phase offset (φ). In addition, two possible fits for the data are plotted: linear summation (Rm+Rv; black curve) and scaled summation (αRmRv; red curve). The two fitted functions are 4th order polynomials.

For both roll and pitch, the peak response to concurrent stimuli is approximately double the amplitude of the responses at zero phase and is significantly larger than the mean responses to stimuli presented separately (indicated on the y-axis as R-v and R-m). This suggests that the response of the fly to input from a single modality is not saturated, and experimental treatments presenting either visual or haltere stimulation alone yield only a fraction of the potential response of a freely flying insect encoding the same motion cues with multiple sensory systems.

To examine the relative contribution of each sensory modality on flight equilibrium reflexes, we fit the amplitude of the response to concurrent stimuli (Rm+v) with a linear sum of isolated stimuli (Rm+Rv). Whereas this appears to fit some phase relationships very well, this simple scheme did not provide a good fit at all phase offsets. A substantially better match resulted from a weighted sum: Math1 where α is the weighting coefficient of the mechanosensory response that depends on the magnitude of the visual velocity, ω, and β is a weighting coefficient for the visual response that depends on the magnitude of mechanical oscillation, u (Fig. 3). The two weighting functions, α and β, represent the feedback of one sensory modality on the other. When either stimulus is presented alone, the respective weighting function must equal 1 [i.e.α (0)=1 and β(0)=1]. Values for α and β, chosen such that they minimized the quantity,∥ Rm+v-(αRmRv)∥, were calculated for the responses shown in Fig. 3. The scaling coefficients, so determined, were similar for both roll and pitch experiments, which implies that the weighting of inputs from each sensory modality is not dependent on the axis of rotation. Averaging the coefficients for both axes yielded the values of α=1.1 and β=0.65. This suggests that for the stimulus magnitudes used in Fig. 2, the haltere-mediated inputs are weighted more heavily than the visual input at all relative stimulus phases. Furthermore, becauseα≈ 1 and β<1, the response to a mechanical stimulus is unaffected by visual input, whereas a concurrent mechanical stimulus significantly diminishes the response to visual input.

To determine how β varies with u, we measured the contribution of visual feedback during concurrent oscillations while varying u. We calculated β for each mechanical stimulus amplitude by solving equation 1, withα =1. The mean value of β for a group of flies is plotted against u for the three phase offsets tested (Fig. 4A). There is no significant difference in the value of β over a wide range of mechanical stimulus amplitudes, implying that any haltere stimulation, regardless of magnitude, results in a fixed decrease in the contribution from the visual system (Fig. 4A). Thus, β appears to resemble a switching function. To test this possibility further, we calculated the single values of both α and β that best fit the data for all three phase relationships and all five amplitudes (Fig. 4B). The resulting values, α=1.17 and β=0.65, provide an excellent fit for the data, implying that β is approximately constant for all u. Furthermore, these weighting values correspond well with values determined independently in the separate phase offset experiments.

Fig. 4.

The attenuation of visually elicited motor responses is not a linear function of the magnitude of haltere stimulation. With α set to 1, the scaling factor β was calculated for multiple flies during concurrent oscillations in which the amplitude of the visual stimulus was fixed at 30° and the amplitude of the mechanical stimulus (u) was varied. (A) β (mean ± s.e.m.) versus u for three phase relationships (φ): 180°, 270° and 90° (N=8, 9 and 9 flies, respectively). (B) Amplitude of sine fit to time-averaged wingbeat responses versus u for φ=180°, 270° and 90° (N=8, 9 and 9, respectively). Data fit by weighted sum (spline approximation of the fit),α RmRv, with α=1.17 and β=0.65.

We next tested whether haltere feedback is still weighted more heavily than visual feedback if the mechanical and visual oscillations are applied along orthogonal stimulus axes. When presented with a combined stimulus of mechanical pitch and visual roll, the flies modulate wing kinematics to create a simultaneous roll and pitch motion. Averaged traces from a single fly illustrate that during concurrent oscillations flies exhibit a pitching response equal to that elicited by a mechanical stimulus presented alone (Rm; Fig. 5A, top trace). At the same time, the fly modulates left minus right wingbeat amplitude to create a roll response that is approximately 60% of the amplitude of a pure visually elicited response (Rv; Fig. 5A, bottom trace). Flies achieve both responses simultaneously through a slight shift in the phase relationship between the amplitude modulation of left and right wings. Because mechanical pitch oscillations yield a wingbeat response with an insignificant rolling component, and likewise visual roll will result in very little modulation of pitch, we can estimate that the pitch response (Rp) is a function of just Rm, and the roll response (Rr) is a function of Rv (Fig. 5B,C). We calculated the values of α and β that best fit the equations RpRm and RrRv. For the converse experiment, mechanical roll/visual pitch, we determined α and β that best fit the equations RpRv and RrRm. The averages of the resulting α and β were 1.0 and 0.68, respectively. Thus, regardless of the axis of rotation, the phase or the amplitude of the stimuli, it appears that mechanosensory feedback is weighted more heavily than visual feedback when both signals are simultaneously active.

Fig. 5.

Responses to mechanical and visual oscillations about orthogonal axes have both a pitch and roll component. (A) The wingbeat response to mechanical pitch (Rm), visual roll (Rv) and the two stimuli concurrently (Rm+v) of a single fly. Each trace represents the time-averaged response to 21 trials of each stimulus. (B) Magnitude of roll and pitch components in response to simultaneous visual roll and mechanical pitch (mean ± s.e.m., N=10 flies). (C) Responses to visual pitch and mechanical roll (mean ± s.e.m., N=13 flies). The y-axis represents normalized wingbeat amplitude (ΔWBA) units.


Flies subjected to concurrent visual and mechanical oscillations at varying phase offsets respond with reflexive compensatory modulations of wingbeat amplitude (Fig. 2). The magnitude of compensatory wingbeat responses is a function of the phase between the two stimuli. The largest response occurs near 180°, the phase difference that would occur when a freely flying animal rotates in space (Fig. 3). The response to concurrent sensory input is accurately fit by a weighted sum of the responses to each stimulus presented separately. The different weighting coefficients indicate that visual input does not suppress or alter the response to mechanical rotation, whereas the presence of mechanical stimulation decreases the gain of the visual response. This effect is most likely mediated by the haltere system, although other mechanosensory pathways could contribute (Hengstenberg, 1991). Curiously, the attenuation of the visual component of the compensatory reflex does not vary with the magnitude of the mechanical stimulus (Fig. 4A). Instead, the weighting function resembles a switch, such that the presence of any mechanical oscillation, regardless of magnitude, is enough to suppress the impact of visual feedback. Mechanosensory feedback attenuates the visual reflexes even when the fly is mechanically and visually rotated about orthogonal axes. Furthermore, in such experiments, flies execute a combined response, indicating the summation of multimodal response functions across different stimulus axes (Fig. 5). Over a wide range of experimental treatments, the wingbeat responses to concurrent oscillations were well fit with similar weighting values, suggesting that the gains on each feedback channel in the flight control system are hard wired.

Interconnections between halteres, visual system and flight motor

A biologically accurate model of the flies' sensorimotor control system must incorporate our current understanding of the convergence of sensory input onto the flight motor. Visual feedback projects to each of the three major muscle groups that mediate flight control: wing, neck and haltere muscles. In Calliphora, lobula plate tangential neurons that encode visual motion synapse directly with motor neurons controlling the neck muscles that adjust gaze during flight (Strausfeld and Seyan, 1985). However, such connections could not be responsible for the mechanosensory-mediated attenuation of the visual response because the head was rigidly fixed to the thorax in our experiments. Also, in Calliphora, Chan et al. (1998) found that visual stimulation evokes directionally specific spiking responses in control muscles of the haltere within the metathorax. This synaptic interaction would provide a mechanism by which visual feedback could modulate haltere input but does not explain the phenomena we have observed in Drosophila, in which haltere input attenuates visual reflexes.

The most likely anatomical sites for the observed interaction between mechanosensory and visual pathways are either the dendrites of descending interneurons within the brain or the flight motor neurons within the thorax. In Calliphora, haltere afferents project directly to the subesophageal ganglion (Chan and Dickinson, 1996) and thus might provide direct or poly-synaptic input to descending interneurons encoding wide-field visual motion information. Haltere afferents and visual interneurons also converge on pathways within the thorax. Although there is little electrophysiological evidence for a direct physiological link between descending visual interneurons and steering muscles, this absence may simply reflect gating in quiescent preparations as occurs in locusts (Reichert and Rowell, 1985). There is anatomical evidence that the motor neurons of at least two wing steering muscles, B1 and B2, are dye-coupled to descending visual neurons in male flesh flies (Neobellieria bullata; Gronenberg and Strausfeld, 1991). The steering muscle B1 receives electrotonic input from haltere afferents in both Calliphora vicina (Fayyazuddin and Dickinson, 1996) and Drosophila melanogaster (Trimarchi and Murphey, 1997). In both species, B1 is known to control changes in wingbeat amplitude (Heide and Götz, 1996; Tu and Dickinson, 1996). Haltere afferents are also known to contact the B2 steering muscle (Fayyazuddin et al., 1993), but this connection has not been studied in detail. Nevertheless, it appears that haltere afferents and visual interneurons converge directly onto the steering muscle motor neurons controlling stroke amplitude.

It is important to note that although the haltere is the likely source of the signal that modifies visual input, it is not the only possibility. Hengstenberg (1991) presented evidence for as many as eight reflexes that can provide feedback to the neck motor system in Calliphora, and many of these could function similarly to detect mechanical oscillations and control wing motion in Drosophila. Although a previous ablation study indicated that the halteres are required for the major component of the wingbeat response to mechanical oscillation (Dickinson, 1999), interpretation of ablation experiments is somewhat ambiguous and we did not repeat such methods in this study. Aside from the compound eyes, other non-haltere sources of equilibrium feedback include the ocelli, prosternal hairs on the neck, and wing campaniform sensilla. These modalities could contribute to both the basic response to mechanical oscillation and the attenuation of the visual reflex during concurrent presentation. Given that the head was fixed to the thorax and the fly was rigidly fixed to the light display when oscillated, it is unlikely that the ocelli or neck receptors are involved in these effects. However, it is impossible to rule out the contribution of wing sensilla, which could respond to changes in loading during mechanical oscillation or Coriolis forces acting on the wing.

Physiological mechanisms of inhibition

No matter what receptors are involved, or where the convergence takes place, the afferents sensitive to mechanical oscillations must somehow alter the membrane properties of cells within the visual pathway. One could propose a variety of circuits involving pre- or post-synaptic inhibition and layers of local interneurons to explain these effects. However, one simple and parsimonious explanation is that the small attenuation of the visual pathway arises indirectly from the spatial arrangement of convergent afferents on post-synaptic neurons. Nonlinear spatial summation has been characterized on the tangential cells of the lobula plate (Single and Borst, 1998). For example, if haltere inputs are positioned closer to the spike initiation zone of a post-synaptic motor neuron than the visual interneurons, the impact of visual input might be diminished in the presence of an active haltere input, which would function to shunt input from more distal synapses. In Calliphora, haltere afferents originating at dF2, the campaniform field considered most likely to encode gyroscopic forces (Fayyazuddin and Dickinson, 1995; Pringle, 1948), synapse very close to the axon of the B1 motor neuron (Chan and Dickinson, 1996), whereas the terminals of descending interneurons, based on studies in Neobellieria bullata (Gronenberg and Strausfeld, 1991), appear to be more medial. The distal location of the haltere terminals, which contain a sizable electrotonic component, has presumably evolved to minimize the latency of equilibrium reflexes (Fayyazuddin and Dickinson, 1996). Thus, the attenuation of the visual input may represent a secondary consequence of a circuit designed to rapidly convey haltere information to motor neurons. An alternative explanation is that the weighting of the two sensory inputs may play a specific functional role in the performance of the flight control system. Obviously, support for this or any other explanation will require further physiological and anatomical studies.

Functional explanations of haltere dominance

In flight control, as in most feedback-mediated control systems, quick reliable feedback is essential for stability and robustness in the presence of disturbances. The haltere mechanoreceptors and the visual system vary greatly in their temporal responses and reliability. Haltere feedback is very fast; the delay from haltere deflection to neck motor neuron firing is approximately 3 ms (Sandeman and Markl, 1980). Visually mediated motor responses are an order of magnitude slower (30 ms), as estimated for flight chases in free-flying houseflies (Fannia canicularis; Land and Collett, 1974). This disparity is due in large part to the relatively slow process of phototransduction, which involves a biochemical cascade. On the other hand, the visual system is much more sensitive than the halteres to slow changes in rotation (Hengstenberg, 1991; Sherman and Dickinson, 2003).

Models of multimodal integration have shown that the feedback weights from multiple sensory channels correlate well with scaling factors generated by a maximum likelihood estimator (Ernst and Banks, 2002; Hillis et al., 2002). A maximum likelihood estimator determines a quantity by taking a scaled sum of all the measurements of that quantity. The weight assigned to each measurement is inversely proportional to the normalized variance in the measurement. Thus, the greater the variability in a measurement, the less influence it is given. Although our results do not indicate that the visual responses have a larger variance than the haltere responses, on an intuitive level the complexity of the visual world could produce a signal with more ambiguity than the signal from the halteres. For example, visual estimation of rotational velocity depends on a wide range of parameters including image contrast, luminance and spatial structure, all of which may vary quite widely during flight in natural settings (Reichardt and Poggio, 1976). In addition, coherent visual motion, such as the swaying of tree branches or grass stems, might inappropriately provide an adequate stimulus for rotation-sensitive circuits within the visual system. Thus, the flight control system might compensate for potential visual miscues by limiting the weight placed on the visual measurement and showing preference to the halteres, which provide an accurate measure of velocity that is not contaminated by the spatial composition of the visual world.

Characterizing and making sense of β

Our analyses have led to a characterization of a rather unusual weighting function, β, which functions as a switch (Fig. 4A). This function has two distinct and somewhat perplexing features: first, a discontinuity near zero and, secondly, a constant gain over a range in which the haltere-mediated wingbeat response rises monotonically. In regards to the first issue, we know from previous research that there must exist a threshold in angular velocity below which the haltere sensors fail to respond to body rotation. This feature of the haltere response was identified previously by Hengstenberg (1991), who reported that haltere-mediated head movements only occurred above rotation velocities of 50 deg. s-1. In our experiments, the smallest mechanical oscillation (amplitude=5°) resulted in a peak angular velocity of 38 deg. s-1, a stimulus that elicited very weak responses. Because the signal-to-noise ratio for the haltere-mediated wingbeat response is large for very low levels of stimulation, we cannot confidently determine β in this region, thus we cannot rule out the possibility that the weighting function rises smoothly to 1 at low stimulus intensities.

An equally confounding feature of the visual gain function is that it remains constant over a wide range of stimulus intensity in which the haltere-mediated wingbeat responses were increasing (Fig. 4A). This latter observation would suggest that the haltere system (or any other sensory modalities, such as wing campaniforms, that contribute to the effect) increases its output monotonically with increasing stimulus amplitude. Studies of campaniform sensilla on the wing, which are serially homologous to those on the haltere, would suggest that during flight this dynamic range is achieved via either recruitment or phase shifts and not by changes in the firing frequency of individual cells (Dickinson, 1990). However, a gradual recruitment of mechanoreceptors is not consistent with the all-or-none change in β. On the other hand, a haltere-encoding mechanism that involves intensity-dependent phase shifts is not inconsistent with our results. Changes in wingbeat amplitude are tightly correlated with the advances in the firing phase of mnb1 (Heide and Götz, 1996; Tu and Dickinson, 1996), and there is some evidence that haltere afferents may be in part responsible for such shifts (Fayyazuddin and Dickinson, 1999). If all the individual mechanoreceptors are recruited at near threshold stimulus levels, and the wingbeat amplitude adjustments are due to the timing and not the magnitude of the input, then there is no reason to expect the level of visual suppression to vary with haltere stimulation. A final possibility is that these effects are mediated by other mechanoreceptors, such as on the wings, antennae or legs, and these have already reached a saturated region of operation. Regardless of the physiological underpinnings, we have shown that this sensory input weighting is extremely robust over a wide range of experimental conditions.

Control system model

The results presented here suggest a flight control model in which each sensory channel when concurrently active is given a particular functional weight. A simple model that incorporates these results with previous findings is shown in Fig. 6. In this model, the dynamics of the two sensory channels are represented by transfer functions. These functions, determined by our previous frequency response analysis (Sherman and Dickinson, 2003), represent the input-output relationship between angular velocity and wingbeat amplitude response. Thus, they comprise multiple elements along the sensorimotor pathway, including signal transduction, sensory processing and flight muscle dynamics, each of which contributes temporal characteristics to the net response. The visual system transfer function can be approximated as a low pass filter, since only slow rotations elicit large responses. Although the haltere-mediated wingbeat response increases with increasing velocity, the gain of the system is approximately constant in the operating region, thus the transfer function can be approximated as a band pass filter (Sherman and Dickinson, 2003). The results described in the present study have provided the appropriate weighting functions for each channel; a switch can model the visual system weighting function, and a unity gain block can model the weight on the haltere signal. While this model does not provide insight into the physiology behind these interactions, it does create a framework useful for characterizing the interaction between multiple sensory feedback channels and the flight motor.

Fig. 6.

Input from the halteres and visual system are combined in a weighted sum by the flight control system. The weighting of the visual feedback is dependent on the presence of haltere input but appears to be constant over a wide range of haltere stimulation. The processing of each sensory input is represented by a transfer function based on our previous frequency response analysis. G, gain; ω, angular velocity.

In summary, we have determined from a systems perspective how feedback from different sensory modalities is integrated into a flight control algorithm. Our results show that inputs from the visual system and halteres are combined in a weighted sum, which favors information from the halteres. Furthermore, the weights on each sensory channel appear to be independent of the phase, magnitude or axis of rotation, suggesting a hard-wired control mechanism. These results provide insight into the mechanisms of feedback in flight control and contribute to a general understanding of multimodal integration.

List of symbols

response to mechanical rotation
mean response to mechanical rotation
response to concurrent mechanical and visual rotation
response to pitch
response to roll
response to visual rotation
mean response to visual rotation
amplitude of mechanical oscillation
weighting coefficient of mechanosensory response
weighting coefficient of visual response
phase offset
angular velocity


The authors wish to thank Mark Frye for helpful comments on this research. This work was supported by grants from the Packard Foundation and the National Science Foundation (FD97-23424) to M.H.D.


View Abstract