Forward shift of feeding buzz components of dolphins and belugas during associative learning reveals a likely connection to reward expectation, pleasure and brain dopamine activation

We train bottlenose dolphins [Tursiops truncatus (Montagu 1821)] and belugas [Delphinapterus leucas (Pallas 1776)] for our research. We use operant conditioning (Skinner, 1957) and the reward is mostly fish. Rewards cause animals to repeat behaviors. Over time, they learn to associate specific signals with reward. Initially, we give the reward (S^R+) as close as possible to the instant that the animals execute the desired action. A discriminative stimulus (S^D) leads the animal to execute the action at the desired time. The S^D for our experiments was a sound or a trainer's hand signal, which begins a chain of animal responses. When the animal responds correctly, at the end of the response chain, the trainer sounds a secondary reinforcer (SR) to bridge the time gap until S^R+ is given. Intervals between S^D, SR and S^R+ are extended as training proceeds. With repetition, the animal develops the strong expectation that SR will result in S^R+. In our case, S^R+ is a fish reward.

We focus on fish rewards, but rewards other than food are sometimes used. Olds and Milner (Olds and Milner, 1954) showed that rats would work for electrical brain stimulation. This discovery led to the idea of pleasure centers in the brain (Olds, 1956). Following the early findings on rats, Lilly and Miller (Lilly and Miller, 1962) showed that dolphins would also work for electrical brain stimulation. Dolphins continuously pressed a paddle to receive the stimulus as the only reward. In an earlier study, one dolphin vocalized after electrical stimulation of a brain reward area (Lilly, 1958). Other studies showed that humans found such stimulation pleasurable. Like rats, monkeys and dolphins, humans repeatedly responded to receive the rewarding electrical stimulus. Both rewarding and aversive areas were found in the human brain (Heath, 1963; Bishop et al., 1963). As in the brains of humans, monkeys and rats, there are dopamine neurons in reward areas of dolphin brains (Manger et al., 2004).

Findings from studies ranging from electrical stimulation to brain chemistry have demonstrated that mammals from rats to primates have reward systems involving dopamine neurons. Not only is brain dopamine released with reward, dopamine release predicts future reward. Wise (Wise, 2008) summarized early work on reward prediction as follows: ‘Schultz and colleagues have shown that the ventral tegmental dopamine neurons implicated in reward function respond not only to food reward itself but, as a result of experience, to predictors of food reward (Romo and Schultz, 1990; Ljungberg et al., 1992). As the animal learns that an environmental stimulus predicts food reward, the 200 millisecond burst of dopaminergic nerve firing that was initially triggered by food presentation itself becomes linked; instead, to the food-predictive stimulus that precedes it.’ Rats have different vocalizations for different emotional states. Lower frequency (22 kHz) calls, mediated by cholinergic neurons, reflect negative emotions such as alarm. Higher frequency (50 kHz) calls, mediated by dopamine, reflect positive emotions connected with reward (Brudzynski, 2007; Burgdorf et al., 2011).

Lilly's dolphin vocalized after stimulation of brain reward centers (Lilly, 1958). There are dopamine neurons in these areas of the dolphin brain (Manger et al., 2004). Dopamine mediates positive emotions reflected in specific vocalizations. These facts suggest to us that dolphins might make specific types of sounds after dopamine release and that such sounds could follow food reward or the expectation of food reward.

We found much evidence in the literature for food or feeding-associated calls from cetaceans. In a group of bottlenose dolphins at Marineland of Florida, Wood (Wood, 1953) described mewing and rasping feeding sounds. Kellogg (Kellogg, 1959) mentioned that echolocation became ‘almost continuous’ as a dolphin approached and took a preferred fish. After taking a fish, a dolphin emitted a ‘brush of pulses’ (Norris et al., 1961). DeRuiter et al. (DeRuiter et al., 2009) correlated the moment of fish capture with sound. Unfortunately, no such correlations were available for the earlier observations.

Griffin et al. (Griffin et al., 1960) coined the term feeding buzz (reviewed by Fenton, 2013) for the rapid increase in pulse rate as a bat closes in on insect prey. Cetaceans also use a feeding buzz as they close in on prey (Miller et al., 1995). Like bats' feeding buzzes, cetaceans' buzzes, in the past, were thought to terminate at prey capture. Thus, cetacean terminal buzzes have been considered mainly as echolocation (Morozov, 1972; Johnson et al., 2004; Miller et al., 2004; Madsen et al., 2005). However, continuation of the buzz after prey capture suggests to us that the buzz may also have emotional content. For example, in echolocation experiments, porpoises (Phocoena phocoena) emitted buzzes as they approached and took a fish. Porpoises began their buzzes approximately 500 ms before and ended them approximately 800 ms after they took a fish (DeRuiter et al., 2009; Verfuss et al., 2009). DeRuiter et al. (DeRuiter et al., 2009) say, ‘Unexplained continuation of buzzes after prey capture raises questions about the function of buzzes.’

There are several reports of wild whale sounds suspected to be markers for food capture. Sperm whales (Physeter macrocephalus) use terminal buzzes or creaks that are suspected to alert other whales up to several kilometers away to the presence of a feeding opportunity (Miller et al., 2004). Beaked whales (Mesoplodon densitorstris) give out buzzes or creaks in feeding situations. These buzzes are likely markers for food capture (Johnson et al., 2004; Madsen et al., 2005). Humpback whales (Megaptera novaeangliae) produce buzzes during feeding moves (Stimpert et al., 2007). Food calls, common in the animal kingdom, are heard when an animal locates food (Clay et al., 2012). In fact, Tyack and Clark (Tyack and Clark, 2000) suggested that ‘these kinds of buzzes could be viewed as preadaptations for a food call.’ We suggest that the mews, rasps, brushes of pulses, buzzes, creaks or squeals heard from feeding cetaceans have an emotional content. They are food calls in addition to previously considered functions for communication and echolocation discrimination.

For many years, we have heard feeding buzzes that continue after the animal takes a fish. Similar, rapid, peak -frequency-varying pulsed sounds occur after an SR signal that the trainer gives to indicate future reward. Both of these particular pulsed sounds remind us of a child's squeal of delight. Thus we named them victory squeals (VS) (Ridgway et al., 1991a; Ridgway and Carder, 1998; Finneran, 2013; Branstetter et al., 2013). Both dolphins and belugas produce many bursts of pulses in many different behavioral contexts. The VS we studied are rapid pulse bursts that vary in amplitude and peak frequency, and occur either after the animal secures a fish or after our SR promising future reward. Because modern land mammals have well-developed brain dopamine reward systems (Romo and Schultz, 1990; Ljungberg et al., 1992; Richardson and Gratton, 1996; Berridge and Robinson, 1998; Wise, 2008; Haber and Knutson, 2010; Schultz, 2010), and some make sounds connected with reward (Lilly, 1958; Brudzynski, 2007; Burgdorf et al., 2011), we considered our VS as possible evidence for such systems in cetaceans.

Here we report VS observations during six different experiments with 12 bottlenose dolphins and two belugas. We also recorded two belugas at an aquarium. All animals were adults. At first, the VS followed seizing a fish; however, after training for all of these experiments, the VS occurred after the trainer's SR signaling future reward. Fig. 1 illustrates this forward shift of the VS.

We used four different SR signals: (1) a tone that was silenced when the animal pressed a plunger at depth (Ridgway et al., 1969; Ridgway et al., 1984); (2) an air whistle that the trainer sounded when the animals made a correct response; (3) an underwater buzzer sounded by the trainer on correct responses; and (4) the onset of a tone. We considered the possibility that the animal was simply mimicking the SR. Of course, the sound turning off could not be readily mimicked, yet the same beluga followed a sound offset and buzzer SR with a similar VS (Fig. 1D, Fig. 2C). Trainers gave SRs of air whistles, underwater tones, underwater buzzers and sound offsets to signal correct responses. In this series, only the underwater buzzer is a pulsed sound. Yet animals employed similar peak-frequency-varying pulse bursts to follow all four types of SR (Figs 1, 2, 3, 4 and 5).

Although we heard the VS following the SR very often during many different experiments, it was incidental. The VS had nothing to do with the goal of the experiments (Ridgway et al., 1969; Ridgway et al., 1984; Ridgway et al., 1991a; Ridgway et al., 1991b; Schlundt et al., 2000; Ridgway et al., 2001; Blackwood, 2003; Moore et al., 2008; Ridgway, 2011; Branstetter et al., 2012; Branstetter et al., 2013). We observed no differences in the trials where no VS was recorded after an SR. Weaker VS may have been lost in background noise of our natural bay waters. VS followed all four of the different SR signals: air whistles, underwater tones, underwater buzzers and the cessation of an underwater tone. When SR was a projected sound turned off by the animal as in Ridgway et al. (Ridgway et al., 1984) (Fig. 2), or when the SR was a signal activated by the trainer to signal correct response, the animal usually responded with a VS (Figs 3, 4 and 5).

We observed the VS on diving experiments in the open ocean with two belugas and one dolphin. An S^D tone signaled dolphins (Ridgway et al., 1969) and belugas (Ridgway et al., 1984; Ridgway et al., 1991a) to dive. The tone continued until the animal pressed a switch at the end of a long marine cable silencing the tone. Silencing the tone served as an SR and the animal responded with a VS (Fig. 2). (Although echolocation may have aided the animal in locating the plunger in the dark depth, we heard the buzz after the S^D tone went off. The VS shown in Fig. 1D is from the same animal near the surface after responding to test tones.) The animal then returned to the surface from 5 m up to several hundred meters away for food reward (Ridgway et al., 1969; Ridgway et al., 1984; Ridgway et al., 1991a). The return to the surface could take 5 min. Therefore, the animal might not receive a fish reward for 5 min after emitting the VS.

Because we had often heard these pulsed calls to follow trainer's air whistle SRs at aquaria and marine parks, we made recordings from two belugas (one male, one female) at Mystic Aquarium, Mystic, CT, USA. Recordings from the male (Naku) are presented in Fig. 4. These whales were not involved in any of the other experiments presented here.

We have heard the VS during different experiments on hearing and sound production (Ridgway and Carder, 1988; Ridgway et al., 1991b; Ridgway and Carder, 1997; Schlundt et al., 2000; Ridgway et al., 2001; Moore et al., 2008; Ridgway, 2011; Branstetter, 2012; Branstetter, 2013). However, in none of these studies did we define the metrics of these incidental vocalizations. Here we compared latency of the VS to the SR with latency of vocal responses to similar sounds in vocal reaction time (RT) experiments (Ridgway et al., 1991b; Ridgway, 2011). In this comparison for four dolphins in 230 trials with VS and 230 with RT, mean ± s.d. VS latency was 449±145 ms and RT latency was 298±60 ms. VS was 151±10 ms longer in latency than RT (mean ± s.e.m.; T=15.6, P<0.0001). For two belugas in 122 trials with VS and 122 with RT, mean (±s.d.) VS latency was 785±206 ms and RT latency was 534±70 ms. For the belugas, VS latency was 250±20 ms longer than RT latency (mean ± s.e.m. error; T=12.3, P<0.0001). Thus, latency for VS was significantly longer than RT to trained responses that did not signal reward (Table 1, Figs 6, 7).

When we observed belugas feeding on bay anchovies (Anchoa sp.), we recorded sound during this opportunistic feeding. The whales were feeding on masses of bay anchovies attracted to illumination at the edge of their netted enclosure (Ridgway and Carder, 1998). First one whale and then the other would lunge to entrap fish and engulf them. During the feeding lunges, each whale emitted a constant clicking, terminating in a VS (Fig. 8), as fish were engulfed. Thus, when feeding on these live fish, the whales produced the VS as a typical feeding buzz as they approached, contacted and engulfed the fish. However, when responding to stimuli for hearing tests (Schlundt et al., 2000), whales reverted to responding with a VS to the SR and not to the fish reward. In this case, the VS was similar to the last part of the feeding buzz shown in Fig. 1B and Fig. 8.

The SRs that we compared in latency to a trained vocal response consisted of the buzzer activated by the trainer. Its amplitude and bandwidth can be seen in Figs 1 and 5. We observed shorter latencies to trained vocal responses (RT) to sounds of similar frequency and amplitude to our SR but that did not signal reward (Table 1, Figs 6, 7). The VS latencies were a mean of 151 ms longer than the trained responses for dolphins and a mean of 250 ms longer for belugas. This comparatively longer vocal latency is in the range of the dopamine release time seen in terrestrial mammals conditioned to expect reward (Schultz, 2010). Thus, the context and timing suggest that VS signals reward expectation triggered by brain dopamine release. During learning, the conditioned reinforcer becomes associated with the primary reinforcer. Our unique observation is that the animal now gives the VS after the SR rather than after the fish (primary reinforcer). The VS is a clear signal that primary reinforcement becomes strongly associated with the secondary reinforcer. The forward-shifted VS is, in our animals, a consistent acoustic signal for reward expectation.

In nature, dolphins and belugas exist in social groupings. The pursuit of food is a major function of such groupings in the wild. Many characteristics of dolphins and their social structure support the possibility of evolved altruistic behavior (Connor and Norris, 1982). For herd animals such as dolphins and belugas, a feeding buzz by one animal can alert the whole group to the presence of prey. Such a signal can certainly be a benefit to the success of the group. Our VS is a pulsed call. Other calls such as food-associated bray calls, the terminal portions of feeding buzzes and creaks made by cetaceans in nature are also pulsed calls. All of these sounds may alert group members to feeding opportunities (Janik, 2000; Miller et al., 2004; Johnson et al., 2004; Madsen et al., 2005). However, attraction of other animals may simply be a byproduct of the behavior (Janik, 2000).

Food calls are natural vocalizations presented by animals ranging from birds to higher primates (Clay et al., 2012). We suggest that feeding buzzes have multiple functions in cetaceans, as they do in bats: they are food calls and they have emotional content. Griffin et al. (Griffin et al., 1960) coined the term feeding buzz (reviewed by Fenton, 2013) for the rapid increase in pulse rate as a bat closes on insect prey. Bats not only have feeding buzzes, but also have social buzzes that may serve as a cue to bring animals together (Fenton, 2013). Feeding buzzes by bats terminate when the bat seizes prey. Bats generate sound with the larynx. Therefore, swallowing might interrupt sound production. Cetaceans produce sound in the nasal system (Dormer, 1979; Ridgway et al., 1980; Ridgway and Carder, 1988; Cranford et al., 2011; Ridgway, 2011), so swallowing does not inhibit sound production. Cetacean feeding buzzes therefore can continue after prey capture (DeRuiter et al., 2009; Verfuss et al., 2009) (Fig. 1).

The feeding buzz terminology has been adapted in cetacean echolocation literature (Johnson et al., 2004; Johnson et al., 2006) and seems quite appropriate to us. We used the term victory squeal (VS) as it seemed appropriate to the context in which we observed the terminal portion of the call produced after the animal secures a fish or after our SR. For example, when the animal dove on hearing a tone (Fig. 2) and reached the switch at depth to silence the tone, it seemed like a victory to us. Chemical changes in reward centers of the brain result in the buzz. We suggest that the VS we observe evolved as a food call that occurs with feeding.

We hear feeding buzzes when dolphins and belugas pursue and capture fish. During the course of operant conditioning, animals learned to associate the SR with the food reward that follows. In time, the VS became ubiquitous for our animals working in structured tasks. Thus, an evolved food call or feeding buzz shifted forward, indicating expectation of future food reward. The VS may arise from a mechanism similar to calls made after electrical brain stimulation in dolphins and some other mammals (Lilly, 1958) and after dopamine release from reward centers in rodents that expect reward (Richardson and Gratton, 1996; Burgdorf et al., 2001).

A food call, feeding buzz or VS after a signal promising future food reward is not a natural phenomenon. However, there are theories from the learning literature of the 1940s and 1950s suggesting an explanation. This forward shifting of the feeding buzz is reminiscent of a notion first proposed by C. L. Hull. Hull (Hull, 1951), building on the work of previous learning psychologists, proposed that an animal, after many repeated pairings of paired stimuli (he used rats in a maze), would anticipate reinforcement. This anticipation ‘motivates’ the animal through the behavior chain from the instant the S^D is given. Hull observed the outward manifestation of the inner neural processes resulting from chemical activation in the brain (Wise, 2008).

We observed similar VS in both dolphins and belugas from widely divergent geographic sources and backgrounds. The feeding buzz was similar and common to all of these animals and all shifted the terminal part of the buzz to follow the SR during learning. Rehn et al. (Rehn et al., 2011) have reported a ‘V4 or excitement call’ from feeding killer whales (Orcinus orca). These killer whales were from separate clans off Canada, the Bering Sea and Kamchatka. The authors suggest that these calls are innate vocal expressions of motivation rather than acquired through social learning. Our VS is likely an expression of motivation as well as an innate expression of a basic reward system present in the brain of mammals. Our animals learned to associate the SR with reward. Most remarkable is that these cetaceans so frequently express reward expectation by vocalizing.

Reward results in brain dopamine release. Dopamine links to anticipatory, preparatory and approach phases of motivated behavior (Romo and Schultz, 1990; Ljungberg et al., 1992; Richardson and Gratton, 1996; Berridge and Robinson, 1998). Dolphins vocalize upon electrical stimulation of reward centers of their brains (Lilly, 1958), which contain dopamine neurons (Manger et al., 2004). We observed numerous forward shifts or transpositions of the terminal portion of a feeding buzz to follow a signal (SR) promising future reward. This forward-shifted food call that we have dubbed a VS is the best cetacean example of an acoustic signal of food reward expectation that was predicted for rats many years ago (Hull, 1951). Dolphins, belugas and many other cetaceans are highly vocal animals. Therefore, the character, timing and context of their sounds may reveal more about their emotional states and about the function of their sounds in communication.

The animals employed in this study (Table 1) were from different cetacean families. Dolphins were of the genus Tursiops from the family Delphinidae, and belugas, genus Delphinapterus, were from the family Monodontidae. They were also from different sources and had different backgrounds. Native hunters collected the two belugas (MUK and NOC) from Hudson Bay, Canada. One bottlenose dolphin, IAY, was from a Pacific population; dolphin BRT was from the Gulf of Mexico; SAY was born at our facility in 1979 from parents of Gulf of Mexico stock; NAY was born at our facility in 1983 sired by IAY and a dam from the Gulf of Mexico (not recorded here). The animals in these experiments were under the continuous care of the US Navy Marine Mammal Program. All experiments were approved by the Program Institutional Animal Care and Use Committee.

We used an experimental approach (Skinner, 1957) to train animals for different projects during which the VS was observed. Our observations were from 12 bottlenose dolphins and two belugas. Initially we reward (S^R+) as close as possible to the time of the desired action. A discriminative stimulus or S^D is introduced so that the desired action occurs at the desired time. The S^D for our experiments is a sound or a trainer's hand signal, which begins a chain of animal responses. When the animal responds correctly, at the end of the response chain, a secondary reinforcer (SR) is produced to bridge the time gap until S^R+ can be given. As training proceeds, intervals between S^D, SR and S^R+ can be greatly extended. With repetition, the animal develops the strong expectation that the SR will result in the S^R+, a fish reward.

The goal of this training was to answer scientific questions about each individual animal. How deep can the individual dive (Ridgway et al., 1969; Ridgway et al., 1984)? What is the hearing threshold at various frequencies and depths (Ridgway et al., 2001)? How is hearing affected by anthropogenic sound and background noise (Schlundt et al., 2000)? How fast can the individual select different vocal responses to different sound stimuli (Ridgway, 2011)?

There were four different S^D–SR combinations. In the first condition, the S^D was an underwater tone that signaled the animal to dive to depth in the open sea (Ridgway et al., 1969; Ridgway et al., 1984; Ridgway et al., 1991a). At depth, the animal pushed a plunger at the end of a long cable switching off the continuous tone S^D. An LC-10 hydrophone situated just above the plunger allowed us to monitor when the animal turned the tone off and to hear any animal sounds (Fig. 1). In the other three conditions, the S^D was a hand signal. The trainer simply pointed to an underwater station, and this started a chain of acoustic queries and responses. On the last correct response of the chain, the trainer activated a SR, which was an air whistle, an underwater tone or an underwater buzzer.

Here we studied the timing of the response to the underwater buzzer in four bottlenose dolphins and two belugas. We compared latency of trained vocal responses to stimuli not indicating reward [reaction time (RT)] with latency of VS following SR. For RT trials, three of the animals (MUK, SAY and IAY) responded with a whistle to a tone stimulus and a pulse burst to a pulsed stimulus. The other three animals (NOC, NAY and BRT) responded with a whistle to a pulsed stimulus and a pulse burst to a tone stimulus. To start a trial of auditory/vocal RT, the trainer uses a hand signal S^D to point the animal to the test station 1 m underwater. When the animal is on station and quiet, the trainer presses a switch to begin the computer program sending out stimuli in random order (Fig. 6). The system presented sound stimuli as long as the trainer held the switch button down. The trainer could give several stimuli in a row in the randomly variable sequence from the computer file, then let up on the switch and interpose a period without stimuli. The animal had to remain on station and quiet until the program presented a sound stimulus or until it heard the SR. The animal returns to the surface for reward on hearing the SR.

The computer program selects stimulus parameters from a table stored on a computer disk. Stimuli are presented by the program in a pre-determined, randomized order with an inter-stimulus interval varying from 1.2 to 2.1 s in 0.1 s random increments. A programmable Wavetek (model 154; Wavetek, San Diego, CA, USA) generated the stimuli through an underwater projector (LC-10 or F42B, US Navy Underwater Sound Laboratory, Orlando, FL, USA) 1 m under the water surface and 1 m in front of the dolphin's blowhole (Ridgway, 2011). A hydrophone (B&K8103, Bruel and Kjær, Denmark) received signals and responses. Custom software digitized the sound and stored it on the computer disk with all relevant stimulus parameters.

The computer displayed each digitized stimulus and response epoch on completion of the RT session. Editors recognized the animal's pulse burst or whistle response placing a cursor at the beginning of the response. The program calculated RT from stimulus onset to vocalization onset. The RT value was stored in a database along with all relevant stimulus parameters. During the trials, all sound was monitored continuously by hydrophones (B&K 8103 with a B&K charge amplifier) and a Racal tape 7D instrumentation recorder. Tape recordings were digitized by a National Instruments PCIMIO-16 E-1 multifunction board (National Instruments, Austin, TX, USA) and saved as binary files. Later we converted the binary files to WAV files using Audacity, a free online software program. Audacity files were imported into Adobe Audition to measure VS latency (onset of the SR to onset of VS).

Response latencies to the underwater SRs were compared (Table 1, Fig. 7) with trained responses to underwater signals of similar amplitude using the methodology described above and by Ridgway (Ridgway, 2011). To compare latency of the VS with trained response times among the study animals, a paired (repeated measures) t-test was applied by species using SAS version 9.1 (SAS Institute, Cary, NC, USA). Significance was defined as a P-value less than 0.05.

Many individuals of the Navy Marine Mammal Program trained animals for experiments cited here. We especially thank Trisha Kamolnick and Mark Todd for training and technical support. Brian Branstetter, Stephanie Venn-Watson, Kevin Carlin, Chris Hammell and Kelly McColl helped with figures and analysis. Allen Goldblatt, Mark Xitco and Stan Kuczaj made helpful suggestions on the manuscript.