Setting

Our model will describe psychological experiments that have the following decision-making scenario. The scenario starts at the initial state S₀ on which the participant bases their action A₀, which brings in a numerical reward R₁. Consequently, the participant finds itself in the next state S₁ and selects another action A₁, which brings in a numerical reward R₂ and state S₂. This process then repeats until the participant makes T decisions, yielding a sequence of observations collected for each participant of the form:

Above, observations fall into three types on a given trial t: the state that the participant visits, denoted by S_t, the action that the participant takes when visiting state S_t, denoted by A_t, and the subsequent reward, R_t+1, that a participant receives upon visiting state S_t and taking action A_t. For simplicity, let us assume that both the space of possible states $\mathcal{S}$M24 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\mathcal{S}} \] \end{document} and the space of possible actions $\mathcal{A}$M16 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\mathcal{A}} \] \end{document} are discrete. The space of possible rewards $\mathcal{R}$M17 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\mathcal{R}} \] \end{document} can be any subset of the real line ℝ. Further, assume the experiment defines subsequent rewards and states as a function of the current state and action according to a Markov transition probability

An experiment described above constitutes a (discrete-time, discrete-state) Markov Decision Process (MDP).

Temporal difference (TD) learning

In the setting described above, human decision-making is often modeled using TD learning. One widely-known algorithm for TD learning is called Q-learning, so-named for its explicit use of a state-action value function denoted by Q. This algorithm supposes that the agent, i.e., the participant in a psychological experiment, tries to learn the “value” of their actions as a function of a given state in terms of future rewards. This notion gives rise to a state-action value function Q(s,a) mapping states s ∈ $\mathcal{S}$M18 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\mathcal{S}} \] \end{document} and actions a ∈ $\mathcal{A}$M20 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\mathcal{A}} \] \end{document} to a real number that reflects the value of this state-action pair. A Q-learner updates this state-action function according to their experiences:

Here, the learner has just taken action A_t in state S_t, receiving the immediate reward R_t+1 and transitioning to a new state S_t+1. A learning rate α accounts for the extent to which the new information, i.e. their reward and the new state-action value, overrides old information about their state-action value function. For instance, one can see that if α = 0, there is no overriding - the estimate stays the same. The discount parameter γ weighs the impact of future rewards. A discount parameter γ = 0 would mean the learner does not care about the future at all, while γ = 1 would mean the learner cares about the sum total of future rewards (which may even cause the algorithm to diverge).

Risk-sensitive TD learning

A variant of the Q-learner allows a learner to be particularly sensitive to smaller, or more negative, rewards, i.e. risky situations. In particular, a risk-sensitive Q-learner weighs the prediction error, which is given by

differently depending on whether the prediction error is positive or negative. This yields the following update:

The parameter k controls the degree to which the learner is risk sensitive. If k = 0, then the learner weighs positive and negative prediction errors equally, in which the updates are the same as before and we say the learner is risk-neutral. If k < 0, then negative prediction errors are weighed more than positive prediction errors. In this case, smaller rewards have a stronger influence relative than larger rewards on the state-action value function Q, resulting in a learner who is considered risk-averse. Similarly if k > 0, the reverse is true: larger rewards have a stronger influence relative to smaller rewards, and the learner is considered risk-seeking.

A learning model with competing critics

With the introduction of risk-sensitive TD learning, we can consider a range of learning behaviors from risk-sensitive to risk-seeking, all modulated by parameter k and reflected in the state-action value function Q. Researchers are often focused on how pessimism or risk-sensitivity, substantiated by k, might vary between individuals. In our model, however, we investigate how risk-sensitivity might vary within individuals. Specifically, we consider two learning systems, one pessimistic (risk-adverse) and one optimistic (risk-seeking).

Our model captures two competing critics by keeping track of two state-action value functions, Q⁺ and Q^–, and updated each function according to:

with prediction errors given by

For simplicity, we initialize Q⁺ and Q^– to zero. Parameters k⁺,k^– are assumed to lie in [0, 1]. Large k⁺ controls the degree to which the learner is risk-seeking and k^– controls the degree to which the learner is risk-sensitive. It is important to point out that we are also not the first to consider multiple risk-sensitive TD learning systems. This idea was recently put forth in (Dabney et al., 2020), where multiple risk-sensitive TD learning systems were thought to be encoded in multiple dopamine neurons. We are also not the first to consider dual competing systems (Collins & Frank, 2014; Daw et al., 2002; Mikhael & Bogacz, 2016; Montague et al., 2016). In the opposing actor learning model in (Collins & Frank, 2014), for example, prediction error from a single learning system controls the dynamics of G (“go”) and N (“no-go”) systems, which in turn are combined linearly to determine decisions. Since it may not be obvious why this model differs from our proposed model, we discuss in the Supplement how the update equations of the two models differ in important ways, resulting in significantly different behaviors and predictions. Similarly, the Supplement also explores differences between our proposed model and a SARSA version of the model as well as a risk-sensitive TD learning model.

A decision-making model with competing critics

Now that we have a model of learning, namely Q⁺ and Q^–, it is sensible to consider how the agents makes decisions based on what they have just learned. This means that the individual has to make the decision of choosing from the available actions, having obtained pessimistic and optimistic estimates for action-value pairs.

A naive approach is what is called the greedy method, meaning that the action with the highest value is chosen. This approach, however, does not account for actions with multiple values (e.g., optimistic and pessimistic values) nor does it allow the individual to do any exploration, during which they might discover a more optimal strategy. A way to incorporate exploration into decision-making is to act greedy 1 – ɛ of the time and for ɛ of the time, the individual explores non-greedy action with equal probabilities. This method referred to as ɛ-greedy and is used by our model.

To integrate multi-valued actions into a ɛ-greedy method, our model supposes that a random variable U_t(a) is selected for each action a uniformly from the interval [Q^–(S_t,a), Q⁺(S_t,a)], whenever an individual has to make a decision in state S_t. Then whenever the individual acts greedily, they select the action A_t that maximizes U_t(a). These decision rules along with learning models comprise Competing-Critics model, which is summarized in Algorithm 1. While we use an ɛ-greedy method, exploration could also be achieved by applying a soft-max function to transform U_t(a) into a probability and select action a according to this probability.

Learning the shape of rewards

Let us first focus on learning behavior by considering the simple case of trivial state and action spaces: $\mathcal{S}$M23 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\mathcal{S}} \] \end{document} = {1} and action $\mathcal{A}$M25 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\mathcal{A}} \] \end{document} = {1}. In this case, learning in the Competing-Critics model is determined completely by the distribution of rewards R_t. We considered what an individual would learn given four different Pearson distributions of R_t, with varying mean μ, standard deviation σ, and skew, while kurtosis was fixed at 2.5. For reference, we also consider the classic Q described at Eq. (1).

Figure 2 illustrates what an individual with balanced parameters, (k⁺, k^–) = (0.9, 0.9), learns over 100 trials. For comparison, we also simulated a traditional, risk-neutral Q learning model by setting k⁺= k^– = 0. Solid dark lines denote state-action value function averaged over simulations and shaded regions represent associated interquartile ranges (IQRs) for each function. One can immediately notice several things. By design, the optimistic value function Q⁺ is on average larger than the neutral value function Q, which is larger than the average pessimistic value function Q^–. In addition, the distribution of each value function appears to converge and can capture shifts in mean rewards μ and scaling of the standard deviation σ. Specifically, the long-term relationship between Q⁺,Q^– and Q is preserved when μ is shifted from 0.5 to 0.25, whereby all value functions shift down by about 0.25. Further, the gap between Q⁺ and Q^– is halved when σ is halved from 0.2 to 0.1; each IQR is also halved. Meanwhile, Q⁺ and Q^– are roughly symmetric around the Q when the reward distribution is symmetric (i.e. zero skew), so that the average of Q⁺ and Q^– is approximately Q. However, moving skew from 0 to 1 is reflected in both the gap between Q⁺ and Q, which lengthens, and the gap between Q^– and Q, which shortens.

Remarkably, the relationship Q⁺ > Q > Q^– is also present within a single simulation run (Figure 3). Intuitively, this makes sense because they capture the behaviors of risk-seeking, risk-neutral and risk-sensitive agents, respectively and it turns out that this ordering can be preserved provided k^± are neither too small or large. See Supplement for the proof of this result. Furthermore, the last subplot also illustrates that introducing a positive skew to the reward distribution R_t, also causes the distribution of Q^± and Q to also have positive skew.

Value functions Q⁺ and Q^– are not only modulated with the reward distribution, but also parameters k^±. Increasing k⁺ moves Q⁺ in a positive direction away from the risk-neutral value function Q, whereas increasing k^– moves Q^– in a negative direction away from the risk-neutral value function Q. With k⁺ pulling Q⁺ in one direction and k^– pulling Q^– in the opposite direction, the midpoint of Q⁺ and Q^– is largely influenced by the gap in k^– and k⁺ (Figure 4A). Meanwhile, the gap between Q⁺ and Q^– is largely influenced by the midpoint of k⁺ and k^–.

Thus, while k⁺ and k^– are the two natural parameters of the learning process, the difference in how agents make choices is well described by a 45° rotation of these coordinates, yielding axes s_r = k⁺ – k^– and s_u = k⁺ + k^–. As visualized in Figure 4B, we refer to the s_r and s_u axes as the risk-sensitivity and uncertainty-sensitivity axes, respectively. These two axes provide orthogonal ways of interpreting and comparing different reward distributions, as in Figure 5. Namely, risk-sensitivity, which can vary from risk-averse to risk-seeking, captures a learner’s bias either against losses or towards gains, and is instantiated in our model as the difference between $\frac{1}{2}(Q^{+}+Q^{-})$M13 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\textstyle{1 \over 2}}({Q^ + } + {Q^ - }) \] \end{document} and the expected reward. In contrast, uncertainty-sensitivity, which can vary from decisive to deliberative, captures a learner’s consideration of actions with large standard deviations in rewards. In our model, this uncertainty-sensitivity is instantiated as the size of the interval between Q^– and Q⁺, wherein the larger that interval, the more likely two actions with similar values of $\frac{1}{2}(Q^{+}+Q^{-})$M14 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\textstyle{1 \over 2}}({Q^ + } + {Q^ - }) \]2 \end{document} are to be seen as competing, viable choices.

While increasing uncertainty-sensitivity can increase the variety of actions that a learner makes, it is distinct from the standard use of an exploration parameter ∈. An exploration parameter ∈ forces the exploration of all possible actions, and is included to ensure that no action is left unexplored. By contrast, uncertainty-sensitivity is a preference axis, and it only encourages the exploration of competitive actions whose intervals overlap with the action with the largest value $\frac{1}{2}(Q^{+}+Q^{-})$M15 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\textstyle{1 \over 2}}({Q^ + } + {Q^ - }) \] \end{document} . The preference aspect of uncertainty-sensitivity is especially clear in cases where many actions with high variance rewards are considered against a single reliable action with a fixed outcome (no variance) and a slightly higher expected reward. In such a setting, a deliberative learner may often pick the high variance actions even though they could correctly report that the fixed outcome had a better expected outcome (by contrast, a risk-seeking learner would report the high variance actions as having better outcomes). Indeed, while both risk-sensitivity and uncertainty-sensitivity can describe why a learner might prefer a high variance reward to a fixed reward with slightly higher expected return, both are required to explain why some learners might exclusively choose the high variance action while some others sample both the high variance action and the fixed outcome. Similarly, the difference between uncertainty and risk-sensitivity can affect the choices when a fixed outcome would preferred, as also illustrated in Figure 5.

In summary, parameters k^± can capture a range of behavior from being too risky to not risky enough and from too decisive to too deliberative. We demonstrate these decision-making behaviors in the next two examples.

Capturing a penchant for gambling

To demonstrate how parameters k^± drive decision-making behavior in our model, let us consider the Iowa Gambling Task (IGT), which asks participants to repeatedly chose between four decks labeled A to D. After each choice, they gain and/or lose monetary rewards. Hundreds of studies have used the IGT to evaluate human decision-making (Chiu, Huang, Duann, & Lin, 2018). Initial findings found healthy controls would learn to select “good” decks (Decks C and D), so-called because, on average, they yielded a net gain (Bechara, Damasio, Damasio, & Anderson, 1994). By contrast, individuals with a damaged prefrontal cortex would continue to select “bad” decks (Decks A and B) despite yielding net losses on average. Selecting bad decks was put forth as a marker of impaired decision-making, or more specifically, an insensitivity to future consequences. This interpretation, however, presumes that the participant’s objective is indeed to make decisions that maximize expected rewards as opposed to making decisions that seeks large gains or avoids large losses. Risk-seeking behavior (i.e. a penchant for gambling), in particular, may encourage individuals to pursue bad decks, since they yield the largest one-time gains.

The IGT can be placed with our MDP framework with A_t ∈ {A,B,C,D} capturing the selected desks, S_t ∈ {1} capturing a trivial case with only one state, and R_t capturing the summed gain and loss per trial. In particular, we will simulate R_t as independent draws from a distribution that depends on the selected deck and matches characteristics described in the Supplement. For example, R_t is drawn uniformly from {$50, $0} when Deck C is selected.

To that point, balanced (k⁺,k^–) = (0.9,0.9) parameters, reflecting risk-neutral behavior, results in a preference for Deck C, i.e. one of the good decks that leads to average net gains (Fig 6A). By contrast, imbalanced (k⁺,k^–) = (0.9,0.1) parameters, reflecting risk-seeking behavior, results in a preference for Deck B, i.e. one of the bad decks that leads to average net losses. In each case, pessimistic state-action values Q^– are larger for good decks (C and D), correctly signifying that these decks are the more risk-averse choices (Figure 6B). Meanwhile, optimistic state-action values Q⁺ are larger for bad decks (A and B), correctly signifying that these decks are the more risk-seeking choices. Imbalanced k^± parameters, however, dramatically underplays the risk of Deck B compared to balanced risk-sensitive parameters. Consequently, the chance of large gains encoded in Q⁺ is suitably enticing to encourage a Deck B preference. That is, Deck B preference, which is actually a well-known phenomenon of healthy participants (Chiu et al., 2018), can be interpreted as a penchant for gambling rather than an insensitivity to future consequences.

As was done in Steingroever, Wetzels, and Wagenmakers (2013), we can also partition the parameter space {(k⁺, k^–)| 0 ≤ k⁺, k^– ≤ 1} by preference for good and bad decks (Fig 6C). This figure tells us that in the “blue” region of the parameter space, bad decks A, B are selected at greater frequency than good decks C, D. In the context of risk-seeking vs risk-averse terminology, our choice of k⁺ >> k^– means that our learner, despite the fact that B incurs incomparably large loss, keeps sticking to it because Q⁺ is driving the choice. In another words, our agent is unable to learn the good decks in the IGT, thus mimicking the behaviors of the participants with prefrontal cortex damage as demonstrated in Lin, Chiu, Lee, and Hsieh (2007).

The ambiguity of deliberation

One of the main conceptual insights of having two orthogonal axes of risk and uncertainty-sensitivities is that it can describe a greater variation in the types of decisions that people might make (or prefer to make) and thus allows for alternate interpretations of some experiments. To illustrate this, consider the 2-stage Markov task (Daw, Gershman, Seymour, Dayan, & Dolan, 2011), in which a participant repeatedly selects images over two stages and where the experiment was explicitly designed in order to probe the difference between model-free and model-based learning.

In the 2-stage Markov task, participants are presented one of three pairs of images at a given stage. At the first stage, all participants are shown the first pair of images and have the option to choose either the left or right image. After choosing an image, participants are shown the second or third pair of images, with the pair selected randomly according to probabilities that depends on their first stage selection. Participants then select an image on the second stage and receive monetary rewards. This task is used in experiments to determine the degree to which individuals are learning about the common (p = 0.7) versus rare (p = 0.3) transition associated with each action in stage 1. To mark this type of learning, the authors point to the probability of staying on the same first-stage decision (i.e. repeating the same first stage decision on a subsequent trial) depending on the type of transition (common vs. rare) and whether or not the person was rewarded on the second stage. In particular, the authors predicted that stay percentages of a model-free learner would differ only based on reward, while a model-based learner’s stay percentage would differ only based on whether the first transition was common or rare. In fact, the data showed that participants’ stay percentage varied by both reward and the transition type. Since neither model predicted this reward-transition interaction, the authors stated that both model-free and model-based learning are occurring.

By contrast, we believe that the observed difference in stay percentages can be well captured by our model, and that the relevant difference between the common and rare stay percentage may be capturing uncertainty-sensitivity. We model the two-stage Markov task as follows. The two-stage Markov task has actions A_t ∈ {left, right} representing selected images, states S_t ∈ {1,2,3} capturing presented image pairs, and rewards R_t capturing rewards after image selection with rewards after the first stage set to zero. Here t counts the total number of actions. That is, t = 0 corresponds to the first time that a participant takes an action in the first stage, and t = 1 corresponds to the first time that a participant takes an action in the second stage. For our model-free model to capture reward-transition interactions, we do not distinguish between first and second stage decisions, using the same model update regardless of the decision stage. This approach effectively treats the switch from second to first stage as a state transition. To allow information to pass from between stages, we use a discount factor γ of 0.9. By contrast, the model-free model in Daw et al. (2011) uses different updates for first and second stage decisions and does not treat the switch from second to first stage as a state transition. Rather, they view the second stage decision as a state transition to a dummy terminal state, and subsequently rely on an eligibility trace to pass information from second to first stage decisions.

The bar graphs in Figure 7 represents the probability in our model of competing critics of sticking to the current choice categorized by whether it resulted in reward or not and whether the transition was common or rare. The plots on the right side of the figure tells us the difference between the probabilities of staying when the transition was common or rare, given rewarded or unrewarded.

As displayed in Figure 7, the characteristic pattern observed in (Daw et al., 2011), where stay percentage depends on both rewarded/unrewarded and the common/rare transition is present with the same trends. Moreover, the degree to which there is a common/rare difference is determined by the parameters along the uncertainty-sensitivity axis (s_u = k^– + k⁺). Namely, when s_u is large, k^– = k⁺ = 0.9, then the model stay percentage is only slightly affected by the reward and the transition, reflecting a more deliberative sampling of actions resulting in less immediate correlations between actions in one trial and subsequent actions in the next. On the other hand, when s_u is small, k^– = k⁺ = 0.1, the empirically observed dependence on rewarded/unrewarded and common/rare is increased. Meanwhile, the risk-sensitivity axis does not appear correlated with the rare-common stay percentage difference.

While the characteristic pattern of stay percentages can by reproduced by varying parameters k^± along the uncertainty-sensitivity axis, it can also be reproduced in other ways. Notably, the models used in Daw et al. (2011) show that the characteristic pattern can be reproduced by varying the degree to which their model-based model is used over their model-free model. In addition, a person’s tendency to explore decisions, as reflected in the exploration parameter ∈, could also increase or decrease stay probabilities in our model. In other words, it is difficult to disambiguate a change in how deliberative a person is with their decisions from their ability to learn transitions or their tendency to explore.

A possible connection to reaction time

Our conceptualization of the Competing-Critics model assumes that the translation of state-action value functions Q⁺ and Q^– into decisions plays out in time, whereby Q⁺ and Q^– determine not only which decisions are made, but also the time until the decision is made, i.e. the reaction time. For example, we hypothesize that Q⁺ signals the time at which an action is a viable option to a learner, so that decisions with larger Q⁺ are considered earlier. Meanwhile, Q^– signals the time at which an action is no longer a viable option.

One way to explicitly connect our model to reaction time is to introduce some strictly decreasing function F, e.g., F(x) = exp(-bx), that transforms U_t(a), which is on the same scale as rewards R_t, to a temporal scale. On trials that the learner behaves greedily (i.e. does not explore), the reaction time could be modeled as min_a F(U_t(a)) with arg min_a F(U_t(a)) determining which action is selected. The probability of selecting a would be left unchanged, since F strictly decreasing implies that

With the introduction of F, there is a one-to-one relationship between reaction time and max_a U_t(a). Thus, we can learn about reaction times by simulating max_a U_t(a) for the learning example with μ = 0.5, σ = 0.2, and skew = 0; the IGT; and the two-stage Markov task (Figure 8).

In all three examples, the mean of max_a U_t(a) varies primarily along the risk-sensitivity axis, with larger values found near (k⁺,k^–) = (1,0) and smaller values found near (k⁺,k^–) = (0,1). Thus, we would hypothesize that an individual who is risk-seeking would have faster reaction times than an individual that is risk-averse. The standard deviation of max_a U_t(a), however, does not enjoy a consistent trend. When there is one option available, as in the learning example (Figure 8A), the standard deviation of max_a U_t(a) varies primarily along the uncertainty-sensitivity axis, with larger values found near (k⁺,k^–) = (1,1) and smaller values found near (k⁺,k^–) = (0,0). This makes sense since the interval (Q⁺(a),Q^–(a)), from which U_t(a) is drawn, lengthens when (k⁺,k^–) moves towards (1,1). Therefore in this learning example, greater deliberation (i.e. consideration of multiple actions) would not correspond with longer reaction times as one might expect, but rather with greater variability in reaction times. This connection falls apart when there are multiple competing options, with the standard deviation of max_a U_t(a) varying primarily along the k^– axis in the IGT and along the k⁺ in the two-stage Markov task (Figure 8B–C). Thus, we hypothesize that the type of learner who would experience greater variability in reaction times will depend on the task.

Alternatively, our model can be modified to include sequential sampling models, which describe reaction times as first passage times out of some specified region of certain stochastic processes such drift-diffusion models. (Fontanesi, Gluth, Spektor, & Rieskamp, 2019; Kilpatrick, Holmes, Eissa, & Josić, 2019; Lefebvre, Summerfield, & Bogacz, 2020; Veliz-Cuba, Kilpatrick, & Josic, 2016). One possibility is to specify a sequential sampling model for each competing action a and select actions according to which corresponds with the fastest first passage times. If one wanted to keep reaction times equal to F(U_t(a)) and actions selected according to the same probability as our model, then this model would need to be constructed implicitly, so that first hitting times have the same distribution as F(U_t(a)) with F defined above. Otherwise, a preferred sequential sampling model could be specified and state-action values Q^± used to modulate properties (e.g., drift rate) of this model. This is a common strategy when integrating TD learning with a sequential sampling model.

Neural encoding of updates

As we mentioned, the rough intuition behind the reinforcement learning update we chose for the state-value functions Q⁺ and Q^– is that they capture the behaviors of risk-seeking and risk-averse learners, respectively. Going even further, we investigate the possibility that dopamine transients encode the update ΔQ⁺ associated with the risk-seeking system and serotonin transients encode the negative of the update ΔQ^– associated with the risk-averse system. In view of this claim, we present one last study, which measured dopamine and serotonin during a decision-making task (Moran et al., 2018).

In this study, participants were asked to make investing decisions on a virtual stock market. In total, participants made 20 investment decisions for 6 markets for a total of 120 decisions. Each participant was allocated $100 at the start of each market and could allocate bets between 0% to 100% in increments of 10%. The participant would gain or lose money depending on their bet. Given a bet A_t on trial t and market value p_t+1 after betting, percent monetary gain (or loss) on trial t was

To model this experiment, we use the simplifying assumption that bets are low or high: A_t = {25%, 75%}, and suppose rewards are

Actions are centered to 50% to account for the hypothesized role of counterfactual in this experiment (Kishida et al., 2016). Hence, R_t is the percent monetary gain relative to the counterfactual gain were a neutral 50% bet made. Following (Moran et al., 2018), trials are split according to a reward prediction error (RPE): the percent monetary gain centered to the mean of its past values and inversely scaled by the standard deviation of its past values.

Let us consider the scenario where a decision made on trial t resulted in a negative RPE, which means the agent has a lower monetary gain relative to expected past gains (Figure 9A). Without accounting for counterfactuals, a risk-neutral system would experience a negative update independent of bet level. Risk-seeking update ΔQ⁺, however, depends on bet level during a negative RPE: large for a low bet (25%) compared to a high bet (75%). The reverse is true for the negative of the risk-averse update ΔQ^–: it is large for a high bet (75%) compared to a low bet (25%).

These characteristics of ΔQ⁺ and -ΔQ^– during negative RPE mirror, respectively, dopamine and serotonin transients in (Moran et al., 2018). The authors hypothesized that the large dopamine transient for a low bet encourages the rewarding decision of betting low, whereas the large serotonin transient for a high bet protects the individual from the risky decision of betting high. Betting low is only rewarding when compared to the counterfactual loss of betting a higher amount and losing. This hypothesis is consistent with the role in the Competing-Critics model of a positive ΔQ⁺ to encourage a rewarding decision and a negative ΔQ^– to protect oneself from risky decisions.

When RPE is positive, which is when the agent has a higher monetary gain relative to expected past gains, the direction of the updates flip. The update ΔQ⁺ is now large for a high bet (75%) compared to a low bet (25%), and the negative of ΔQ^– is large for a low bet (25%) compared to a high bet (75%). Again, these characteristics mirror dopamine and serotonin transients in (Kishida et al. 2016; Moran et al., 2018). In this case, it was hypothesized that the relatively large dopamine transient for a high bet encourages the rewarding decision of betting high, whereas the relatively large serotonin transient for a low bet protects the individual from the risky decision of betting low. As before, betting low is only considered risky when compared to the counterfactual loss of what they could have gained if they bet higher.

As an aside, we point out that average updates ΔQ⁺ and -ΔQ^– are generally more positive than they are negative. This asymmetry respects the fact that dopamine and serotonin transients have a biophysical constraint whereby positive transients are easily induced but negative transients are not.

Following (Moran et al., 2018), we consider how updates ΔQ⁺ and -ΔQ^– influence how a person subsequently bets (Figure 9B–C). Trials are split further based on the subsequent decision made on the next trial. The negative of update ΔQ^– is largest when switching from a high to low bet during negative RPE and from a low to high bet during positive RPE. These trends mirror serotonin transients in (Moran et al., 2018), where a relatively large serotonin transient preceded a lowering of a bet when RPE was negative and preceded a raising or holding of a bet when RPE was positive. These findings provided further support that serotonin transients protect an individual from actual and counterfactual losses.

Meanwhile, the update ΔQ⁺ is largest when keeping a bet low during negative RPE and when keeping a bet high during positive RPE. Since dopamine transients were not investigated as a function of subsequent bets in (Moran et al., 2018), we have the following hypothesis: a relatively large dopamine transient reinforces a low bet when RPE was negative and reinforces a high bet when RPE was positive.