<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD with MathML3 v1.2 20190208//EN" "JATS-archivearticle1-mathml3.dtd"> <article xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article" dtd-version="1.2"><front><journal-meta><journal-id journal-id-type="nlm-ta">elife</journal-id><journal-id journal-id-type="publisher-id">eLife</journal-id><journal-title-group><journal-title>eLife</journal-title></journal-title-group><issn pub-type="epub" publication-format="electronic">2050-084X</issn><publisher><publisher-name>eLife Sciences Publications, Ltd</publisher-name></publisher></journal-meta><article-meta><article-id pub-id-type="publisher-id">64620</article-id><article-id pub-id-type="doi">10.7554/eLife.64620</article-id><article-categories><subj-group subj-group-type="display-channel"><subject>Research Article</subject></subj-group><subj-group subj-group-type="heading"><subject>Neuroscience</subject></subj-group></article-categories><title-group><article-title>Valence biases in reinforcement learning shift across adolescence and modulate subsequent memory</article-title></title-group><contrib-group><contrib contrib-type="author" id="author-217149"><name><surname>Rosenbaum</surname><given-names>Gail M</given-names></name><contrib-id authenticated="true" contrib-id-type="orcid">https://orcid.org/0000-0002-6306-0508</contrib-id><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund4"/><xref ref-type="fn" rid="con1"/><xref ref-type="fn" rid="conf1"/></contrib><contrib contrib-type="author" id="author-217150"><name><surname>Grassie</surname><given-names>Hannah L</given-names></name><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="fn" rid="con2"/><xref ref-type="fn" rid="conf1"/></contrib><contrib contrib-type="author" corresp="yes" id="author-147430"><name><surname>Hartley</surname><given-names>Catherine A</given-names></name><contrib-id authenticated="true" contrib-id-type="orcid">https://orcid.org/0000-0003-0177-7295</contrib-id><email>cah369@nyu.edu</email><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="aff" rid="aff2">2</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="other" rid="fund3"/><xref ref-type="other" rid="fund5"/><xref ref-type="fn" rid="con3"/><xref ref-type="fn" rid="conf2"/></contrib><aff id="aff1"><label>1</label><institution>Department of Psychology, New York University</institution><addr-line><named-content content-type="city">New York</named-content></addr-line><country>United States</country></aff><aff id="aff2"><label>2</label><institution>Center for Neural Science, New York University</institution><addr-line><named-content content-type="city">New York</named-content></addr-line><country>United States</country></aff></contrib-group><contrib-group content-type="section"><contrib contrib-type="editor"><name><surname>Schlichting</surname><given-names>Margaret L</given-names></name><role>Reviewing Editor</role><aff><institution>University of Toronto</institution><country>Canada</country></aff></contrib><contrib contrib-type="senior_editor"><name><surname>Frank</surname><given-names>Michael J</given-names></name><role>Senior Editor</role><aff><institution>Brown University</institution><country>United States</country></aff></contrib></contrib-group><pub-date date-type="publication" publication-format="electronic"><day>24</day><month>01</month><year>2022</year></pub-date><pub-date pub-type="collection"><year>2022</year></pub-date><volume>11</volume><elocation-id>e64620</elocation-id><history><date date-type="received" iso-8601-date="2020-11-04"><day>04</day><month>11</month><year>2020</year></date><date date-type="accepted" iso-8601-date="2021-12-24"><day>24</day><month>12</month><year>2021</year></date></history><permissions><copyright-statement>© 2022, Rosenbaum et al</copyright-statement><copyright-year>2022</copyright-year><copyright-holder>Rosenbaum et al</copyright-holder><ali:free_to_read/><license xlink:href="http://creativecommons.org/licenses/by/4.0/"><ali:license_ref>http://creativecommons.org/licenses/by/4.0/</ali:license_ref><license-p>This article is distributed under the terms of the <ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution License</ext-link>, which permits unrestricted use and redistribution provided that the original author and source are credited.</license-p></license></permissions><self-uri content-type="pdf" xlink:href="elife-64620-v1.pdf"/><abstract><p>As individuals learn through trial and error, some are more influenced by good outcomes, while others weight bad outcomes more heavily. Such valence biases may also influence memory for past experiences. Here, we examined whether valence asymmetries in reinforcement learning change across adolescence, and whether individual learning asymmetries bias the content of subsequent memory. Participants ages 8–27 learned the values of ‘point machines,’ after which their memory for trial-unique images presented with choice outcomes was assessed. Relative to children and adults, adolescents overweighted worse-than-expected outcomes during learning. Individuals’ valence biases modulated incidental memory, such that those who prioritized worse- (or better-) than-expected outcomes during learning were also more likely to remember images paired with these outcomes, an effect reproduced in an independent dataset. Collectively, these results highlight age-related changes in the computation of subjective value and demonstrate that a valence-asymmetric valuation process influences how information is prioritized in episodic memory.</p></abstract><kwd-group kwd-group-type="author-keywords"><kwd>valence asymmetries</kwd><kwd>reinforcement learning</kwd><kwd>memory</kwd><kwd>adolescence</kwd><kwd>decision making</kwd><kwd>individual diferences</kwd></kwd-group><kwd-group kwd-group-type="research-organism"><title>Research organism</title><kwd>Human</kwd></kwd-group><funding-group><award-group id="fund1"><funding-source><institution-wrap><institution-id institution-id-type="FundRef">http://dx.doi.org/10.13039/501100003986</institution-id><institution>Jacobs Foundation</institution></institution-wrap></funding-source><principal-award-recipient><name><surname>Hartley</surname><given-names>Catherine</given-names></name></principal-award-recipient></award-group><award-group id="fund2"><funding-source><institution-wrap><institution-id institution-id-type="FundRef">http://dx.doi.org/10.13039/100000001</institution-id><institution>National Science Foundation</institution></institution-wrap></funding-source><award-id>1654393</award-id><principal-award-recipient><name><surname>Hartley</surname><given-names>Catherine</given-names></name></principal-award-recipient></award-group><award-group id="fund3"><funding-source><institution-wrap><institution>NYU Vulnerable Brain Project</institution></institution-wrap></funding-source><principal-award-recipient><name><surname>Hartley</surname><given-names>Catherine</given-names></name></principal-award-recipient></award-group><award-group id="fund4"><funding-source><institution-wrap><institution-id institution-id-type="FundRef">http://dx.doi.org/10.13039/100000026</institution-id><institution>National Institute on Drug Abuse</institution></institution-wrap></funding-source><award-id>F32DA047047</award-id><principal-award-recipient><name><surname>Rosenbaum</surname><given-names>Gail M</given-names></name></principal-award-recipient></award-group><award-group id="fund5"><funding-source><institution-wrap><institution-id institution-id-type="FundRef">http://dx.doi.org/10.13039/100000025</institution-id><institution>National Institute of Mental Health</institution></institution-wrap></funding-source><award-id>R01MH126183</award-id><principal-award-recipient><name><surname>Hartley</surname><given-names>Catherine</given-names></name></principal-award-recipient></award-group><funding-statement>The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.</funding-statement></funding-group><custom-meta-group><custom-meta specific-use="meta-only"><meta-name>Author impact statement</meta-name><meta-value>Relative to children and adults, adolescents placed greater weight on negative prediction errors during learning and these age-varying learning idiosyncrasies biased subsequent memory for information associated with valenced outcomes.</meta-value></custom-meta></custom-meta-group></article-meta></front><body><sec id="s1" sec-type="intro"><title>Introduction</title><p>Throughout our lives, we encounter many new or uncertain situations in which we must learn, through trial and error, which actions are beneficial and which are best avoided. Determining which behaviors will earn praise from a teacher, which social media posts will be liked by peers, or which route to work will have the least traffic is often accomplished by exploring different actions, and learning from the good or bad outcomes that they yield. Importantly, individuals differ in the extent to which their evaluations (<xref ref-type="bibr" rid="bib27">Daw et al., 2002</xref>; <xref ref-type="bibr" rid="bib35">Frank et al., 2004</xref>; <xref ref-type="bibr" rid="bib41">Gershman, 2015</xref>; <xref ref-type="bibr" rid="bib52">Lefebvre et al., 2017</xref>; <xref ref-type="bibr" rid="bib88">Sharot and Garrett, 2016</xref>) and their memories (<xref ref-type="bibr" rid="bib57">Madan et al., 2014</xref>, <xref ref-type="bibr" rid="bib58">Madan et al., 2017</xref>; <xref ref-type="bibr" rid="bib83">Rouhani and Niv, 2019</xref>) are influenced by good versus bad experiences. For example, consider a diner who has a delicious meal on her first visit to a new sushi restaurant, but on her next visit, the meal is not very good. A tendency to place greater weight on past positive experiences might make her both more likely to remember the good dining experience and more likely to return and try the restaurant again. In contrast, if the recent negative experience exerts an outsized influence, it may be more easily called to mind and she may forego another visit to that restaurant in favor of a surer bet. In this manner, asymmetric prioritization of past positive versus negative outcomes may render these valenced experiences more persistent in our memories and systematically alter how we make future decisions about uncertain prospects.</p><p>Understanding how experiential learning informs decision-making under uncertainty may be particularly important during adolescence, when teens’ burgeoning independence offers more frequent exposure to novel contexts in which the potential positive or negative outcomes of an action may be uncertain. Epidemiological data reveal an adolescent peak in the prevalence of many ‘risky’ behaviors that carry potential negative consequences (e.g., criminal behavior [<xref ref-type="bibr" rid="bib93">Steinberg, 2013</xref>], risky sexual behavior [<xref ref-type="bibr" rid="bib85">Satterwhite et al., 2013</xref>]). Moreover, consistent with proposals that adolescent risk taking might be driven by heightened sensitivity to rewarding outcomes (<xref ref-type="bibr" rid="bib15">Casey et al., 2008</xref>; <xref ref-type="bibr" rid="bib39">Galván, 2013</xref>; <xref ref-type="bibr" rid="bib89">Silverman et al., 2015</xref>; <xref ref-type="bibr" rid="bib92">Steinberg, 2008</xref>; <xref ref-type="bibr" rid="bib99">van Duijvenvoorde et al., 2017</xref>), several neuroimaging studies have observed that adolescents exhibit neural responses to reward that are greater in magnitude than those of children or adults (<xref ref-type="bibr" rid="bib12">Braams et al., 2015</xref>; <xref ref-type="bibr" rid="bib23">Cohen et al., 2010</xref>; <xref ref-type="bibr" rid="bib38">Galvan et al., 2006</xref>; <xref ref-type="bibr" rid="bib89">Silverman et al., 2015</xref>; <xref ref-type="bibr" rid="bib100">Van Leijenhorst et al., 2010</xref>). These findings suggest that as adolescents learn to evaluate novel situations through trial and error, positive experiences might exert an outsized influence on their subsequent actions and choices.</p><p>Reinforcement learning (RL) models mathematically formalize the process of evaluating actions based on their resulting good and bad outcomes (<xref ref-type="bibr" rid="bib94">Sutton and Barto, 1998</xref>). In such models, action value estimates are iteratively revised based on prediction errors or the extent to which an experienced outcome deviates from one’s current expectation. The magnitude of the resulting value update is scaled by an individual’s learning rate. Valence asymmetries in the estimation of action values can be captured by positing two distinct learning rates for positive versus negative prediction errors, leading to differential adjustment of value estimates following outcomes that are better or worse than one’s expectations. Importantly, an RL algorithm with such valence-dependent learning rates estimates subjective values in a ‘risk-sensitive’ manner (<xref ref-type="bibr" rid="bib61">Mihatsch and Neuneier, 2002</xref>; <xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref>). A learner with a greater positive than negative learning rate will, across repeated choices, come to assign a greater value to a risky prospect (i.e., with variable outcomes) than to a safer choice with equivalent expected value (EV) that consistently yields intermediate outcomes, whereas a learner with the opposite asymmetry will estimate the risky option as being relatively less valuable.</p><p>Outcomes that violate our expectations might also be particularly valuable to remember. Beyond the central role of prediction errors in the estimation of action values, these learning signals also appear to influence what information is prioritized in episodic memory (<xref ref-type="bibr" rid="bib34">Ergo et al., 2020</xref>). Past studies have demonstrated enhanced memory for stimuli presented concurrently with outcomes that elicit positive (<xref ref-type="bibr" rid="bib26">Davidow et al., 2016</xref>; <xref ref-type="bibr" rid="bib44">Jang et al., 2019</xref>), negative (<xref ref-type="bibr" rid="bib47">Kalbe and Schwabe, 2020</xref>), or high-magnitude (independent of valence) prediction errors (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>), suggesting that prediction errors can facilitate memory encoding and consolidation processes. The common role of prediction errors in driving value-based learning and facilitating memory may reflect, in part, a tendency to allocate greater attention to stimuli that are uncertain (<xref ref-type="bibr" rid="bib28">Dayan et al., 2000</xref>; <xref ref-type="bibr" rid="bib70">Pearce and Hall, 1980</xref>). However, it is unclear whether idiosyncratic valence asymmetries in RL computations might give rise to corresponding asymmetries in the information that is prioritized for memory. Moreover, while few studies have explored the development of these interactive learning systems, a recent empirical study observing an effect of prediction errors on recognition memory in adolescents, but not adults (<xref ref-type="bibr" rid="bib26">Davidow et al., 2016</xref>), suggests that the influence of RL signals on memory may be differentially tuned across development.</p><p>In the present study, we examined whether valence asymmetries in RL change across adolescent development, conferring age differences in risk preferences. We additionally hypothesized that individuals’ learning asymmetries might asymmetrically bias their memory for images that coincide with positive versus negative prediction errors. Several past studies have characterized developmental changes in learning from valenced outcomes (<xref ref-type="bibr" rid="bib18">Christakou et al., 2013</xref>; <xref ref-type="bibr" rid="bib42">Hauser et al., 2015</xref>; <xref ref-type="bibr" rid="bib45">Jones et al., 2014</xref>; <xref ref-type="bibr" rid="bib59">Master et al., 2020</xref>; <xref ref-type="bibr" rid="bib63">Moutoussis et al., 2018</xref>; <xref ref-type="bibr" rid="bib96">van den Bos et al., 2012</xref>). However, the probabilistic reinforcement structures used in each of these studies demanded that the learner adopt specific valence asymmetries during value estimation in order to maximize reward in the task (<xref ref-type="bibr" rid="bib68">Nussenbaum and Hartley, 2019</xref>). For instance, in one study, child, adolescent, and adult participants were rewarded on 80% of choices for one option and 20% of choices for a second option (<xref ref-type="bibr" rid="bib96">van den Bos et al., 2012</xref>). In this task, a positive learning asymmetry yields better performance than a neutral or negative asymmetry (<xref ref-type="bibr" rid="bib68">Nussenbaum and Hartley, 2019</xref>). Indeed, adults exhibited a more optimal pattern of learning, with higher positive than negative learning rates, while children and adolescents did not (<xref ref-type="bibr" rid="bib96">van den Bos et al., 2012</xref>). Thus, choice behavior in these studies might reflect both potential age differences in the optimality of RL, as well as context-independent differences in the weighting of positive versus negative prediction errors (<xref ref-type="bibr" rid="bib16">Cazé and van der Meer, 2013</xref>; <xref ref-type="bibr" rid="bib68">Nussenbaum and Hartley, 2019</xref>).</p><p>In Experiment 1 of the present study, we assessed whether valence asymmetries in RL varied from childhood to adulthood, using a risk-sensitive RL task (<xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref>) in which probabilistic and deterministic choice options have equal EV, making no particular learning asymmetry optimal. This parameterization allows any biases in the weighting of positive versus negative prediction errors to be revealed through subjects’ systematic risk-averse or risk-seeking choice behavior. Each choice outcome in the task was associated with a trial-unique image, enabling assessment of whether valenced learning asymmetries also biased subsequent memory for images that coincided with good or bad outcomes.</p><p>To determine whether this hypothesized correspondence between valence biases in learning and memory generalized across experimental tasks and samples of different ages, in Experiment 2, we conducted a reanalysis of data from a previous study (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>). In this study, a group of adults completed a task in which they reported value estimates for a series of images, and later completed a memory test for the images they encountered during learning. The original manuscript reported that subsequent memory varied as a function of PE magnitude, but not valence. Here, we tested whether a valence-dependent effect of PE on memory might be evident after accounting for idiosyncratic valence biases in learning.</p></sec><sec id="s2" sec-type="results"><title>Results</title><sec id="s2-1"><title>Experiment 1</title><p>Participants (N = 62) ages 8–27 (<italic>M</italic> = 17.63, SD = 5.76) completed a risk-sensitive RL task (<xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref>). In this task, participants learned, through trial and error, the values and probabilities associated with probabilistic and deterministic ‘point machines’ (<xref ref-type="fig" rid="fig1">Figure 1A and B</xref>). On each trial (183 trials), participants made a free (two-choice options) or forced (single-choice option) selection of a point machine. Within free-choice trials, ‘risky’ trials presented a pair consisting of one probabilistic and one deterministic option, where neither option strictly dominated the other and evidence of individuals’ subjective values was revealed by their choices. On ‘test’ trials, in which one option dominated the other, we could assess objectively the accuracy of participants’ learning. We presented feedback (number of points) from each choice on a ‘ticket’ that also displayed a trial-unique picture of an object. A subsequent memory test allowed us to explore the interaction between choice outcomes and memory encoding across age (<xref ref-type="fig" rid="fig1">Figure 1C</xref>).</p><fig id="fig1" position="float"><label>Figure 1.</label><caption><title>Task structure.</title><p>(<bold>A</bold>) Schematic of the structure of a trial in the risk-sensitive reinforcement learning task. (<bold>B</bold>) The probabilities and point values associated with each of five ‘point machines’ (colors were counterbalanced). (<bold>C</bold>) Example memory trial.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/fig1.jpg"/></fig><sec id="s2-1-1"><title>Test trial performance</title><p>To ensure that participants learned the probabilities and outcomes associated with each machine, we first examined performance on test trials, in which one option dominated the other. Test trial accuracy significantly improved across the task (generalized linear mixed-effects model: <italic>z</italic> = 8.56, p<0.001, OR = 2.03, 95% CI [1.72, 2.38]), with accuracy improving from a mean of 0.63 in the first block to means of 0.80 and 0.84 in blocks 2 and 3, respectively. There was no main effect of age (<italic>z</italic> = 0.51, p=0.612, OR = 1.06, 95% CI [0.86, 1.30]) or interaction between age and trial number (<italic>z</italic> = 0.22, p=0.830, OR <italic>=</italic> 1.02, 95% CI [0.87, 1.19]; <xref ref-type="fig" rid="app1fig1">Appendix 1—figure 1A</xref>). These results suggest that accuracy on this coarse measure of value learning did not change with age in our task.</p></sec><sec id="s2-1-2"><title>Explicit reports</title><p>Following the learning task, we probed participants’ explicit knowledge about the point machines. Consistent with participants’ high accuracy on test trials, accuracy was also high on participants’ reports of whether each point machine was probabilistic or deterministic (<italic>M</italic> = 0.85) and for the point values associated with each machine (<italic>M</italic> = 0.84). Linear regressions suggested that performance on these explicit accuracy metrics did not vary with linear age (probabilistic/deterministic response accuracy by age: <italic>b</italic> = –0.02, 95% CI [–0.06, 0.03], <italic>t</italic>(60) = –0.88, p=0.382, <italic>f</italic><sup>2</sup> = 0.01, 95% CI [0, 0.13]; point value response accuracy by age: <italic>b</italic> = 0.02, 95% CI [–0.04, 0.07], <italic>t</italic>(60) = 0.65, p=0.516, <italic>f</italic><sup>2</sup> = 0.01, 95% CI [0, 0.11]).</p></sec><sec id="s2-1-3"><title>Response time</title><p>We explored whether response time (RT) varied with age during the learning task. We found a significant interaction between age and trial number (linear mixed-effects model: <italic>t</italic>(11279) = –2.10, p=0.036, <italic>b</italic> = –0.02, 95% CI [–0.04, 0]) predicting log-transformed RT. Although RT did not differ by age early in the experiment, older participants responded faster than younger participants by the end of the experiment.</p></sec><sec id="s2-1-4"><title>Decision-making</title><p>Importantly, in our task, there were two pairs of machines in which both probabilistic and deterministic options yielded the same EV (i.e., 100% 20 points and 50/50% 0/40 points; 100% 40 points and 50/50% 0/80 points). A primary goal of this study was to examine participants’ tendency to choose probabilistic versus deterministic machines when EV was equivalent. On these equal-EV risk trials, participants chose the probabilistic option on 37% of trials (SD = 21%). This value was significantly lower than 50% (one-sample <italic>t</italic>-test: <italic>t</italic>(61) = 4.87, p<0.001, <italic>d</italic> = 0.62, 95% CI [0.37, 0.95]), suggesting that, despite exhibiting heterogeneity in risk preferences, participants as a group were generally risk averse.</p><p>Next, we tested whether choices of the probabilistic machines, compared to choices for equal-EV deterministic machines, changed with age. The best-fitting model included both linear and quadratic age terms (<italic>F</italic>(1,59) = 4.58, p=0.036), indicating that risk taking changed nonlinearly with age. Contrary to our hypothesis that risk-seeking choices would be highest in adolescents, we observed a significant quadratic effect of age, such that adolescents chose the probabilistic options less often than children or adults (quadratic age effect in a linear regression including both linear and quadratic age terms: <italic>b</italic> = 0.06, 95% CI [0, 0.12], <italic>t</italic>(59) = 2.14, p=0.036, <italic>f</italic><sup>2</sup> = 0.08, 95% CI [0, 0.29]; <xref ref-type="fig" rid="fig2">Figure 2</xref>; see <xref ref-type="fig" rid="app1fig1">Appendix 1—figure 1</xref> for plots depicting risk taking across the task as a function of age). The linear effect of age was not significant (<italic>b</italic> = –0.01, 95% CI [–0.06, 0.12], <italic>t</italic>(59) = –0.44, p=0.662, <italic>f</italic><sup>2</sup> = 0.01, 95% CI [0, 0.11]). We also conducted a regression using the two-lines approach (<xref ref-type="bibr" rid="bib90">Simonsohn, 2018</xref>) and found a significant u-shaped pattern of risk taking with age, where the proportion of probabilistic choices decreased from age 8–16.45 (<italic>b</italic> = –0.03, <italic>z</italic> = –1.97, p=0.048) and increased from age 16.45–27 (<italic>b</italic> = 0.02, <italic>z</italic> = 1.98, p=0.048). Age patterns were qualitatively similar when considering the subset of trials in which participants faced choice options with unequal EV (i.e., the 0/80 point machine vs. the safe 20 point machine; see Appendix 1 and <xref ref-type="fig" rid="app1fig2">Appendix 1—figure 2</xref> for full results).</p><fig id="fig2" position="float"><label>Figure 2.</label><caption><title>Probabilistic choices by age.</title><p>Probabilistic (i.e., risky) choices by age on trials in which the risky and safe machines had equal expected value (EV). Data points depict the mean percentage of trials where each participant selected the probabilistic choice option as a function of age. The regression line is from a linear regression including linear and quadratic age terms (significant quadratic effect of age: <italic>b</italic> = 0.06, 95% CI [0, 0.12], <italic>t</italic>(59) = 2.14, p=0.036, <italic>f</italic><sup>2</sup> = .08, 95% CI [0, 0.29], N = 62). Shaded region represents 95% CIs for estimates.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/fig2.jpg"/></fig></sec><sec id="s2-1-5"><title>Reinforcement learning modeling</title><p>To better understand the learning processes underlying individuals’ decision-making, we compared the fit of four RL models to participants’ choice behavior. The first was a temporal difference (TD) model with one learning rate (<italic>α</italic>). The second was a risk-sensitive temporal difference (RSTD) model with separate learning rates for better-than-expected (<italic>α</italic><sup>+</sup>) and worse-than-expected (<italic>α</italic><sup>-</sup>) outcomes, allowing us to index valence biases in learning. The third model included four learning rates (FourLR), with separate <italic>α</italic><sup>+</sup> and <italic>α<sup>-</sup></italic> for free and forced choices, as past studies have found learning may differ as a function of agency (<xref ref-type="bibr" rid="bib17">Chambon et al., 2020</xref>; <xref ref-type="bibr" rid="bib21">Cockburn et al., 2014</xref>). Finally, the fourth model was a Utility model, which transforms outcome values into utilities with an exponential subjective utility function with a free parameter (<italic>ρ</italic>) capturing individual risk preferences (<xref ref-type="bibr" rid="bib72">Pratt, 1964</xref>), updated value estimates using a single learning rate. For all models, machine values were transformed to range from 0 to 1, and values were initialized at 0.5 (equivalent to 40 points). A softmax function with an additional parameter <italic>β</italic> was used to convert the relative estimated values of the two machines into a probability of choosing each machine presented for maximum likelihood estimation.</p><p>The RSTD (median Bayesian information criterion (BIC) = 131.93) and Utility (median BIC = 131.06) models both provided a better fit to participants’ choice data than both the TD (median BIC = 145.35) and FourLR (median BIC = 141.25) models (<xref ref-type="fig" rid="app1fig5">Appendix 1—figure 5</xref>). Assessment of whether the RSTD or Utility model provided the best fit to participants’ data was equivocal. At the group level, median ΔBIC was 0.87, while at the subject level, the median ΔBIC was 0.33. Thus, neither ΔBIC metric provides clear evidence in favor of either model (ΔBIC > 6 ; <xref ref-type="bibr" rid="bib77">Raftery, 1995</xref>).</p><p>To further arbitrate between the RSTD and Utility models, we ran posterior predictive checks and confirmed that simulations from both models generated using subjects’ fit parameter values yielded choice behavior that exhibited strong correspondence to the real participant data (see <xref ref-type="fig" rid="app1fig9">Appendix 1—figure 9</xref>). However, data simulated from the RSTD model exhibited a significantly stronger correlation with actual choices (<italic>r</italic> = 0.92) than those simulated using the Utility model (<italic>r</italic> = 0.89; <italic>t</italic>(61) = 2.58, p=0.012). Because the RSTD model fit choice data approximately as well as the Utility model, provided a significantly better qualitative fit to the choice data, and yielded an index of valence biases in learning, we focused our remaining analyses on the RSTD model (see Appendix 1 for additional model comparison analyses and for an examination of the relation between the Utility model and subsequent memory data).</p><p>We computed an asymmetry index (AI) for each participant, which reflects the relative size of <italic>α</italic><sup>+</sup> and <italic>α</italic><sup>-</sup>, from the RSTD model. Mean AI was –0.22 (SD = 0.50). Mirroring the age patterns observed in risk taking, a linear regression model with a quadratic age term fit better than the model with only linear age (<italic>F</italic>(1,59) = 5.88, p=0.018), and there was a significant quadratic age pattern in AI (<italic>b</italic> = 0.17, 95% CI [0.03, 0.31], <italic>t</italic>(59) = 2.43, p=0.018, <italic>f</italic><sup>2</sup> = 0.10, 95% CI [0, 0.33]; <xref ref-type="fig" rid="fig3">Figure 3</xref>). Further, the u-shaped relationship between AI and age was significant, with a decrease in AI from ages 8–17 (<italic>b</italic> = –0.08, <italic>z</italic> = –3.82, p<0.001), and an increase from ages 17–27 (<italic>b</italic> = 0.05, <italic>z</italic> = 2.17, p=0.030). This pattern was driven primarily by age-related changes in <italic>α</italic><sup>-</sup>, which was greater in adolescents relative to children and adults (better fit for linear regression including quadratic term: <italic>F</italic>(1,59) = 9.04, p=0.004; quadratic age: <italic>b</italic> = –0.09, 95% CI [–0.16, –0.03], <italic>t</italic>(59) = –3.01, p=0.004, <italic>f</italic><sup>2</sup> = 0.15, 95% CI [0.02, 0.43]; <xref ref-type="fig" rid="app1fig10">Appendix 1—figure 10B</xref>). According to the two-lines approach, <italic>α</italic><sup>-</sup> significantly increased from ages 8–18 (<italic>b</italic> = 0.04, <italic>z</italic> = 3.24, p=0.001) and decreased from ages 18–27 (<italic>b</italic> = –0.04, <italic>z</italic> = –3.57, p<0.001). Conversely, there were no linear or quadratic effects of age for <italic>α</italic><sup>+</sup> (all <italic>p</italic>s>0.24; <xref ref-type="fig" rid="app1fig10">Appendix 1—figure 10A</xref>). Finally, there were no significant linear or quadratic age patterns in the <italic>β</italic> parameter (<italic>p</italic>s>.15, see Appendix 1 for full results; <xref ref-type="fig" rid="app1fig10">Appendix 1—figure 10C</xref>).</p><fig id="fig3" position="float"><label>Figure 3.</label><caption><title>Asymmetry index (AI) by age.</title><p>The regression line is from a linear regression model including linear and quadratic age terms (<italic>b</italic> = 0.17, 95% CI [0.03, 0.31], <italic>t</italic>(59) = 2.43, p=0.018, <italic>f</italic><sup>2</sup> = 0.10, 95% CI [0, 0.33], N = 62). Data points represent individual participants. Shaded region represents 95% CIs for estimates.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/fig3.jpg"/></fig><p>Prior work has found that valence biases tend to be positive in free choices, but neutral or negative in forced choices (<xref ref-type="bibr" rid="bib17">Chambon et al., 2020</xref>; <xref ref-type="bibr" rid="bib21">Cockburn et al., 2014</xref>). While model comparison indicated that the FourLR model did not provide the best account of participants' learning process, we nonetheless conducted an exploratory analysis in which we used parameter estimates from the FourLR model to test whether learning asymmetries varied as a function of agency in our study. While the <italic>α</italic>+ and AI were both higher for free compared to forced trials, median AIs were negative for both free and forced choices (see Appendix 1 for full results; <xref ref-type="fig" rid="app1fig12">Appendix 1—figure 12</xref>).</p></sec><sec id="s2-1-6"><title>Memory performance</title><p>Next, we examined accuracy during the surprise memory test for images that were presented with choice outcomes. Participants correctly identified 54% (SD = 14%) of images presented alongside choice feedback (i.e., Hits) and incorrectly indicated that 24% (SD = 15%) of foil images had been presented during the choice task (False Alarms). Mean <italic>d</italic>′ was 0.93 (SD = 0.48). Hit rate did not significantly change with linear or quadratic age (<italic>p</italic>s>0.14). However, false alarm rate significantly increased with linear age (linear regression: <italic>b</italic> = 0.04, 95% CI [0.00; 0.08], <italic>t</italic>(60) = 2.14, p=0.037, <italic>f</italic><sup>2</sup> = 0.08, 95% CI [0, 0.28]; <xref ref-type="fig" rid="app1fig3">Appendix 1—figure 3A</xref>). There was a marginal linear decrease in <italic>d′</italic> with age (linear regression: <italic>b</italic> = –0.11, 95% CI [–0.23, 0.01], <italic>t</italic>(60) = 1.84, p=0.070, <italic>f</italic><sup>2</sup> = 0.06, 95% CI [0, 0.24]; <xref ref-type="fig" rid="app1fig3">Appendix 1—figure 3B</xref>), suggesting that adults performed slightly worse on the memory test than younger participants.</p></sec><sec id="s2-1-7"><title>Influence of choice context on memory</title><p>We next tested whether the decision context in which images were presented influenced memory encoding. To explore this possibility, we first tested whether participants preferentially remembered images presented with outcomes of probabilistic versus deterministic machines. Participants were significantly more likely to remember pictures presented following a choice that yielded probabilistic rather than deterministic outcomes (probabilistic: <italic>M</italic> = 0.56, SD = 0.15; deterministic: <italic>M =</italic> 0.52, <italic>SD =</italic> 0.15; <italic>t</italic>(61) = 3.08, p=0.003, <italic>d</italic> = 0.39, 95% CI [0.13, 0.65]). This result suggests that pictures were better remembered when they followed the choice of a machine that consistently generated prediction errors (PEs), which may reflect preferential allocation of attention toward outcomes of uncertain choices (<xref ref-type="bibr" rid="bib28">Dayan et al., 2000</xref>; <xref ref-type="bibr" rid="bib70">Pearce and Hall, 1980</xref>).</p><p>Next, we explored whether valence biases in learning could account for individual variability in subsequent memory. In theory, larger-magnitude PEs provide stronger learning signals. Thus, we hypothesized that participants would have better memory for items coinciding with larger PEs. We also expected that this effect might differ as a function of idiosyncratic valence biases, with participants preferentially remembering items coinciding with signed PEs where the sign was consistent with the valence bias of their AI. Of note, this model did not explicitly include a variable indicating whether outcomes followed probabilistic or deterministic choices. Rather, whether the choice was probabilistic or deterministic was reflected in the PE magnitude variable, which was typically higher for probabilistic choices. In a generalized linear mixed-effects model, we predicted memory accuracy as a function of AI, PE valence, PE magnitude, and their interaction. We also tested for effects of linear and quadratic age, false alarm rate, as a measure of participants’ tendency to generally deem items as old, and trial number in the memory task, to account for fatigue as the task progressed (<xref ref-type="fig" rid="fig4">Figure 4A</xref>). We had no a priori hypothesis about how any effect of valence bias on memory might interact with participants’ confidence in their ‘old’ and ‘new’ judgments. Therefore, consistent with prior research examining memory accuracy (e.g., <xref ref-type="bibr" rid="bib33">Dunsmoor et al., 2015</xref>; <xref ref-type="bibr" rid="bib65">Murty et al., 2016</xref>), we collapsed across ‘definitely’ and ‘maybe’ confidence ratings for our primary analysis (but see Appendix 1 for an exploratory ordinal regression analysis).</p><fig id="fig4" position="float"><label>Figure 4.</label><caption><title>The relation between valence biases in learning and incidental memory for pictures presented with choice outcomes (Experiment 1).</title><p>(<bold>A</bold>) Results from generalized mixed-effects regression depicting fixed effects on memory accuracy. Whiskers represent 95% CI. (<bold>B</bold>) Estimated marginal means plot showing the three-way interaction between AI, PE valence, and PE magnitude (<italic>z</italic> = 3.45, p=0.001, OR = 1.12, 95% CI [1.05, 1.19], N = 62). Individuals with higher AIs were more likely to remember images associated with larger positive PEs, and those with lower AIs were more likely to remember images associated with larger negative PEs. Shaded areas represent 95% CI for estimates. ***p < .001.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/fig4.jpg"/></fig><p>As expected, accuracy was significantly higher for those with a higher false alarm rate (suggesting a bias towards old responses; <italic>z =</italic> 4.86, p<0.001, OR = 1.41, 95% CI [1.23, 1.61]), and accuracy decreased as the task progressed (main effect of memory trial number: <italic>z</italic> = –5.83, p<0.001, OR = 0.82, 95% CI [0.76, 0.87]). There was a significant effect of unsigned PE magnitude on memory (<italic>z</italic> = 4.75, p<0.001, OR = 1.19, 95% CI [1.11, 1.28]), such images that coincided with largerPEs were better remembered. There was also a significant three-way interaction between PE magnitude, PE valence, and AI on memory accuracy (<italic>z</italic> = 3.45, p=0.001, OR = 1.12, 95% CI [1.05, 1.19]), such that people with more positive AIs were more likely to remember images associated with larger positive PEs (<xref ref-type="fig" rid="fig4">Figure 4B</xref>). The converse was also true: those with lower AIs were more likely to remember images presented concurrently with outcomes that elicited higher-magnitude negative PEs. Ordinal regression results that considered all four levels of confidence in recollection judgments (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1</xref>, <xref ref-type="fig" rid="app1fig4">Appendix 1—figure 4</xref>) yielded consistent results and suggested that effects were primarily driven by high-confidence responses. Notably, neither linear (<italic>z</italic> = 0.32, p=0.750, OR = 1.02, 95% CI [0.89, 1.17]) nor quadratic age (<italic>z</italic> = –0.18, p=0.856, OR = 0.99, 95% CI [0.84, 1.15]) were significant predictors of memory, suggesting that AI parsimoniously accounted for individual differences in memory.</p><p>To test whether differences in memory for outcomes of deterministic versus probabilistic trials might have driven the observed AI × PE magnitude × PE valence interaction effect, we reran the regression model only within the subset of trials in which participants made probabilistic choices. Our results did not change — we observed both a main effect of PE magnitude (<italic>z</italic> = 2.22, p=0.026, OR = 1.11, 95% CI [1.01, 1.23], N = 62) and a significant PE valence × PE magnitude × AI interaction (<italic>z</italic> = 2.34, p=0.019, OR = 1.11, 95% CI [1.02, 1.21], N = 62).</p><p>Finally, we tested for effects of agency — whether an image coincided with the outcome of a free or forced choice — on memory performance. We did not find a significant main effect of agency on memory, and agency did not significantly modulate the AI × PE magnitude × PE valence interaction effect (see Appendix 1 for full results; <xref ref-type="fig" rid="app1fig13">Appendix 1—figure 13</xref>).</p></sec><sec id="s2-1-8"><title>Self-reported risk taking</title><p>One possible explanation for our unexpected u-shaped relationship between age and risk preferences in our choice task is that the adolescents in our sample might have been atypically risk averse. To investigate this possibility, we examined the relation between age and self-reported risk taking to the Domain-Specific Risk Taking (DOSPERT) scale (<xref ref-type="bibr" rid="bib9">Blais and Weber, 2006</xref>). A linear regression model including quadratic age was a better fit than the model including linear age alone (<italic>F</italic>(1,59) = 9.55, p=0.003). Specifically, consistent with prior reports of increased self-reported risk taking in adolescents, we found a significant inverted u-shaped quadratic age pattern (<xref ref-type="fig" rid="fig5">Figure 5</xref><italic>, b</italic> = –0.42, 95% CI [-0.69, –0.15], <italic>t</italic>(59) = –3.09, p=0.003, <italic>f</italic><sup>2</sup> = 0.16, 95% CI [0.02, 0.44]). There was not a significant linear age pattern in self-reported risk taking (<italic>b</italic> = 0.15, 95% CI [–0.09, 0.39], <italic>t</italic>(59) = 1.27, p=0.208, <italic>f</italic><sup>2</sup> = 0.04, 95% CI [0, 0.20]). A two-lines regression analysis indicated that risk taking increased until age 15.29 (<italic>b</italic> = 0.23, <italic>z</italic> = 2.20, p=0.028) and decreased thereafter (<italic>b</italic> = –0.09, <italic>z</italic> = –2.03, p=0.042). Despite the fact that both choices in our task and self-report risk taking exhibited nonlinear age-related changes, there was not a significant correlation between DOSPERT score and risk taking in the task (<italic>r</italic> = –0.12, 95% CI [–0.36, 0.13], <italic>t</italic>(60) = –0.95, p=0.347).</p><fig id="fig5" position="float"><label>Figure 5.</label><caption><title>Self-reported risk taking by age.</title><p>Self-reported risk taking on the Domain-Specific Risk Taking (DOSPERT) scale changed nonlinearly with age (linear regression: <italic>b</italic> = –0.42, 95% CI [–0.69,–0.15], <italic>t</italic>(59) = –3.09, p=0.003, <italic>f</italic><sup>2</sup> = 0.16, 95% CI [0.02, 0.44], N = 62). Shaded region represents 95% CIs for estimates.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/fig5.jpg"/></fig></sec></sec><sec id="s2-2"><title>Experiment 2</title><p>Next, we assessed the generalizability of the observed effect of valence biases in learning on memory by conducting a reanalysis of a previously published independent dataset from a study that used a different experimental task in an adult sample (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>). Notably, results from this study suggested that unsigned PEs (i.e., PEs of greater magnitude, whether negative or positive) facilitated subsequent memory, but no signed effect was observed. Here, we examined whether signed valence-specific effects might be evident when we account for individual differences in valence biases in learning.</p><p>Participants (N = 305) completed a Pavlovian learning task in which they encountered indoor and outdoor scenes. One type of scene had higher average value than the other. On each trial, an indoor or outdoor image was displayed, and participants provided an explicit prediction for the average value of that scene type. After the learning task, participants completed a memory test for the scenes.</p><p>To quantify valence biases in this task, we fit an ‘Explicit Prediction’ RL model that was similar to the RSTD model used in Experiment 1, but was fit to participants’ trial-by-trial predictions rather than to choices. Like RSTD, the Explicit Prediction model included <italic>α</italic><sup>+</sup> and <italic>α</italic><sup>-</sup>, allowing us to quantify each participant’s AI based on the relative size of their best-fit <italic>α</italic><sup>+</sup> and <italic>α</italic><sup>-</sup> parameters. Mean AI was –0.11 (<italic>SD =</italic> 0.34). Next, we ran a generalized linear mixed-effects model, as in Experiment 1, to examine whether PE valence and magnitude interacted with AI to predict subsequent memory, controlling for false alarm rate and memory trial number. Results are reported in <xref ref-type="fig" rid="fig6">Figure 6</xref>.</p><fig id="fig6" position="float"><label>Figure 6.</label><caption><title>The relation between valence biases in learning and incidental memory for pictures presented with trial outcomes (Experiment 2).</title><p>Reanalysis of data from <xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>. (<bold>A</bold>) Results from generalized mixed-effects regression depicting fixed effects on memory accuracy. Whiskers represent 95% CI. (<bold>B</bold>) Estimated marginal means plot showing the three-way interaction between AI, PE valence, and PE magnitude (<italic>z</italic> = 2.19, p=0.029, OR = 1.07, 95% CI [1.01, 1.13], N = 305). Individuals with higher AIs were more likely to remember images associated with larger positive PEs, and those with lower AIs were more likely to remember images associated with larger negative PEs. Shaded areas represent 95% CI for estimates. *p < .05, ***p < .001.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/fig6.jpg"/></fig><p>Consistent with the results reported in the original manuscript (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>), as well as the findings in Experiment 1, there was a strong main effect of unsigned PE (i.e., PE magnitude) on memory (<italic>z</italic> = 5.09, p<0.001, OR = 1.19, 95% CI [1.12, 1.28]). However, aligned with our results from Experiment 1, we also observed a significant three-way interaction between AI, PE magnitude, and PE valence (<italic>z</italic> = 2.19, p=0.029, OR = 1.07, 95% CI [1.01, 1.13]). Qualitative examination of this interaction effect suggests that the pattern of results differed slightly from that in Experiment 1. In Experiment 2, differences in memory performance as a function of AI were primarily apparent for images coinciding with negative PEs (<xref ref-type="fig" rid="fig6">Figure 6B</xref>): those who learned more from negative PEs also had better episodic memory for images that coincided with increasingly large negative PEs, while all participants appeared to have stronger memory for images coinciding with larger positive PEs. Notably, the interaction pattern here mirrors that within the subset of forced trials from Experiment 1 (<xref ref-type="fig" rid="app1fig13">Appendix 1—figure 13B</xref>) where, as in Experiment 2, participants learned from observed outcomes, but did not make free choices. One possibility is that PE magnitude and PE valence enhance memory through separate mechanisms, with a universal positive effect of unsigned PEs but a contextually (depending on choice agency) and individually variable effect of PE valence.</p></sec></sec><sec id="s3" sec-type="discussion"><title>Discussion</title><p>In this study, we examined whether asymmetry in learning from good versus bad choice outcomes changed across adolescence, and whether valence biases in RL also influenced episodic memory encoding. Specifically, we hypothesized that adolescents would place greater weight on good than bad outcomes during learning, a potential cognitive bias that may contribute to the increased risk taking during adolescence evident in real-world epidemiological data (<xref ref-type="bibr" rid="bib48">Kann et al., 2018</xref>). We indeed observed nonlinear age differences in valence-based learning asymmetries, but in the direction opposite from our prediction. Adolescents learned more from outcomes that were worse than expected, which was reflected in less risk taking relative to children and adults. Within this developmental sample, individual differences in learning biases were mirrored in subsequent memory. People who learned more from surprising negative versus positive outcomes also had better memory for images that coincided with negative outcomes, and vice versa. Although the precise pattern of results differed slightly, this relation between idiosyncratic valence biases in RL and corresponding biases in subsequent memory was also evident in a second independent sample in a different task (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>), suggesting that this finding is generalizable. Collectively, these results highlight age-related changes across adolescence in the computation of subjective value and demonstrate that an individually varying valence asymmetric valuation process also influences how information is prioritized in memory.</p><p>Age-related shifts in learning rate asymmetry were driven primarily by changes in negative, rather than positive, learning rates. Whereas negative learning rates changed nonlinearly with age, there was no evidence for significant age differences in positive learning rates. This absence of age-related change in reward learning may seem counterintuitive given a large literature characterizing heightened reward sensitivity in adolescence (for reviews, see <xref ref-type="bibr" rid="bib39">Galván, 2013</xref>; <xref ref-type="bibr" rid="bib89">Silverman et al., 2015</xref>; <xref ref-type="bibr" rid="bib98">van Duijvenvoorde et al., 2016</xref>); however, these effects have largely been observed in tasks in which learning was not required. Moreover, heightened reactivity to negatively valenced stimuli has also been observed in adolescents, relative to children (<xref ref-type="bibr" rid="bib59">Master et al., 2020</xref>) and adults (<xref ref-type="bibr" rid="bib40">Galván and McGlennen, 2013</xref>), and adolescents have been found to exhibit greater sensitivity to negative social evaluative feedback than adults (<xref ref-type="bibr" rid="bib78">Rodman et al., 2017</xref>). While a relatively small number of studies have used RL models to characterize age-related changes in valence-specific value updating (<xref ref-type="bibr" rid="bib18">Christakou et al., 2013</xref>; <xref ref-type="bibr" rid="bib42">Hauser et al., 2015</xref>; <xref ref-type="bibr" rid="bib45">Jones et al., 2014</xref>; <xref ref-type="bibr" rid="bib59">Master et al., 2020</xref>; <xref ref-type="bibr" rid="bib63">Moutoussis et al., 2018</xref>; <xref ref-type="bibr" rid="bib96">van den Bos et al., 2012</xref>), age patterns reported in these studies vary substantially and none observed the same pattern of valence asymmetries present in our data. Variability in these findings may be due in part to substantial variation in the task reward structures, each of which required specific asymmetric settings of learning rates in order to perform optimally (<xref ref-type="bibr" rid="bib16">Cazé and van der Meer, 2013</xref>; <xref ref-type="bibr" rid="bib68">Nussenbaum and Hartley, 2019</xref>). This task variation limits the ability to differentiate age differences in optimal learning from systematic age differences in the influence of positive versus negative prediction errors on subjective value computation (<xref ref-type="bibr" rid="bib68">Nussenbaum and Hartley, 2019</xref>). In contrast, our study used a paradigm in which risky and safe options had equal EV, allowing us to index risk preferences and corresponding valence biases in a context where there was no optimal strategy. Given the lack of convergence in the literature to date, further studies characterizing valence asymmetries in learning using unconfounded measures will be needed to ascertain how broadly the biases we observed generalize to learning contexts with varying reward statistics (e.g., different outcome probabilities or outcomes that are truly negative instead of neutral).</p><p>Across two experimental samples, participants’ idiosyncratic tendencies to place greater weight on outcomes that elicited either positive or negative prediction errors was, in turn, associated with a propensity to form stronger incidental memories for images paired with these outcomes during learning. This correspondence between valence biases in evaluation and in memory is consistent with past findings. Greater risk-seeking choice behavior has been associated with better memory for the magnitude of extreme win outcomes (<xref ref-type="bibr" rid="bib55">Ludvig et al., 2018</xref>) as well as greater recalled frequency of win outcomes (<xref ref-type="bibr" rid="bib57">Madan et al., 2014</xref>, <xref ref-type="bibr" rid="bib58">Madan et al., 2017</xref>), whereas risk-averse choices have been associated with the opposite pattern. Our results extend these findings by linking individual risk preferences to an underlying learning algorithm that predicts the valence specificity of corresponding memory biases and by demonstrating that these biases extend to episodic features incidentally associated with valenced outcomes. Moreover, while several prior studies employing computational analyses of learning have variably observed enhanced memory for images coinciding with outcomes that elicit positive (<xref ref-type="bibr" rid="bib26">Davidow et al., 2016</xref>; <xref ref-type="bibr" rid="bib44">Jang et al., 2019</xref>), negative (<xref ref-type="bibr" rid="bib47">Kalbe and Schwabe, 2020</xref>), or high-magnitude (independent of valence) PEs (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>; <xref ref-type="bibr" rid="bib83">Rouhani and Niv, 2019</xref>), our findings suggest that consideration of individual differences in the prioritization of positive versus negative PEs may be critical in understanding how these aspects of value-based learning signals relate to memory encoding.</p><p>Attention likely played a critical role in the observed learning and memory effects. Although our study did not include direct measures of attention, there is a large literature demonstrating the critical role of attention in both RL (<xref ref-type="bibr" rid="bib28">Dayan et al., 2000</xref>; <xref ref-type="bibr" rid="bib43">Holland and Schiffino, 2016</xref>; <xref ref-type="bibr" rid="bib70">Pearce and Hall, 1980</xref>; <xref ref-type="bibr" rid="bib75">Radulescu et al., 2019</xref>) and memory formation (<xref ref-type="bibr" rid="bib20">Chun and Turk-Browne, 2007</xref>). Prominent theoretical accounts have proposed that attention should be preferentially allocated to stimuli that are more uncertain (<xref ref-type="bibr" rid="bib28">Dayan et al., 2000</xref>; <xref ref-type="bibr" rid="bib70">Pearce and Hall, 1980</xref>). In our study, memory was better for items that coincided with probabilistic compared to deterministic outcomes. This finding likely reflects greater attention to the episodic features associated with outcomes of uncertain choice options. Importantly, however, our memory findings could not be solely explained via an uncertainty-driven attention account as the relation between idiosyncratic asymmetric valence biases and memory was also evident within the subset of trials with probabilistic outcomes. Thus, our observed memory effects may reflect differential attention to valenced outcomes that varies systematically across individuals in a manner that can be accounted for by asymmetries in their learning rates. Such valence biases in attention have been widely observed in clinical disorders (<xref ref-type="bibr" rid="bib2">Bar-Haim et al., 2007</xref>; <xref ref-type="bibr" rid="bib62">Mogg and Bradley, 2016</xref>) and may also be individually variable within non-clinical populations.</p><p>In Experiment 1 of the present study, participants observed the outcomes of both free and forced choices. Prior studies have demonstrated differential effects of free versus forced choices on both learning and memory (<xref ref-type="bibr" rid="bib17">Chambon et al., 2020</xref>; <xref ref-type="bibr" rid="bib21">Cockburn et al., 2014</xref>; <xref ref-type="bibr" rid="bib49">Katzman and Hartley, 2020</xref>), which may reflect greater allocation of attention to contexts in which individuals have agency. Valence asymmetries in learning have been found to vary as a function of whether choices are free or forced, such that participants tend to exhibit a greater positive learning rate bias for free than for forced choices (<xref ref-type="bibr" rid="bib17">Chambon et al., 2020</xref>; <xref ref-type="bibr" rid="bib21">Cockburn et al., 2014</xref>). Here, we did not observe positive learning rate asymmetries for free choices, and a model that included separate valenced learning rates for free versus forced choices was not favored by model comparison. Studies have also found that subsequent memory is facilitated for images associated with free, relative to forced, choices (<xref ref-type="bibr" rid="bib49">Katzman and Hartley, 2020</xref>; <xref ref-type="bibr" rid="bib64">Murty et al., 2015</xref>). In Experiment 1, there was no significant effect of agency on memory. However, in Experiment 2, in which participants provided explicit predictions of choice outcomes, but did not make free choices, the qualitative pattern of learning and memory biases differed from that observed in Experiment 1, and closely resembled the pattern present within the subset of forced-choice trials from that experiment. Namely, in each of these conditions where participants were not able to make free choices, all participants, regardless of AI, exhibited better memory for images presented with large positive PEs. Thus, while our study was not explicitly designed to test for such effects, this preliminary evidence suggests that choice agency may modulate the relation between valence biases in learning and corresponding biases in long-term memory, a hypothesis that should be directly assessed in future studies.</p><p>While one interpretation of our results is that asymmetric value updating influences the prioritization of events in memory, recent theoretical proposals (<xref ref-type="bibr" rid="bib8">Biderman et al., 2020</xref>; <xref ref-type="bibr" rid="bib53">Lengyel and Dayan, 2008</xref>; <xref ref-type="bibr" rid="bib87">Shadlen and Shohamy, 2016</xref>) and empirical findings (<xref ref-type="bibr" rid="bib1">Bakkour et al., 2019</xref>; <xref ref-type="bibr" rid="bib11">Bornstein et al., 2017</xref>; <xref ref-type="bibr" rid="bib32">Duncan et al., 2019</xref>) suggest a potential alternative account. According to this work, sampling of specific valenced episodes from memory can influence decision-making and serve as a different way of making choices under uncertainty than the sort of incremental value computation formalized in RL models. Under this conceptualization, a tendency to preferentially encode or retrieve past positive or negative experiences may, in turn, drive risk-averse or risk-seeking choice biases. While our task design does not enable clear arbitration between these alternative directional hypotheses, our results provide additional evidence of a tight coupling between valuation and episodic memory, and further underscore the importance in examining individual differences in valence asymmetries in these processes.</p><p>Traditional behavioral economic models of choice suggest that risk preferences stem from a nonlinear transformation of objective value into subjective utility (<xref ref-type="bibr" rid="bib7">Bernoulli, 1954</xref>; <xref ref-type="bibr" rid="bib46">Kahneman and Tversky, 1979</xref>), with decreases in the marginal utility produced by each unit of objective value (i.e., a concave utility curve) producing risk aversion. Our present study was motivated by the insight that such risk-averse, or risk-seeking, preferences can also arise from an RL process that asymmetrically integrates valenced prediction errors (<xref ref-type="bibr" rid="bib61">Mihatsch and Neuneier, 2002</xref>; <xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref>). In Experiment 1, we fit both a traditional behavioral economic model with exponential subjective utilities as well as a model with valenced learning rates. Notably, there was a very close correspondence between learning asymmetries derived from the valenced learning rate model and the risk preference parameter from the utility model, and model comparison indicated that these models provided comparably good accounts of participants’ choice data. Thus, future research will be needed to arbitrate between utility and valenced learning rate models of decisions under risk. However, a potential parsimonious account is that a risk-sensitive learning algorithm could represent a biologically plausible process for the construction of risk preferences (<xref ref-type="bibr" rid="bib25">Dabney et al., 2020</xref>), in which distortions of value are produced through differential subjective weighting of good and bad choice outcomes (<xref ref-type="bibr" rid="bib61">Mihatsch and Neuneier, 2002</xref>; <xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref>).</p><p>Contrary to our a priori hypothesis, and to epidemiological (<xref ref-type="bibr" rid="bib48">Kann et al., 2018</xref>; <xref ref-type="bibr" rid="bib93">Steinberg, 2013</xref>) and theoretical (<xref ref-type="bibr" rid="bib15">Casey et al., 2008</xref>; <xref ref-type="bibr" rid="bib56">Luna, 2009</xref>; <xref ref-type="bibr" rid="bib92">Steinberg, 2008</xref>) work suggesting that adolescence is a period of increased risk taking, we found that adolescents took fewer risks than children or adults in our task. While at first glance these results might appear anomalous, within the same sample, we found that adolescents reported greater real-world risk taking than both children and adults. This lack of correspondence between task-based and self-reported indices of risk taking is consistent with previous findings in adults (<xref ref-type="bibr" rid="bib76">Radulescu et al., 2020</xref>), and suggests that these two measures reflect separable constructs. Past empirical studies assessing developmental changes in risky choice in laboratory tasks have observed varied results (<xref ref-type="bibr" rid="bib30">Defoe et al., 2015</xref>; <xref ref-type="bibr" rid="bib79">Rosenbaum et al., 2018</xref>; <xref ref-type="bibr" rid="bib80">Rosenbaum and Hartley, 2019</xref>), but highlight two potential features of tasks that may elicit heightened adolescent risk taking. Adolescents may be more likely to take risks in tasks that require learning about risk through experience versus explicit description (<xref ref-type="bibr" rid="bib79">Rosenbaum et al., 2018</xref>), and in which the probabilistic negative outcomes are rare (e.g., the Iowa Gambling Task; <xref ref-type="bibr" rid="bib6">Bechara et al., 1997</xref>), qualities that are also true of many real-world risk-taking contexts. While our task involved experiential learning, risky choices resulted in rewarding outcomes on half of the trials and non-win outcomes on the other half. Thus, undesirable outcomes were not rare and there were no true negative outcomes. Highlighting the important influence of such contextual features on decision-making across development, a recent study found that adolescents were more prone than adults to ‘underweight’ rare outcomes in decision-making relative to their true probabilities, conferring a greater propensity to take risks in situations where rare outcomes are unfavorable (<xref ref-type="bibr" rid="bib81">Rosenbaum et al., 2021</xref>). Collectively, these findings suggest that specific details of an experimental design may influence the age-related patterns of risk taking observed in laboratory tasks (<xref ref-type="bibr" rid="bib80">Rosenbaum and Hartley, 2019</xref>) and suggest that greater ecological validity of task designs might be best achieved by mirroring the key statistical properties of real-world decision contexts of interest.</p><p>The present findings raise the suggestion that, for a given individual, valence asymmetries in value-based learning might become more negative from childhood into adolescence, and more positive from adolescence into young adulthood. However, an important caveat is that such patterns of developmental change cannot be validly inferred from cross-sectional studies, which are confounded by potential effects of cohort (<xref ref-type="bibr" rid="bib86">Schaie, 1965</xref>). Past studies have demonstrated that valence asymmetries in RL are indeed malleable within a given individual, exhibiting sensitivity to the statistics of the learning environment (e.g., the informativeness of positive versus negative outcomes; <xref ref-type="bibr" rid="bib73">Pulcu and Browning, 2017</xref>) as well as to endogenous manipulations such as the pharmacological manipulation of neuromodulatory systems (<xref ref-type="bibr" rid="bib60">Michely et al., 2020</xref>). Future longitudinal studies will be needed to definitively establish whether valence biases in learning exhibit systematic age-related changes within an individual over developmental time.</p><p>Adolescence is conventionally viewed as a period of heightened reward-seeking, begging the question of why adolescents might exhibit the strongest negative valence bias in learning and memory. Theoretical consideration of the adaptive role of valence asymmetries may provide a parsimonious resolution to this apparent contradiction (<xref ref-type="bibr" rid="bib16">Cazé and van der Meer, 2013</xref>). Somewhat counterintuitively, greater updating for negative versus positive prediction errors (i.e., a negative valence bias) yields systematic distortions in subjective value that effectively increase the contrast between outcomes in the reward domain (e.g., a participant with a negative learning asymmetry will represent the risky 80- and 40-point machines as being more different from each other than a participant with a positive learning asymmetry), facilitating optimal reward-motivated action selection. This tuning of learning rates is particularly beneficial in environments in which potential rewards are abundant (<xref ref-type="bibr" rid="bib16">Cazé and van der Meer, 2013</xref>), which may be true during adolescence when social elements of the environment acquire unique reward value (<xref ref-type="bibr" rid="bib10">Blakemore, 2008</xref>; <xref ref-type="bibr" rid="bib66">Nardou et al., 2019</xref>). While negative valence biases may be adaptive for reward-guided decision-making, a propensity to form more persistent memories for negative outcomes may also contribute to adolescents’ heightened vulnerability to psychopathology (<xref ref-type="bibr" rid="bib51">Lee et al., 2014</xref>; <xref ref-type="bibr" rid="bib69">Paus et al., 2008</xref>). A recent study using computational formalizations found that adults who were biased toward remembering images associated with negative, relative to positive, prediction errors also exhibited greater depressive symptoms (<xref ref-type="bibr" rid="bib83">Rouhani and Niv, 2019</xref>). Moreover, negative biases in real-world autobiographical memory are a hallmark of depression and anxiety in both adolescents (<xref ref-type="bibr" rid="bib50">Kuyken and Howell, 2006</xref>) and adults (<xref ref-type="bibr" rid="bib31">Dillon and Pizzagalli, 2018</xref>; <xref ref-type="bibr" rid="bib37">Gaddy and Ingram, 2014</xref>). Future research should examine how valence biases in learning and memory, as well as the reward statistics of an individual’s real-world environment, relate to vulnerability or resilience to psychopathology across adolescent development. Finally, given an extensive literature demonstrating the pronounced influence of neuromodulatory systems on both valence biases in RL (<xref ref-type="bibr" rid="bib24">Cox et al., 2015</xref>; <xref ref-type="bibr" rid="bib35">Frank et al., 2004</xref>; <xref ref-type="bibr" rid="bib36">Frank et al., 2007</xref>; <xref ref-type="bibr" rid="bib60">Michely et al., 2020</xref>) and value-guided memory (<xref ref-type="bibr" rid="bib54">Lisman and Grace, 2005</xref>; <xref ref-type="bibr" rid="bib84">Sara, 2009</xref>), future studies might examine how developmental changes within these systems relate to the age-related shifts in valence biases observed here.</p></sec><sec id="s4" sec-type="materials|methods"><title>Materials and methods</title><sec id="s4-1"><title>Experiment 1</title><sec id="s4-1-1"><title>Participants</title><p>Sixty-two participants ages 8–27 years were included in our final sample (mean age = 17.63, SD = 5.76, 32 females). Nine additional participants completed the study but were removed from the sample due to poor task performance (described further below). This sample size is consistent with prior studies that used age as a continuous predictor and have found significant age differences in decision-making (e.g., <xref ref-type="bibr" rid="bib29">Decker et al., 2015</xref>; <xref ref-type="bibr" rid="bib71">Potter et al., 2017</xref>; <xref ref-type="bibr" rid="bib97">van den Bos et al., 2015</xref>). All participants had no previous diagnosis of a learning disorder, no current psychiatric medication use, and normal color vision according to self- or parent report.</p></sec><sec id="s4-1-2"><title>Risk-sensitive RL task</title><p>In the present study, participants completed a risk-sensitive RL task adapted from <xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref> in which participants learned, through trial and error, the values and probabilities associated with five ‘point machines’ (<xref ref-type="fig" rid="fig1">Figure 1A</xref>). Three machines were deterministic and gave their respective payoffs 100% of the time (<xref ref-type="fig" rid="fig1">Figure 1B</xref>). Two machines were probabilistic (or risky) and gave their respective payoffs 50% of the time and zero points the other 50% (<xref ref-type="fig" rid="fig1">Figure 1B</xref>). Importantly, EV could be deconfounded from risk as there were two pairs of machines in which both probabilistic and deterministic options yielded the same EV (i.e., 100% 20 points and 50/50% 0/40 points; 100% 40 points and 50/50% 0/80 points). We presented each choice outcome on a ‘ticket’ that also displayed a trial-unique picture of an object. A subsequent memory test allowed us to explore the interaction between choice outcomes and memory encoding across age. The task was programmed in MATLAB Version R2017a (The MathWorks, Inc, Natick, MA).</p><p>All participants completed a tutorial that involved detailed instructions and practice trials with machines that had the same probability structure as the machines they would encounter in the later task (i.e., one machine always gave 1 point, the other gave 0 point on half of trials and 2 points on the other half). Then, participants completed the RL task, which included 183 trials. There were 66 total ‘risky’ choices between probabilistic and deterministic machines. 42 of these risky trials involved choices between machines with equal EV, while 24 trials required choices between the probabilistic 0/80 machine and the deterministic 20 point machine. Participants also experienced 75 single-option ‘forced’ trials (15 for each of the five machines) to ensure each participant learned about values and probabilities associated with all of the machines. During forced trials, only one machine appeared on the screen, and the participant pressed a button to indicate the location of the machine (left or right). Finally, there were 42 test trials in which one machine’s value had absolute dominance over the other, with all outcomes of one option being greater than or equal to all outcomes of the other option (e.g., 100% chance of 40 points versus 50% of 40 points and 50% of 0 points). Test trials allowed us to gauge participants’ learning and understanding of the task. We excluded nine participants who did not choose correctly on at least 60% of test trials in the last 2/3 of the task (four children ages 8–9, three adolescents ages 14–16, and two adults ages 24–25). The trials were divided into blocks with 1/3 of the trials in each block, and after each block, participants could choose to take a short break. Unbeknownst to participants, trials were pseudo-randomized, such that 1/3 of each type of trial was presented in each block of the task, with the order of trial types randomized within each block. Outcomes of risky machines were additionally pseudo-randomized so that within each series of eight choices from a given risky machine, four choices resulted in a win and four resulted in zero point, in a random order.</p><p>On each trial, participants were asked to make a choice within 3 s after the machines were presented. If they chose in time, the outcome of the choice was presented on a ‘ticket’ along with a randomly selected, trial-unique picture of an object for 2 s (<xref ref-type="fig" rid="fig1">Figure 1A</xref>). If they did not respond in time, the words ‘TOO SLOW’ were presented, without a picture, for 1 s before the task moved to the next trial. Across all participants, only 37 (out of 11,346) total trials were missed for slow responses, with a maximum of 7 missed trials for one participant.</p><p>After completing the choice task, participants were probed for their explicit memory of points associated with each machine. For every machine, a participant was first asked, “Did this machine always give you the same number of points, or did it sometimes give 0 points and sometimes give you more points?” If the participant indicated that the machine always gave the same number of points, they were asked, “How many points did this machine give you each time you chose it?” Otherwise, they were asked, “How many points did this machine give you when it did not give 0 points?” To respond to this second question, participants selected from all possible point outcomes presented in the task (0, 20, 40, 80). There was no time limit for responding to these questions.</p><p>Next, participants completed a surprise memory test, in which all 183 images presented during the task and 183 novel images were presented in random order (<xref ref-type="fig" rid="fig1">Figure 1C</xref>). Images corresponding to the few choice trials that were missed due to slow responses were recategorized as novel. Ratings were on a scale from 1 (definitely saw during the task) to 4 (definitely did not see during the task), and participants had unlimited time to indicate their responses. All images were obtained from the Bank of Standardized Stimuli (BOSS; <xref ref-type="bibr" rid="bib13">Brodeur et al., 2010</xref>; <xref ref-type="bibr" rid="bib14">Brodeur et al., 2014</xref>) and were selected to be familiar and nameable for the age range in our sample. For each participant, half of the set of photos was randomly chosen to be presented during the task and half were assigned to be novel images for the memory test.</p></sec><sec id="s4-1-3"><title>Self-reported risk taking</title><p>To assess the predictive validity of our findings for real-world risk taking, participants completed the DOSPERT scale (<xref ref-type="bibr" rid="bib9">Blais and Weber, 2006</xref>). The DOSPERT indexes participants’ likelihood of taking risks in five domains: monetary, health and safety, recreational, ethical, and social. We computed the mean self-reported likelihood of risk taking across all behaviors on the DOSPERT as a measure of real-world risk taking. Age-appropriate variants of the DOSPERT were administered to children (8–12 years old), adolescents (13–17 years old), and adults (ages 18 and older) (<xref ref-type="bibr" rid="bib3">Barkley-Levenson et al., 2013</xref>; <xref ref-type="bibr" rid="bib91">Somerville et al., 2017</xref>; <xref ref-type="bibr" rid="bib98">van Duijvenvoorde et al., 2016</xref>).</p></sec><sec id="s4-1-4"><title>Reasoning assessment</title><p>We administered the Vocabulary and Matrix Reasoning sections of the Wechsler Abbreviated Scale of Intelligence (WASI; <xref ref-type="bibr" rid="bib101">Wechsler, 2011</xref>), which index verbal cognition and abstract reasoning, to ensure that these measures were not confounded with age within our sample. WASI scores did not vary by linear or quadratic age (<italic>p</italic>s>.2). Thus, we did not include this measure in subsequent analyses.</p></sec><sec id="s4-1-5"><title>Procedure</title><p>Participants first provided informed consent (adults) or assent and parental consent (children and adolescents). Next, participants completed the risk-sensitive RL task and memory test, followed by the DOSPERT questionnaire and the WASI. Participants were paid $15 for completing the experiment, which lasted approximately 1 hr. Although participants were told that an additional bonus payment would be based on the number of points they earned in the risk-sensitive RL task, all participants received the same $5 bonus payment. The study protocol was approved by the New York University Institutional Review Board.</p></sec><sec id="s4-1-6"><title>Analyses</title><sec id="s4-1-6-1"><title>Reinforcement-learning models</title><p>Four RL models were fit to participants’ choices in the task.</p><sec id="s4-1-6-1-1"><title>TD model</title><p>We fit a TD learning model (<xref ref-type="bibr" rid="bib94">Sutton and Barto, 1998</xref>), in which the estimated value of choosing a given machine (<italic>Q<sub>M</sub></italic>) is updated on each trial (<italic>t</italic>) according to the following function: <italic>Q<sub>M</sub></italic>(<italic>t</italic> + 1) = <italic>Q<sub>M</sub></italic>(<italic>t</italic>) + <italic>α</italic> * δ(<italic>t</italic>), in which δ(<italic>t</italic>) = <italic>r</italic>(<italic>t</italic>) – <italic>Q<sub>M</sub></italic>(<italic>t</italic>) is the prediction error, representing how much better or worse the reward outcome (<italic>r</italic>) is than the estimated value of that machine. δ is scaled by a learning rate <italic>α</italic>, a free parameter that is estimated separately for each participant.</p></sec><sec id="s4-1-6-1-2"><title>RSTD model</title><p>The RSTD model is similar to the TD model but includes two separate learning rates for prediction errors of different signs. Specifically, when δ is positive, the value of the chosen machine is updated according to the equation: Q<sub>M</sub>(t + 1) = Q<sub>M</sub>(t) + <italic>α</italic><sup>+</sup> * δ(<italic>t</italic>). When δ is negative, the chosen machine’s value is updated as Q<sub>M</sub>(t + 1) = Q<sub>M</sub>(t) + <italic>α</italic><sup>-</sup> * δ(<italic>t</italic>). Including two learning rates allows the model to be sensitive to the risk preferences revealed by participants’ choices across the probabilistic and deterministic (‘risky versus safe’) choice pairs in the paradigm (<xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref>). For a given individual, if <italic>α</italic><sup>+</sup> is greater than <italic>α</italic><sup>-</sup>, Q-values of the machines with variable outcomes will be greater than those of deterministic machines with equal EV, and the individual will be more likely to make risk-seeking choices. Conversely if <italic>α</italic><sup>-</sup> is greater than <italic>α</italic><sup>+</sup>, the Q-values of the risky machines will be lower than their EVs, making risk-averse choices more likely. To index the relative difference between <italic>α</italic><sup>+</sup> and <italic>α</italic><sup>-</sup>, we computed an AI as AI = (<italic>α</italic><sup>+</sup> - <italic>α</italic><sup>-</sup>)/(<italic>α</italic><sup>+</sup> + <italic>α<sup>-</sup></italic>), where an AI > 0 reflects greater weighting of positive relative to negative prediction errors, whereas an AI < 0 reflects greater relative weighting of negative prediction errors (<xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref>).</p></sec><sec id="s4-1-6-1-3"><title>FourLR model</title><p>In our task, participants made both free and forced choices. Past research suggests that valence biases in learning may differ as a function of choice agency (<xref ref-type="bibr" rid="bib17">Chambon et al., 2020</xref>; <xref ref-type="bibr" rid="bib21">Cockburn et al., 2014</xref>). To test this possibility, we assessed the fit of a FourLR model, which was the same as the RSTD model except that it included four learning rates instead of two, with separate <italic>α</italic><sup>+</sup> and <italic>α</italic><sup>-</sup> parameters for free and forced choices.</p></sec><sec id="s4-1-6-1-4"><title>Utility model</title><p>As a further point of comparison with the TD, RSTD, and FourLR models, we estimated a utility model that employed the same value update equation as the TD model, <italic>Q<sub>M</sub></italic>(<italic>t</italic> + 1) = <italic>Q<sub>M</sub></italic>(<italic>t</italic>) + <italic>α</italic> * δ(<italic>t</italic>). However, δ was defined according to the equation δ(<italic>t</italic>) = <italic>r</italic>(<italic>t</italic>)<sup>ρ</sup> – <italic>Q<sub>M</sub></italic>(<italic>t</italic>), in which the reward outcome is exponentially transformed by <italic>ρ</italic>, which represents the curvature of each individual’s subjective utility function (<xref ref-type="bibr" rid="bib72">Pratt, 1964</xref>). <italic>ρ</italic> < 1 corresponds to a concave utility function, which yields risk aversion as a result of diminishing sensitivity to returns (<xref ref-type="bibr" rid="bib95">Tversky and Kahneman, 1992</xref>). In contrast, <italic>ρ</italic> > 1 corresponds to a convex utility function that yields risk-seeking behavior.</p><p>In all models, Q-values were converted to probabilities of choosing each option in a trial using the softmax rule, P<sub>M1</sub> = e<sup>β*Q(t)M1</sup>/(e<sup>β*Q(t)M1</sup>+ e<sup>β*Q(t)M2</sup>), where P<sub>M1</sub> is the predicted probability of choosing Machine 1, with the inverse temperature parameter <italic>β</italic> capturing how sensitive an individual’s choices are to the difference in value between the two machines. Notably, outcomes of the forced trials were included in the value updating step for each model. However, forced trials were not included in the modeling stage in which learned values are passed through the softmax function to determine choice probabilities as there was only a single-choice option on these trials.</p></sec></sec></sec></sec><sec id="s4-2"><title>Model fitting</title><p>Prior to model fitting, outcome values were rescaled between 0 and 1, with 1 representing the maximum possible point outcome (80). We fit all RL models for each participant via maximum a posteriori estimation in MATLAB using the optimization function fminunc. <italic>Q-</italic>values were initialized at 0.5 (equivalent to 40 points). Bounds and priors for each of the parameters are listed in <xref ref-type="table" rid="table1">Table 1</xref>. There was no linear or quadratic relationship between BIC and age in any of the models (all <italic>p</italic>s>0.1).</p><table-wrap id="table1" position="float"><label>Table 1.</label><caption><title>Bounds, priors, and recoverability for parameters in each model.</title></caption><table frame="hsides" rules="groups"><thead><tr><th align="left" valign="bottom">Model</th><th align="left" valign="bottom">Parameter</th><th align="left" valign="bottom">Bounds</th><th align="left" valign="bottom">Prior</th><th align="left" valign="bottom">Recoverability</th></tr></thead><tbody><tr><td align="left" valign="bottom">TD</td><td align="left" valign="bottom"><italic>α</italic></td><td align="left" valign="bottom">0,1</td><td align="left" valign="bottom">Beta(2,2)</td><td align="char" char="." valign="bottom">0.84</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>β</italic></td><td align="left" valign="bottom">0.000001, 30</td><td align="left" valign="bottom">Gamma(2,3)</td><td align="char" char="." valign="bottom">0.88</td></tr><tr><td align="left" valign="bottom">RSTD</td><td align="left" valign="bottom"><italic>α<sup>+</sup></italic></td><td align="left" valign="bottom">0,1</td><td align="left" valign="bottom">Beta(2,2)</td><td align="char" char="." valign="bottom">0.79</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>α<sup>-</sup></italic></td><td align="left" valign="bottom">0,1</td><td align="left" valign="bottom">Beta(2,2)</td><td align="char" char="." valign="bottom">0.88</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>β</italic></td><td align="left" valign="bottom">0.000001, 30</td><td align="left" valign="bottom">Gamma(2,3)</td><td align="char" char="." valign="bottom">0.90</td></tr><tr><td align="left" valign="bottom">FourLR</td><td align="left" valign="bottom"><italic>α<sup>+</sup> free</italic></td><td align="left" valign="bottom">0,1</td><td align="left" valign="bottom">Beta(2,2)</td><td align="char" char="." valign="bottom">0.79</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>α<sup>-</sup> free</italic></td><td align="left" valign="bottom">0,1</td><td align="left" valign="bottom">Beta(2,2)</td><td align="char" char="." valign="bottom">0.89</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>α<sup>+</sup> forced</italic></td><td align="left" valign="bottom">0,1</td><td align="left" valign="bottom">Beta(2,2)</td><td align="char" char="." valign="bottom">0.76</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>α<sup>-</sup> forced</italic></td><td align="left" valign="bottom">0,1</td><td align="left" valign="bottom">Beta(2,2)</td><td align="char" char="." valign="bottom">0.78</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>β</italic></td><td align="left" valign="bottom">0.000001, 30</td><td align="left" valign="bottom">Gamma(2,3)</td><td align="char" char="." valign="bottom">0.90</td></tr><tr><td align="left" valign="bottom">Utility</td><td align="left" valign="bottom"><italic>α</italic></td><td align="left" valign="bottom">0,1</td><td align="left" valign="bottom">Beta(2,2)</td><td align="char" char="." valign="bottom">0.75</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>β</italic></td><td align="left" valign="bottom">0.000001, 30</td><td align="left" valign="bottom">Gamma(2,3)</td><td align="char" char="." valign="bottom">0.88</td></tr><tr><td align="left" valign="bottom"> </td><td align="left" valign="bottom"><italic>ρ</italic></td><td align="left" valign="bottom">0, 2.5</td><td align="left" valign="bottom">Gamma(1.5,1.5)</td><td align="char" char="." valign="bottom">0.88</td></tr></tbody></table><table-wrap-foot><fn><p>Priors for <italic>α</italic> and <italic>β</italic> were based on those used in <xref ref-type="bibr" rid="bib67">Niv et al., 2012</xref>.</p></fn><fn><p>TD, temporal difference; RSTD, risk-sensitive temporal difference; LR, learning rate.</p></fn></table-wrap-foot></table-wrap></sec><sec id="s4-3"><title>Parameter and model recovery</title><p>For each model, we simulated data for 10,000 subjects with values of each parameter drawn randomly and uniformly from the range of possible parameter values. Next, we fit the simulated data using the same model. We tested for recoverability of model parameters by correlating the parameter that generated the data with the parameters produced through model fitting. These correlations are displayed in <xref ref-type="table" rid="table1">Table 1</xref>. All parameters for TD, RSTD, FourLR, and Utility models showed high recoverability.</p><p>To examine the identifiability of the TD, RSTD, FourLR, and Utility models, we generated simulated data using each model and fit all four of the models, including those that were <italic>not</italic> used to generate the data to each simulated dataset (e.g., we fit all the TD-generated subjects with the TD model as well as the RSTD, FourLR, and Utility models). We then used BIC, a quality-of-fit metric that penalizes models for additional parameters, to assess whether the generating model was also the best-fitting model for each subject. Recoverability was reasonable for all models except the least-parsimonious FourLR model (<xref ref-type="table" rid="table2">Table 2</xref>). Aside from the subjects generated by the FourLR model, for all pairwise comparisons between generating and comparison models, the majority of simulated subjects were best fit by the generating model. The RSTD-simulated subjects who were better fit by the TD model were those who had less extreme AI values (<xref ref-type="fig" rid="app1fig7">Appendix 1—figure 7</xref>), and thus could be more parsimoniously captured by a model with a single learning rate. We also found that RSTD model parameters were reasonably well recovered across the range of AI observed in our empirical sample (see <xref ref-type="fig" rid="app1fig8">Appendix 1—figure 8</xref>).</p><table-wrap id="table2" position="float"><label>Table 2.</label><caption><title>Model recovery.</title></caption><table frame="hsides" rules="groups"><thead><tr><th align="left" valign="bottom"/><th align="left" valign="bottom"/><th align="left" colspan="4" valign="bottom">Comparison model</th></tr><tr><th align="left" valign="bottom"> </th><th align="left" valign="bottom"> </th><th align="left" valign="bottom">TD</th><th align="left" valign="bottom">RSTD</th><th align="left" valign="bottom">FourLR</th><th align="left" valign="bottom">Utility</th></tr></thead><tbody><tr><td align="left" rowspan="4" valign="bottom"> Generating model</td><td align="left" valign="bottom">TD</td><td align="left" valign="bottom">-</td><td align="char" char="." valign="bottom">0.98</td><td align="char" char="." valign="bottom">1.00</td><td align="char" char="." valign="bottom">0.97</td></tr><tr><td align="left" valign="bottom">RSTD</td><td align="char" char="." valign="bottom">0.57</td><td align="left" valign="bottom">-</td><td align="char" char="." valign="bottom">0.99</td><td align="char" char="." valign="bottom">0.65</td></tr><tr><td align="left" valign="bottom">FourLR</td><td align="char" char="." valign="bottom">0.50</td><td align="char" char="." valign="bottom">0.31</td><td align="left" valign="bottom">-</td><td align="char" char="." valign="bottom">0.39</td></tr><tr><td align="left" valign="bottom">Utility</td><td align="char" char="." valign="bottom">0.58</td><td align="char" char="." valign="bottom">0.76</td><td align="char" char="." valign="bottom">0.99</td><td align="left" valign="bottom">-</td></tr></tbody></table><table-wrap-foot><fn><p>TD, temporal difference; RSTD, risk-sensitive temporal difference; LR, learning rate.</p></fn></table-wrap-foot></table-wrap><p>Values in this table indicate the proportion of participants simulated by the generating model who are best fit by the generating model in a pairwise comparison with each alternative model.</p></sec><sec id="s4-4"><title>Statistical analyses</title><p>Statistical analyses were performed in R version 4.0.2 (<xref ref-type="bibr" rid="bib74">R Development Core Team, 2016</xref>) with a two-tailed alpha threshold of p<0.05. For tests of trial-wise effects, we ran linear mixed-effects regression (lmer) or generalized linear mixed-effects regression (glmer) models (lme4 package; <xref ref-type="bibr" rid="bib5">Bates et al., 2015</xref>), which included participant as a random effect, and estimated random intercepts and slopes for each fixed effect. We used the ‘bobyqa’ optimizer with 1 million iterations. Trial number was included in lmer and glmer regression models. All independent variables were z-scored. We began with this maximal model, which converged for all analyses except one, for which we systematically reduced the complexity of the model until it converged (<xref ref-type="bibr" rid="bib4">Barr et al., 2013</xref>; see Appendix 1). In RT analyses, we removed responses that were less than 0.2 s (n = 22, out of 11,309 total trials, with a maximum of 9 for one participant) and log-transformed RT prior to running regressions. To test for linear effects of age, we included z-scored age in regression models. Potential quadratic age effects were assessed by adding a squared z-scored age term in a regression model. We used the anova function to arbitrate between these regression models and report only linear age effects if the addition of quadratic age did not significantly improve model fit. To probe whether a quadratic age effect qualifies as u-shaped, we used the two-lines approach (<xref ref-type="bibr" rid="bib90">Simonsohn, 2018</xref>), which algorithmically determines a break point in the distribution and tests whether regression lines on either side of the break point have significant slopes with opposite signs.</p></sec><sec id="s4-5"><title>Reporting</title><p>For one-way and paired <italic>t</italic>-tests, we report <italic>t</italic>-statistics, p-values, and Cohen’s d with 95% confidence intervals (CIs; using the function cohens_d in the rstatix package [one-way <italic>t</italic>-test] or the effectsize package [paired <italic>t</italic>-test]).</p><p>For linear regressions, we report unstandardized regression coefficients, <italic>t</italic>-statistics, and p-values. We also report Cohen’s <italic>f<sup>2</sup></italic> with 95% CIs, a standardized effect size measure (<xref ref-type="bibr" rid="bib22">Cohen, 1992</xref>) computed by squaring the output of the function cohens_f in the effectsize package.</p><p>For multilevel models, we report test statistics (<italic>t</italic> for linear mixed-effects models and <italic>z</italic> for generalized linear mixed-effects models), p-values, and unstandardized effect sizes with 95% CIs (unstandardized coefficients for linear mixed-effects models, and odds ratios for generalized linear mixed-effects models).</p></sec><sec id="s4-6"><title>Experiment 2</title><p>Next, we looked for evidence that valence biases in learning influence memory in a previously published independent dataset (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>). Notably, results from this study suggested that unsigned PEs (i.e., PEs of greater magnitude whether negative or positive) facilitate subsequent memory, but no signed effect was observed. Here, we examined whether signed valence-specific effects might be evident when we account for individual differences in learning.</p><p>Briefly, adult participants completed a Pavlovian learning task (i.e., participants did not make choices and could not influence the observed outcomes) in which they encountered indoor and outdoor scenes. One type of scene had higher average value than the other. On each trial, an indoor or outdoor image was displayed, and participants provided an explicit prediction for the average value of that scene type. After the learning task, participants completed a memory test for the scenes. A detailed description of the experimental paradigm can be found in the original publication (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>).</p><p>In order to derive each participant’s AI, we fit an ‘Explicit Prediction’ RL model to the participants’ estimation data (see Appendix 1 for more details on our model specification and fitting procedure). Similar to our RSTD model, this model included separate learning rates for trials with positive and negative PEs.</p><p>Importantly, the RSTD model and the Explicit Prediction model differed in that the RSTD model included a β parameter, while the Explicit Prediction model did not. In Experiment 1, this extra parameter allowed us to use the softmax function to convert the relative estimated values of the two machines into a probability of choosing each machine presented, which we then compared to participants’ actual choices during maximum likelihood estimation. In contrast, in Experiment 2, participants explicitly reported their value predictions (and did not make choices), so the model’s free parameters were fit by minimizing the difference between the model’s value estimates and participants’ explicit predictions.</p></sec></sec></body><back><sec id="s5" sec-type="additional-information"><title>Additional information</title><fn-group content-type="competing-interest"><title>Competing interests</title><fn fn-type="COI-statement" id="conf1"><p>No competing interests declared</p></fn><fn fn-type="COI-statement" id="conf2"><p>No competing interests declared</p></fn></fn-group><fn-group content-type="author-contribution"><title>Author contributions</title><fn fn-type="con" id="con1"><p>Conceptualization, Data curation, Formal analysis, Funding acquisition, Methodology, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review and editing</p></fn><fn fn-type="con" id="con2"><p>Data curation, Formal analysis, Investigation, Project administration, Writing – original draft</p></fn><fn fn-type="con" id="con3"><p>Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Writing – original draft, Writing – review and editing</p></fn></fn-group><fn-group content-type="ethics-information"><title>Ethics</title><fn fn-type="other"><p>Human subjects: Participants provided informed consent (adults) or assent and parental consent (children and adolescents). The study protocol was approved by the New York University Institutional Review Board (IRB#2016-1194).</p></fn></fn-group></sec><sec id="s6" sec-type="supplementary-material"><title>Additional files</title><supplementary-material id="transrepform"><label>Transparent reporting form</label><media mime-subtype="pdf" mimetype="application" xlink:href="elife-64620-transrepform1-v1.pdf"/></supplementary-material><supplementary-material id="supp1"><label>Supplementary file 1.</label><caption><title>Results from an ordinal model predicting memory performance.</title></caption><media mime-subtype="docx" mimetype="application" xlink:href="elife-64620-supp1-v1.docx"/></supplementary-material></sec><sec id="s7" sec-type="data-availability"><title>Data availability</title><p>Data and code are available on the Open Science Framework. Experiment 1 data were generated in the present study. Experiment 2 data are provided in our repository, but were collected as part of the following study: Rouhani, N., Norman, K. A., & Niv, Y. (2018). Dissociable effects of surprising rewards on learning and memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(9), 1430-1443. <ext-link ext-link-type="uri" xlink:href="https://doi.org/10.1037/xlm0000518">https://doi.org/10.1037/xlm0000518</ext-link>.</p><p>The following dataset was generated:</p><p><element-citation id="dataset1" publication-type="data" specific-use="isSupplementedBy"><person-group person-group-type="author"><name><surname>Rosenbaum</surname><given-names>GM</given-names></name><name><surname>Grassie</surname><given-names>HL</given-names></name><name><surname>Hartley</surname><given-names>CA</given-names></name></person-group><year iso-8601-date="2020">2020</year><data-title>Valence biases in reinforcement learning shift across adolescence and modulate subsequent memory</data-title><source>Open Science Framework</source><pub-id pub-id-type="doi">10.17605/OSF.IO/SRTGC</pub-id></element-citation></p></sec><ack id="ack"><title>Acknowledgements</title><p>We thank Nina Rouhani and Yael Niv for sharing their data and providing feedback on the analyses in Experiment 2, and Alexandra Cohen and Kate Nussenbaum for providing constructive feedback on the manuscript.</p><p>This work was funded by an NSF CAREER grant (1654393), an NIMH BRAINS grant (R01MH126183), the NYU Vulnerable Brain Project, and a Jacobs Foundation Early Career Award awarded to CAH, and by a National Institute on Drug Abuse NRSA awarded to GMR (F32DA047047).</p></ack><ref-list><title>References</title><ref id="bib1"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bakkour</surname><given-names>A</given-names></name><name><surname>Palombo</surname><given-names>DJ</given-names></name><name><surname>Zylberberg</surname><given-names>A</given-names></name><name><surname>Kang</surname><given-names>YH</given-names></name><name><surname>Reid</surname><given-names>A</given-names></name><name><surname>Verfaellie</surname><given-names>M</given-names></name><name><surname>Shadlen</surname><given-names>MN</given-names></name><name><surname>Shohamy</surname><given-names>D</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>The hippocampus supports deliberation during value-based decisions</article-title><source>eLife</source><volume>8</volume><elocation-id>e46080</elocation-id><pub-id pub-id-type="doi">10.7554/eLife.46080</pub-id><pub-id pub-id-type="pmid">31268419</pub-id></element-citation></ref><ref id="bib2"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bar-Haim</surname><given-names>Y</given-names></name><name><surname>Lamy</surname><given-names>D</given-names></name><name><surname>Pergamin</surname><given-names>L</given-names></name><name><surname>Bakermans-Kranenburg</surname><given-names>MJ</given-names></name><name><surname>van IJzendoorn</surname><given-names>MH</given-names></name></person-group><year iso-8601-date="2007">2007</year><article-title>Threat-related attentional bias in anxious and nonanxious individuals: a meta-analytic study</article-title><source>Psychological Bulletin</source><volume>133</volume><fpage>1</fpage><lpage>24</lpage><pub-id pub-id-type="doi">10.1037/0033-2909.133.1.1</pub-id><pub-id pub-id-type="pmid">17201568</pub-id></element-citation></ref><ref id="bib3"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Barkley-Levenson</surname><given-names>EE</given-names></name><name><surname>Van Leijenhorst</surname><given-names>L</given-names></name><name><surname>Galván</surname><given-names>A</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>Behavioral and neural correlates of loss aversion and risk avoidance in adolescents and adults</article-title><source>Developmental Cognitive Neuroscience</source><volume>3</volume><fpage>72</fpage><lpage>83</lpage><pub-id pub-id-type="doi">10.1016/j.dcn.2012.09.007</pub-id><pub-id pub-id-type="pmid">23245222</pub-id></element-citation></ref><ref id="bib4"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Barr</surname><given-names>DJ</given-names></name><name><surname>Levy</surname><given-names>R</given-names></name><name><surname>Scheepers</surname><given-names>C</given-names></name><name><surname>Tily</surname><given-names>HJ</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>Random effects structure for confirmatory hypothesis testing: Keep it maximal</article-title><source>Journal of Memory and Language</source><volume>68</volume><fpage>255</fpage><lpage>278</lpage><pub-id pub-id-type="doi">10.1016/j.jml.2012.11.001</pub-id></element-citation></ref><ref id="bib5"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bates</surname><given-names>D</given-names></name><name><surname>Mächler</surname><given-names>M</given-names></name><name><surname>Bolker</surname><given-names>B</given-names></name><name><surname>Walker</surname><given-names>S</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Fitting Linear Mixed-Effects Models using lme4</article-title><source>Journal of Statistical Software</source><volume>67</volume><elocation-id>51</elocation-id><pub-id pub-id-type="doi">10.18637/jss.v067.i01</pub-id></element-citation></ref><ref id="bib6"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bechara</surname><given-names>A</given-names></name><name><surname>Damasio</surname><given-names>H</given-names></name><name><surname>Tranel</surname><given-names>D</given-names></name><name><surname>Damasio</surname><given-names>AR</given-names></name></person-group><year iso-8601-date="1997">1997</year><article-title>Deciding advantageously before knowing the advantageous strategy</article-title><source>Science</source><volume>275</volume><fpage>1293</fpage><lpage>1295</lpage><pub-id pub-id-type="doi">10.1126/science.275.5304.1293</pub-id><pub-id pub-id-type="pmid">9036851</pub-id></element-citation></ref><ref id="bib7"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bernoulli</surname><given-names>D</given-names></name></person-group><year iso-8601-date="1954">1954</year><article-title>Exposition of a New Theory on the Measurement of Risk</article-title><source>Econometrica: Journal of the Econometric Society</source><volume>22</volume><elocation-id>23</elocation-id><pub-id pub-id-type="doi">10.2307/1909829</pub-id></element-citation></ref><ref id="bib8"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Biderman</surname><given-names>N</given-names></name><name><surname>Bakkour</surname><given-names>A</given-names></name><name><surname>Shohamy</surname><given-names>D</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>What Are Memories For? The Hippocampus Bridges Past Experience with Future Decisions</article-title><source>Trends in Cognitive Sciences</source><volume>24</volume><fpage>542</fpage><lpage>556</lpage><pub-id pub-id-type="doi">10.1016/j.tics.2020.04.004</pub-id><pub-id pub-id-type="pmid">32513572</pub-id></element-citation></ref><ref id="bib9"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Blais</surname><given-names>AR</given-names></name><name><surname>Weber</surname><given-names>EU</given-names></name></person-group><year iso-8601-date="2006">2006</year><article-title>A Domain-Specific Risk-Taking (DOSPERT) scale for adult populations</article-title><source>Judgment and Decision Making</source><volume>1</volume><fpage>33</fpage><lpage>47</lpage></element-citation></ref><ref id="bib10"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Blakemore</surname><given-names>SJ</given-names></name></person-group><year iso-8601-date="2008">2008</year><article-title>The social brain in adolescence</article-title><source>Nature Reviews. Neuroscience</source><volume>9</volume><fpage>267</fpage><lpage>277</lpage><pub-id pub-id-type="doi">10.1038/nrn2353</pub-id><pub-id pub-id-type="pmid">18354399</pub-id></element-citation></ref><ref id="bib11"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bornstein</surname><given-names>AM</given-names></name><name><surname>Khaw</surname><given-names>MW</given-names></name><name><surname>Shohamy</surname><given-names>D</given-names></name><name><surname>Daw</surname><given-names>ND</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Reminders of past choices bias decisions for reward in humans</article-title><source>Nature Communications</source><volume>8</volume><elocation-id>15958</elocation-id><pub-id pub-id-type="doi">10.1038/ncomms15958</pub-id><pub-id pub-id-type="pmid">28653668</pub-id></element-citation></ref><ref id="bib12"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Braams</surname><given-names>BR</given-names></name><name><surname>van Duijvenvoorde</surname><given-names>ACK</given-names></name><name><surname>Peper</surname><given-names>JS</given-names></name><name><surname>Crone</surname><given-names>EA</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Longitudinal changes in adolescent risk-taking: a comprehensive study of neural responses to rewards, pubertal development, and risk-taking behavior</article-title><source>The Journal of Neuroscience</source><volume>35</volume><fpage>7226</fpage><lpage>7238</lpage><pub-id pub-id-type="doi">10.1523/JNEUROSCI.4764-14.2015</pub-id><pub-id pub-id-type="pmid">25948271</pub-id></element-citation></ref><ref id="bib13"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Brodeur</surname><given-names>MB</given-names></name><name><surname>Dionne-Dostie</surname><given-names>E</given-names></name><name><surname>Montreuil</surname><given-names>T</given-names></name><name><surname>Lepage</surname><given-names>M</given-names></name></person-group><year iso-8601-date="2010">2010</year><article-title>The Bank of Standardized Stimuli (BOSS), a new set of 480 normative photos of objects to be used as visual stimuli in cognitive research</article-title><source>PLOS ONE</source><volume>5</volume><elocation-id>e10773</elocation-id><pub-id pub-id-type="doi">10.1371/journal.pone.0010773</pub-id><pub-id pub-id-type="pmid">20532245</pub-id></element-citation></ref><ref id="bib14"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Brodeur</surname><given-names>MB</given-names></name><name><surname>Guérard</surname><given-names>K</given-names></name><name><surname>Bouras</surname><given-names>M</given-names></name></person-group><year iso-8601-date="2014">2014</year><article-title>Bank of Standardized Stimuli (BOSS) phase II: 930 new normative photos</article-title><source>PLOS ONE</source><volume>9</volume><elocation-id>e106953</elocation-id><pub-id pub-id-type="doi">10.1371/journal.pone.0106953</pub-id><pub-id pub-id-type="pmid">25211489</pub-id></element-citation></ref><ref id="bib15"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Casey</surname><given-names>BJ</given-names></name><name><surname>Getz</surname><given-names>S</given-names></name><name><surname>Galvan</surname><given-names>A</given-names></name></person-group><year iso-8601-date="2008">2008</year><article-title>The adolescent brain</article-title><source>Developmental Review</source><volume>28</volume><fpage>62</fpage><lpage>77</lpage><pub-id pub-id-type="doi">10.1016/j.dr.2007.08.003</pub-id><pub-id pub-id-type="pmid">18688292</pub-id></element-citation></ref><ref id="bib16"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cazé</surname><given-names>RD</given-names></name><name><surname>van der Meer</surname><given-names>MAA</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>Adaptive properties of differential learning rates for positive and negative outcomes</article-title><source>Biological Cybernetics</source><volume>107</volume><fpage>711</fpage><lpage>719</lpage><pub-id pub-id-type="doi">10.1007/s00422-013-0571-5</pub-id><pub-id pub-id-type="pmid">24085507</pub-id></element-citation></ref><ref id="bib17"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Chambon</surname><given-names>V</given-names></name><name><surname>Théro</surname><given-names>H</given-names></name><name><surname>Vidal</surname><given-names>M</given-names></name><name><surname>Vandendriessche</surname><given-names>H</given-names></name><name><surname>Haggard</surname><given-names>P</given-names></name><name><surname>Palminteri</surname><given-names>S</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>Information about action outcomes differentially affects learning from self-determined versus imposed choices</article-title><source>Nature Human Behaviour</source><volume>4</volume><fpage>1067</fpage><lpage>1079</lpage><pub-id pub-id-type="doi">10.1038/s41562-020-0919-5</pub-id><pub-id pub-id-type="pmid">32747804</pub-id></element-citation></ref><ref id="bib18"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Christakou</surname><given-names>A</given-names></name><name><surname>Gershman</surname><given-names>SJ</given-names></name><name><surname>Niv</surname><given-names>Y</given-names></name><name><surname>Simmons</surname><given-names>A</given-names></name><name><surname>Brammer</surname><given-names>M</given-names></name><name><surname>Rubia</surname><given-names>K</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>Neural and Psychological Maturation of Decision-making in Adolescence and Young Adulthood</article-title><source>Journal of Cognitive Neuroscience</source><volume>25</volume><fpage>1807</fpage><lpage>1823</lpage><pub-id pub-id-type="doi">10.1162/jocn_a_00447</pub-id><pub-id pub-id-type="pmid">23859647</pub-id></element-citation></ref><ref id="bib19"><element-citation publication-type="software"><person-group person-group-type="author"><name><surname>Christensen</surname><given-names>RHB</given-names></name></person-group><year iso-8601-date="2019">2019</year><data-title>ordinal: Regression Models for Ordinal Data</data-title><version designator="2019.12-10">2019.12-10</version><source>Github</source><ext-link ext-link-type="uri" xlink:href="https://github.com/runehaubo/ordinal">https://github.com/runehaubo/ordinal</ext-link></element-citation></ref><ref id="bib20"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Chun</surname><given-names>MM</given-names></name><name><surname>Turk-Browne</surname><given-names>NB</given-names></name></person-group><year iso-8601-date="2007">2007</year><article-title>Interactions between attention and memory</article-title><source>Current Opinion in Neurobiology</source><volume>17</volume><fpage>177</fpage><lpage>184</lpage><pub-id pub-id-type="doi">10.1016/j.conb.2007.03.005</pub-id><pub-id pub-id-type="pmid">17379501</pub-id></element-citation></ref><ref id="bib21"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cockburn</surname><given-names>J</given-names></name><name><surname>Collins</surname><given-names>AGE</given-names></name><name><surname>Frank</surname><given-names>MJ</given-names></name></person-group><year iso-8601-date="2014">2014</year><article-title>A reinforcement learning mechanism responsible for the valuation of free choice</article-title><source>Neuron</source><volume>83</volume><fpage>551</fpage><lpage>557</lpage><pub-id pub-id-type="doi">10.1016/j.neuron.2014.06.035</pub-id><pub-id pub-id-type="pmid">25066083</pub-id></element-citation></ref><ref id="bib22"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cohen</surname><given-names>J</given-names></name></person-group><year iso-8601-date="1992">1992</year><article-title>A power primer</article-title><source>Psychological Bulletin</source><volume>112</volume><fpage>155</fpage><lpage>159</lpage><pub-id pub-id-type="doi">10.1037//0033-2909.112.1.155</pub-id><pub-id pub-id-type="pmid">19565683</pub-id></element-citation></ref><ref id="bib23"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cohen</surname><given-names>JR</given-names></name><name><surname>Asarnow</surname><given-names>RF</given-names></name><name><surname>Sabb</surname><given-names>FW</given-names></name><name><surname>Bilder</surname><given-names>RM</given-names></name><name><surname>Bookheimer</surname><given-names>SY</given-names></name><name><surname>Knowlton</surname><given-names>BJ</given-names></name><name><surname>Poldrack</surname><given-names>RA</given-names></name></person-group><year iso-8601-date="2010">2010</year><article-title>A unique adolescent response to reward prediction errors</article-title><source>Nature Neuroscience</source><volume>13</volume><fpage>669</fpage><lpage>671</lpage><pub-id pub-id-type="doi">10.1038/nn.2558</pub-id><pub-id pub-id-type="pmid">20473290</pub-id></element-citation></ref><ref id="bib24"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cox</surname><given-names>SML</given-names></name><name><surname>Frank</surname><given-names>MJ</given-names></name><name><surname>Larcher</surname><given-names>K</given-names></name><name><surname>Fellows</surname><given-names>LK</given-names></name><name><surname>Clark</surname><given-names>CA</given-names></name><name><surname>Leyton</surname><given-names>M</given-names></name><name><surname>Dagher</surname><given-names>A</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Striatal D1 and D2 signaling differentially predict learning from positive and negative outcomes</article-title><source>NeuroImage</source><volume>109</volume><fpage>95</fpage><lpage>101</lpage><pub-id pub-id-type="doi">10.1016/j.neuroimage.2014.12.070</pub-id><pub-id pub-id-type="pmid">25562824</pub-id></element-citation></ref><ref id="bib25"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dabney</surname><given-names>W</given-names></name><name><surname>Kurth-Nelson</surname><given-names>Z</given-names></name><name><surname>Uchida</surname><given-names>N</given-names></name><name><surname>Starkweather</surname><given-names>CK</given-names></name><name><surname>Hassabis</surname><given-names>D</given-names></name><name><surname>Munos</surname><given-names>R</given-names></name><name><surname>Botvinick</surname><given-names>M</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>A distributional code for value in dopamine-based reinforcement learning</article-title><source>Nature</source><volume>577</volume><fpage>671</fpage><lpage>675</lpage><pub-id pub-id-type="doi">10.1038/s41586-019-1924-6</pub-id><pub-id pub-id-type="pmid">31942076</pub-id></element-citation></ref><ref id="bib26"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Davidow</surname><given-names>JY</given-names></name><name><surname>Foerde</surname><given-names>K</given-names></name><name><surname>Galván</surname><given-names>A</given-names></name><name><surname>Shohamy</surname><given-names>D</given-names></name></person-group><year iso-8601-date="2016">2016</year><article-title>An Upside to Reward Sensitivity: The Hippocampus Supports Enhanced Reinforcement Learning in Adolescence</article-title><source>Neuron</source><volume>92</volume><fpage>93</fpage><lpage>99</lpage><pub-id pub-id-type="doi">10.1016/j.neuron.2016.08.031</pub-id><pub-id pub-id-type="pmid">27710793</pub-id></element-citation></ref><ref id="bib27"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Daw</surname><given-names>ND</given-names></name><name><surname>Kakade</surname><given-names>S</given-names></name><name><surname>Dayan</surname><given-names>P</given-names></name></person-group><year iso-8601-date="2002">2002</year><article-title>Opponent interactions between serotonin and dopamine</article-title><source>Neural Networks</source><volume>15</volume><fpage>603</fpage><lpage>616</lpage><pub-id pub-id-type="doi">10.1016/s0893-6080(02)00052-7</pub-id><pub-id pub-id-type="pmid">12371515</pub-id></element-citation></ref><ref id="bib28"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dayan</surname><given-names>P</given-names></name><name><surname>Kakade</surname><given-names>S</given-names></name><name><surname>Montague</surname><given-names>PR</given-names></name></person-group><year iso-8601-date="2000">2000</year><article-title>Learning and selective attention</article-title><source>Nature Neuroscience</source><volume>3</volume><fpage>1218</fpage><lpage>1223</lpage><pub-id pub-id-type="doi">10.1038/81504</pub-id><pub-id pub-id-type="pmid">11127841</pub-id></element-citation></ref><ref id="bib29"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Decker</surname><given-names>JH</given-names></name><name><surname>Lourenco</surname><given-names>FS</given-names></name><name><surname>Doll</surname><given-names>BB</given-names></name><name><surname>Hartley</surname><given-names>CA</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Experiential reward learning outweighs instruction prior to adulthood</article-title><source>Cognitive, Affective & Behavioral Neuroscience</source><volume>15</volume><fpage>310</fpage><lpage>320</lpage><pub-id pub-id-type="doi">10.3758/s13415-014-0332-5</pub-id><pub-id pub-id-type="pmid">25582607</pub-id></element-citation></ref><ref id="bib30"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Defoe</surname><given-names>IN</given-names></name><name><surname>Dubas</surname><given-names>JS</given-names></name><name><surname>Figner</surname><given-names>B</given-names></name><name><surname>van Aken</surname><given-names>MAG</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>A meta-analysis on age differences in risky decision making: adolescents versus children and adults</article-title><source>Psychological Bulletin</source><volume>141</volume><fpage>48</fpage><lpage>84</lpage><pub-id pub-id-type="doi">10.1037/a0038088</pub-id><pub-id pub-id-type="pmid">25365761</pub-id></element-citation></ref><ref id="bib31"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dillon</surname><given-names>DG</given-names></name><name><surname>Pizzagalli</surname><given-names>DA</given-names></name></person-group><year iso-8601-date="2018">2018</year><article-title>Mechanisms of Memory Disruption in Depression</article-title><source>Trends in Neurosciences</source><volume>41</volume><fpage>137</fpage><lpage>149</lpage><pub-id pub-id-type="doi">10.1016/j.tins.2017.12.006</pub-id><pub-id pub-id-type="pmid">29331265</pub-id></element-citation></ref><ref id="bib32"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Duncan</surname><given-names>K</given-names></name><name><surname>Semmler</surname><given-names>A</given-names></name><name><surname>Shohamy</surname><given-names>D</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Modulating the Use of Multiple Memory Systems in Value-based Decisions with Contextual Novelty</article-title><source>Journal of Cognitive Neuroscience</source><volume>31</volume><fpage>1455</fpage><lpage>1467</lpage><pub-id pub-id-type="doi">10.1162/jocn_a_01447</pub-id><pub-id pub-id-type="pmid">31322467</pub-id></element-citation></ref><ref id="bib33"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dunsmoor</surname><given-names>JE</given-names></name><name><surname>Murty</surname><given-names>VP</given-names></name><name><surname>Davachi</surname><given-names>L</given-names></name><name><surname>Phelps</surname><given-names>EA</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Emotional learning selectively and retroactively strengthens memories for related events</article-title><source>Nature</source><volume>520</volume><fpage>345</fpage><lpage>348</lpage><pub-id pub-id-type="doi">10.1038/nature14106</pub-id><pub-id pub-id-type="pmid">25607357</pub-id></element-citation></ref><ref id="bib34"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ergo</surname><given-names>K</given-names></name><name><surname>De Loof</surname><given-names>E</given-names></name><name><surname>Verguts</surname><given-names>T</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>Reward Prediction Error and Declarative Memory</article-title><source>Trends in Cognitive Sciences</source><volume>24</volume><fpage>388</fpage><lpage>397</lpage><pub-id pub-id-type="doi">10.1016/j.tics.2020.02.009</pub-id><pub-id pub-id-type="pmid">32298624</pub-id></element-citation></ref><ref id="bib35"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Frank</surname><given-names>MJ</given-names></name><name><surname>Seeberger</surname><given-names>LC</given-names></name><name><surname>O’reilly</surname><given-names>RC</given-names></name></person-group><year iso-8601-date="2004">2004</year><article-title>By carrot or by stick: cognitive reinforcement learning in parkinsonism</article-title><source>Science</source><volume>306</volume><fpage>1940</fpage><lpage>1943</lpage><pub-id pub-id-type="doi">10.1126/science.1102941</pub-id><pub-id pub-id-type="pmid">15528409</pub-id></element-citation></ref><ref id="bib36"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Frank</surname><given-names>MJ</given-names></name><name><surname>Moustafa</surname><given-names>AA</given-names></name><name><surname>Haughey</surname><given-names>HM</given-names></name><name><surname>Curran</surname><given-names>T</given-names></name><name><surname>Hutchison</surname><given-names>KE</given-names></name></person-group><year iso-8601-date="2007">2007</year><article-title>Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning</article-title><source>PNAS</source><volume>104</volume><fpage>16311</fpage><lpage>16316</lpage><pub-id pub-id-type="doi">10.1073/pnas.0706111104</pub-id><pub-id pub-id-type="pmid">17913879</pub-id></element-citation></ref><ref id="bib37"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Gaddy</surname><given-names>MA</given-names></name><name><surname>Ingram</surname><given-names>RE</given-names></name></person-group><year iso-8601-date="2014">2014</year><article-title>A meta-analytic review of mood-congruent implicit memory in depressed mood</article-title><source>Clinical Psychology Review</source><volume>34</volume><fpage>402</fpage><lpage>416</lpage><pub-id pub-id-type="doi">10.1016/j.cpr.2014.06.001</pub-id><pub-id pub-id-type="pmid">24980699</pub-id></element-citation></ref><ref id="bib38"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Galvan</surname><given-names>A</given-names></name><name><surname>Hare</surname><given-names>TA</given-names></name><name><surname>Parra</surname><given-names>CE</given-names></name><name><surname>Penn</surname><given-names>J</given-names></name><name><surname>Voss</surname><given-names>H</given-names></name><name><surname>Glover</surname><given-names>G</given-names></name><name><surname>Casey</surname><given-names>BJ</given-names></name></person-group><year iso-8601-date="2006">2006</year><article-title>Earlier development of the accumbens relative to orbitofrontal cortex might underlie risk-taking behavior in adolescents</article-title><source>The Journal of Neuroscience</source><volume>26</volume><fpage>6885</fpage><lpage>6892</lpage><pub-id pub-id-type="doi">10.1523/JNEUROSCI.1062-06.2006</pub-id><pub-id pub-id-type="pmid">16793895</pub-id></element-citation></ref><ref id="bib39"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Galván</surname><given-names>A</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>The Teenage Brain: Sensitivity to Rewards</article-title><source>Current Directions in Psychological Science</source><volume>22</volume><fpage>88</fpage><lpage>93</lpage><pub-id pub-id-type="doi">10.1177/0963721413480859</pub-id></element-citation></ref><ref id="bib40"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Galván</surname><given-names>A</given-names></name><name><surname>McGlennen</surname><given-names>KM</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>Enhanced striatal sensitivity to aversive reinforcement in adolescents versus adults</article-title><source>Journal of Cognitive Neuroscience</source><volume>25</volume><fpage>284</fpage><lpage>296</lpage><pub-id pub-id-type="doi">10.1162/jocn_a_00326</pub-id><pub-id pub-id-type="pmid">23163417</pub-id></element-citation></ref><ref id="bib41"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Gershman</surname><given-names>SJ</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Do learning rates adapt to the distribution of rewards?</article-title><source>Psychonomic Bulletin & Review</source><volume>22</volume><fpage>1320</fpage><lpage>1327</lpage><pub-id pub-id-type="doi">10.3758/s13423-014-0790-3</pub-id></element-citation></ref><ref id="bib42"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Hauser</surname><given-names>TU</given-names></name><name><surname>Iannaccone</surname><given-names>R</given-names></name><name><surname>Walitza</surname><given-names>S</given-names></name><name><surname>Brandeis</surname><given-names>D</given-names></name><name><surname>Brem</surname><given-names>S</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Cognitive flexibility in adolescence: neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development</article-title><source>NeuroImage</source><volume>104</volume><fpage>347</fpage><lpage>354</lpage><pub-id pub-id-type="doi">10.1016/j.neuroimage.2014.09.018</pub-id><pub-id pub-id-type="pmid">25234119</pub-id></element-citation></ref><ref id="bib43"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Holland</surname><given-names>PC</given-names></name><name><surname>Schiffino</surname><given-names>FL</given-names></name></person-group><year iso-8601-date="2016">2016</year><article-title>Mini-review: Prediction errors, attention and associative learning</article-title><source>Neurobiology of Learning and Memory</source><volume>131</volume><fpage>207</fpage><lpage>215</lpage><pub-id pub-id-type="doi">10.1016/j.nlm.2016.02.014</pub-id><pub-id pub-id-type="pmid">26948122</pub-id></element-citation></ref><ref id="bib44"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Jang</surname><given-names>AI</given-names></name><name><surname>Nassar</surname><given-names>MR</given-names></name><name><surname>Dillon</surname><given-names>DG</given-names></name><name><surname>Frank</surname><given-names>MJ</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Positive reward prediction errors during decision-making strengthen memory encoding</article-title><source>Nature Human Behaviour</source><volume>3</volume><fpage>719</fpage><lpage>732</lpage><pub-id pub-id-type="doi">10.1038/s41562-019-0597-3</pub-id></element-citation></ref><ref id="bib45"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Jones</surname><given-names>RM</given-names></name><name><surname>Somerville</surname><given-names>LH</given-names></name><name><surname>Li</surname><given-names>J</given-names></name><name><surname>Ruberry</surname><given-names>EJ</given-names></name><name><surname>Powers</surname><given-names>A</given-names></name><name><surname>Mehta</surname><given-names>N</given-names></name><name><surname>Dyke</surname><given-names>J</given-names></name><name><surname>Casey</surname><given-names>BJ</given-names></name></person-group><year iso-8601-date="2014">2014</year><article-title>Adolescent-specific patterns of behavior and neural activity during social reinforcement learning</article-title><source>Cognitive, Affective & Behavioral Neuroscience</source><volume>14</volume><fpage>683</fpage><lpage>697</lpage><pub-id pub-id-type="doi">10.3758/s13415-014-0257-z</pub-id><pub-id pub-id-type="pmid">24550063</pub-id></element-citation></ref><ref id="bib46"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kahneman</surname><given-names>D</given-names></name><name><surname>Tversky</surname><given-names>A</given-names></name></person-group><year iso-8601-date="1979">1979</year><article-title>Prospect Theory: An Analysis of Decision under Risk</article-title><source>Econometrica: Journal of the Econometric Society</source><volume>47</volume><elocation-id>263</elocation-id><pub-id pub-id-type="doi">10.2307/1914185</pub-id></element-citation></ref><ref id="bib47"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kalbe</surname><given-names>F</given-names></name><name><surname>Schwabe</surname><given-names>L</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>Beyond arousal: Prediction error related to aversive events promotes episodic memory formation</article-title><source>Journal of Experimental Psychology. Learning, Memory, and Cognition</source><volume>46</volume><fpage>234</fpage><lpage>246</lpage><pub-id pub-id-type="doi">10.1037/xlm0000728</pub-id><pub-id pub-id-type="pmid">31169402</pub-id></element-citation></ref><ref id="bib48"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kann</surname><given-names>L</given-names></name><name><surname>McManus</surname><given-names>T</given-names></name><name><surname>Harris</surname><given-names>WA</given-names></name><name><surname>Shanklin</surname><given-names>SL</given-names></name><name><surname>Flint</surname><given-names>KH</given-names></name><name><surname>Queen</surname><given-names>B</given-names></name><name><surname>Lowry</surname><given-names>R</given-names></name><name><surname>Chyen</surname><given-names>D</given-names></name><name><surname>Whittle</surname><given-names>L</given-names></name><name><surname>Thornton</surname><given-names>J</given-names></name><name><surname>Lim</surname><given-names>C</given-names></name><name><surname>Bradford</surname><given-names>D</given-names></name><name><surname>Yamakawa</surname><given-names>Y</given-names></name><name><surname>Leon</surname><given-names>M</given-names></name><name><surname>Brener</surname><given-names>N</given-names></name><name><surname>Ethier</surname><given-names>KA</given-names></name></person-group><year iso-8601-date="2018">2018</year><article-title>Youth Risk Behavior Surveillance — United States, 2017</article-title><source>MMWR. Surveillance Summaries</source><volume>67</volume><fpage>1</fpage><lpage>114</lpage><pub-id pub-id-type="doi">10.15585/mmwr.ss6708a1</pub-id></element-citation></ref><ref id="bib49"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Katzman</surname><given-names>PL</given-names></name><name><surname>Hartley</surname><given-names>CA</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>The value of choice facilitates subsequent memory across development</article-title><source>Cognition</source><volume>199</volume><elocation-id>104239</elocation-id><pub-id pub-id-type="doi">10.1016/j.cognition.2020.104239</pub-id></element-citation></ref><ref id="bib50"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kuyken</surname><given-names>W</given-names></name><name><surname>Howell</surname><given-names>R</given-names></name></person-group><year iso-8601-date="2006">2006</year><article-title>Facets of autobiographical memory in adolescents with major depressive disorder and never-depressed controls</article-title><source>Cognition & Emotion</source><volume>20</volume><fpage>466</fpage><lpage>487</lpage><pub-id pub-id-type="doi">10.1080/02699930500342639</pub-id><pub-id pub-id-type="pmid">26529216</pub-id></element-citation></ref><ref id="bib51"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Lee</surname><given-names>FS</given-names></name><name><surname>Heimer</surname><given-names>H</given-names></name><name><surname>Giedd</surname><given-names>JN</given-names></name><name><surname>Lein</surname><given-names>ES</given-names></name><name><surname>Šestan</surname><given-names>N</given-names></name><name><surname>Weinberger</surname><given-names>DR</given-names></name><name><surname>Casey</surname><given-names>BJ</given-names></name></person-group><year iso-8601-date="2014">2014</year><article-title>Mental health. Adolescent mental health--opportunity and obligation</article-title><source>Science</source><volume>346</volume><fpage>547</fpage><lpage>549</lpage><pub-id pub-id-type="doi">10.1126/science.1260497</pub-id><pub-id pub-id-type="pmid">25359951</pub-id></element-citation></ref><ref id="bib52"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Lefebvre</surname><given-names>G</given-names></name><name><surname>Lebreton</surname><given-names>M</given-names></name><name><surname>Meyniel</surname><given-names>F</given-names></name><name><surname>Bourgeois-Gironde</surname><given-names>S</given-names></name><name><surname>Palminteri</surname><given-names>S</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Behavioural and neural characterization of optimistic reinforcement learning</article-title><source>Nature Human Behaviour</source><volume>1</volume><elocation-id>0067</elocation-id><pub-id pub-id-type="doi">10.1038/s41562-017-0067</pub-id></element-citation></ref><ref id="bib53"><element-citation publication-type="confproc"><person-group person-group-type="author"><name><surname>Lengyel</surname><given-names>M</given-names></name><name><surname>Dayan</surname><given-names>P</given-names></name></person-group><year iso-8601-date="2008">2008</year><conf-name>Hippocampal contributions to control: the third way</conf-name><article-title>Advances in Neural Information Processing Systems</article-title></element-citation></ref><ref id="bib54"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Lisman</surname><given-names>JE</given-names></name><name><surname>Grace</surname><given-names>AA</given-names></name></person-group><year iso-8601-date="2005">2005</year><article-title>The hippocampal-VTA loop: controlling the entry of information into long-term memory</article-title><source>Neuron</source><volume>46</volume><fpage>703</fpage><lpage>713</lpage><pub-id pub-id-type="doi">10.1016/j.neuron.2005.05.002</pub-id><pub-id pub-id-type="pmid">15924857</pub-id></element-citation></ref><ref id="bib55"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ludvig</surname><given-names>EA</given-names></name><name><surname>Madan</surname><given-names>CR</given-names></name><name><surname>McMillan</surname><given-names>N</given-names></name><name><surname>Xu</surname><given-names>Y</given-names></name><name><surname>Spetch</surname><given-names>ML</given-names></name></person-group><year iso-8601-date="2018">2018</year><article-title>Living near the edge: How extreme outcomes and their neighbors drive risky choice</article-title><source>Journal of Experimental Psychology. General</source><volume>147</volume><fpage>1905</fpage><lpage>1918</lpage><pub-id pub-id-type="doi">10.1037/xge0000414</pub-id><pub-id pub-id-type="pmid">29565605</pub-id></element-citation></ref><ref id="bib56"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Luna</surname><given-names>B</given-names></name></person-group><year iso-8601-date="2009">2009</year><article-title>Developmental changes in cognitive control through adolescence</article-title><source>Advances in Child Development and Behavior</source><volume>37</volume><fpage>233</fpage><lpage>278</lpage><pub-id pub-id-type="doi">10.1016/s0065-2407(09)03706-9</pub-id><pub-id pub-id-type="pmid">19673164</pub-id></element-citation></ref><ref id="bib57"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Madan</surname><given-names>CR</given-names></name><name><surname>Ludvig</surname><given-names>EA</given-names></name><name><surname>Spetch</surname><given-names>ML</given-names></name></person-group><year iso-8601-date="2014">2014</year><article-title>Remembering the best and worst of times: memories for extreme outcomes bias risky decisions</article-title><source>Psychonomic Bulletin & Review</source><volume>21</volume><fpage>629</fpage><lpage>636</lpage><pub-id pub-id-type="doi">10.3758/s13423-013-0542-9</pub-id><pub-id pub-id-type="pmid">24189991</pub-id></element-citation></ref><ref id="bib58"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Madan</surname><given-names>CR</given-names></name><name><surname>Ludvig</surname><given-names>EA</given-names></name><name><surname>Spetch</surname><given-names>ML</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>The role of memory in distinguishing risky decisions from experience and description</article-title><source>Quarterly Journal of Experimental Psychology</source><volume>70</volume><fpage>2048</fpage><lpage>2059</lpage><pub-id pub-id-type="doi">10.1080/17470218.2016.1220608</pub-id><pub-id pub-id-type="pmid">27602887</pub-id></element-citation></ref><ref id="bib59"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Master</surname><given-names>SL</given-names></name><name><surname>Eckstein</surname><given-names>MK</given-names></name><name><surname>Gotlieb</surname><given-names>N</given-names></name><name><surname>Dahl</surname><given-names>R</given-names></name><name><surname>Wilbrecht</surname><given-names>L</given-names></name><name><surname>Collins</surname><given-names>AGE</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>Distentangling the systems contributing to changes in learning during adolescence</article-title><source>Developmental Cognitive Neuroscience</source><volume>41</volume><elocation-id>100732</elocation-id><pub-id pub-id-type="doi">10.1016/j.dcn.2019.100732</pub-id><pub-id pub-id-type="pmid">31826837</pub-id></element-citation></ref><ref id="bib60"><element-citation publication-type="preprint"><person-group person-group-type="author"><name><surname>Michely</surname><given-names>J</given-names></name><name><surname>Eldar</surname><given-names>E</given-names></name><name><surname>Erdman</surname><given-names>A</given-names></name><name><surname>Martin</surname><given-names>IM</given-names></name><name><surname>Dolan</surname><given-names>RJ</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>SSRIs Modulate Asymmetric Learning from Reward and Punishment</article-title><source>bioRxiv</source><pub-id pub-id-type="doi">10.1101/2020.05.21.108266</pub-id></element-citation></ref><ref id="bib61"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Mihatsch</surname><given-names>O</given-names></name><name><surname>Neuneier</surname><given-names>R</given-names></name></person-group><year iso-8601-date="2002">2002</year><article-title>Risk-Sensitive Reinforcement Learning</article-title><source>Machine Learning</source><volume>49</volume><fpage>267</fpage><lpage>290</lpage><pub-id pub-id-type="doi">10.1023/A:1017940631555</pub-id></element-citation></ref><ref id="bib62"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Mogg</surname><given-names>K</given-names></name><name><surname>Bradley</surname><given-names>BP</given-names></name></person-group><year iso-8601-date="2016">2016</year><article-title>Anxiety and attention to threat: Cognitive mechanisms and treatment with attention bias modification</article-title><source>Behaviour Research and Therapy</source><volume>87</volume><fpage>76</fpage><lpage>108</lpage><pub-id pub-id-type="doi">10.1016/j.brat.2016.08.001</pub-id><pub-id pub-id-type="pmid">27616718</pub-id></element-citation></ref><ref id="bib63"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Moutoussis</surname><given-names>M</given-names></name><name><surname>Bullmore</surname><given-names>ET</given-names></name><name><surname>Goodyer</surname><given-names>IM</given-names></name><name><surname>Fonagy</surname><given-names>P</given-names></name><name><surname>Jones</surname><given-names>PB</given-names></name><name><surname>Dolan</surname><given-names>RJ</given-names></name><name><surname>Dayan</surname><given-names>P</given-names></name><collab>Neuroscience in Psychiatry Network Research Consortium</collab></person-group><year iso-8601-date="2018">2018</year><article-title>Change, stability, and instability in the Pavlovian guidance of behaviour from adolescence to young adulthood</article-title><source>PLOS Computational Biology</source><volume>14</volume><elocation-id>e1006679</elocation-id><pub-id pub-id-type="doi">10.1371/journal.pcbi.1006679</pub-id><pub-id pub-id-type="pmid">30596638</pub-id></element-citation></ref><ref id="bib64"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Murty</surname><given-names>VP</given-names></name><name><surname>DuBrow</surname><given-names>S</given-names></name><name><surname>Davachi</surname><given-names>L</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>The simple act of choosing influences declarative memory</article-title><source>The Journal of Neuroscience</source><volume>35</volume><fpage>6255</fpage><lpage>6264</lpage><pub-id pub-id-type="doi">10.1523/JNEUROSCI.4181-14.2015</pub-id><pub-id pub-id-type="pmid">25904779</pub-id></element-citation></ref><ref id="bib65"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Murty</surname><given-names>VP</given-names></name><name><surname>FeldmanHall</surname><given-names>O</given-names></name><name><surname>Hunter</surname><given-names>LE</given-names></name><name><surname>Phelps</surname><given-names>EA</given-names></name><name><surname>Davachi</surname><given-names>L</given-names></name></person-group><year iso-8601-date="2016">2016</year><article-title>Episodic memories predict adaptive value-based decision-making</article-title><source>Journal of Experimental Psychology</source><volume>145</volume><fpage>548</fpage><lpage>558</lpage><pub-id pub-id-type="doi">10.1037/xge0000158</pub-id></element-citation></ref><ref id="bib66"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Nardou</surname><given-names>R</given-names></name><name><surname>Lewis</surname><given-names>EM</given-names></name><name><surname>Rothhaas</surname><given-names>R</given-names></name><name><surname>Xu</surname><given-names>R</given-names></name><name><surname>Yang</surname><given-names>A</given-names></name><name><surname>Boyden</surname><given-names>E</given-names></name><name><surname>Dölen</surname><given-names>G</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Oxytocin-dependent reopening of a social reward learning critical period with MDMA</article-title><source>Nature</source><volume>569</volume><fpage>116</fpage><lpage>120</lpage><pub-id pub-id-type="doi">10.1038/s41586-019-1075-9</pub-id><pub-id pub-id-type="pmid">30944474</pub-id></element-citation></ref><ref id="bib67"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Niv</surname><given-names>Y</given-names></name><name><surname>Edlund</surname><given-names>JA</given-names></name><name><surname>Dayan</surname><given-names>P</given-names></name><name><surname>O’Doherty</surname><given-names>JP</given-names></name></person-group><year iso-8601-date="2012">2012</year><article-title>Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain</article-title><source>The Journal of Neuroscience</source><volume>32</volume><fpage>551</fpage><lpage>562</lpage><pub-id pub-id-type="doi">10.1523/JNEUROSCI.5498-10.2012</pub-id><pub-id pub-id-type="pmid">22238090</pub-id></element-citation></ref><ref id="bib68"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Nussenbaum</surname><given-names>K</given-names></name><name><surname>Hartley</surname><given-names>CA</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Reinforcement learning across development: What insights can we draw from a decade of research?</article-title><source>Developmental Cognitive Neuroscience</source><volume>40</volume><elocation-id>100733</elocation-id><pub-id pub-id-type="doi">10.1016/j.dcn.2019.100733</pub-id><pub-id pub-id-type="pmid">31770715</pub-id></element-citation></ref><ref id="bib69"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Paus</surname><given-names>T</given-names></name><name><surname>Keshavan</surname><given-names>M</given-names></name><name><surname>Giedd</surname><given-names>JN</given-names></name></person-group><year iso-8601-date="2008">2008</year><article-title>Why do many psychiatric disorders emerge during adolescence?</article-title><source>Nature Reviews. Neuroscience</source><volume>9</volume><fpage>947</fpage><lpage>957</lpage><pub-id pub-id-type="doi">10.1038/nrn2513</pub-id><pub-id pub-id-type="pmid">19002191</pub-id></element-citation></ref><ref id="bib70"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Pearce</surname><given-names>JM</given-names></name><name><surname>Hall</surname><given-names>G</given-names></name></person-group><year iso-8601-date="1980">1980</year><article-title>A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli</article-title><source>Psychological Review</source><volume>87</volume><fpage>532</fpage><lpage>552</lpage><pub-id pub-id-type="pmid">7443916</pub-id></element-citation></ref><ref id="bib71"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Potter</surname><given-names>TCS</given-names></name><name><surname>Bryce</surname><given-names>NV</given-names></name><name><surname>Hartley</surname><given-names>CA</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Cognitive components underpinning the development of model-based learning</article-title><source>Developmental Cognitive Neuroscience</source><volume>25</volume><fpage>272</fpage><lpage>280</lpage><pub-id pub-id-type="doi">10.1016/j.dcn.2016.10.005</pub-id></element-citation></ref><ref id="bib72"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Pratt</surname><given-names>JW</given-names></name></person-group><year iso-8601-date="1964">1964</year><article-title>Risk Aversion in the Small and in the Large</article-title><source>Econometrica: Journal of the Econometric Society</source><volume>32</volume><elocation-id>122</elocation-id><pub-id pub-id-type="doi">10.2307/1913738</pub-id></element-citation></ref><ref id="bib73"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Pulcu</surname><given-names>E</given-names></name><name><surname>Browning</surname><given-names>M</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Affective bias as a rational response to the statistics of rewards and punishments</article-title><source>eLife</source><volume>6</volume><elocation-id>e27879</elocation-id><pub-id pub-id-type="doi">10.7554/eLife.27879</pub-id><pub-id pub-id-type="pmid">28976304</pub-id></element-citation></ref><ref id="bib74"><element-citation publication-type="software"><person-group person-group-type="author"><collab>R Development Core Team</collab></person-group><year iso-8601-date="2016">2016</year><data-title>R: A language and environment for statistical computing</data-title><version designator="2.6.2">2.6.2</version><publisher-loc>Vienna, Austria</publisher-loc><publisher-name>R Foundation for Statistical Computing</publisher-name><ext-link ext-link-type="uri" xlink:href="https://www.r-project.org">https://www.r-project.org</ext-link></element-citation></ref><ref id="bib75"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Radulescu</surname><given-names>A</given-names></name><name><surname>Niv</surname><given-names>Y</given-names></name><name><surname>Ballard</surname><given-names>I</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Holistic Reinforcement Learning: The Role of Structure and Attention</article-title><source>Trends in Cognitive Sciences</source><volume>23</volume><fpage>278</fpage><lpage>292</lpage><pub-id pub-id-type="doi">10.1016/j.tics.2019.01.010</pub-id><pub-id pub-id-type="pmid">30824227</pub-id></element-citation></ref><ref id="bib76"><element-citation publication-type="preprint"><person-group person-group-type="author"><name><surname>Radulescu</surname><given-names>A</given-names></name><name><surname>Holmes</surname><given-names>K</given-names></name><name><surname>Niv</surname><given-names>Y</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>On the Convergent Validity of Risk Sensitivity Measures</article-title><source>PsyArXiv</source><ext-link ext-link-type="uri" xlink:href="https://doi.org/10.31234/osf.io/qdhx4">https://doi.org/10.31234/osf.io/qdhx4</ext-link></element-citation></ref><ref id="bib77"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Raftery</surname><given-names>AE</given-names></name></person-group><year iso-8601-date="1995">1995</year><article-title>Bayesian Model Selection in Social Research</article-title><source>Sociological Methodology</source><volume>25</volume><elocation-id>111</elocation-id><pub-id pub-id-type="doi">10.2307/271063</pub-id></element-citation></ref><ref id="bib78"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rodman</surname><given-names>AM</given-names></name><name><surname>Powers</surname><given-names>KE</given-names></name><name><surname>Somerville</surname><given-names>LH</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Development of self-protective biases in response to social evaluative feedback</article-title><source>PNAS</source><volume>114</volume><fpage>13158</fpage><lpage>13163</lpage><pub-id pub-id-type="doi">10.1073/pnas.1712398114</pub-id><pub-id pub-id-type="pmid">29180428</pub-id></element-citation></ref><ref id="bib79"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rosenbaum</surname><given-names>GM</given-names></name><name><surname>Venkatraman</surname><given-names>V</given-names></name><name><surname>Steinberg</surname><given-names>L</given-names></name><name><surname>Chein</surname><given-names>JM</given-names></name></person-group><year iso-8601-date="2018">2018</year><article-title>The Influences of Described and Experienced Information on Adolescent Risky Decision Making</article-title><source>Developmental Review</source><volume>47</volume><fpage>23</fpage><lpage>43</lpage><pub-id pub-id-type="doi">10.1016/j.dr.2017.09.003</pub-id><pub-id pub-id-type="pmid">29527087</pub-id></element-citation></ref><ref id="bib80"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rosenbaum</surname><given-names>GM</given-names></name><name><surname>Hartley</surname><given-names>CA</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Developmental perspectives on risky and impulsive choice</article-title><source>Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences</source><volume>374</volume><elocation-id>20180133</elocation-id><pub-id pub-id-type="doi">10.1098/rstb.2018.0133</pub-id><pub-id pub-id-type="pmid">30966918</pub-id></element-citation></ref><ref id="bib81"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rosenbaum</surname><given-names>GM</given-names></name><name><surname>Venkatraman</surname><given-names>V</given-names></name><name><surname>Steinberg</surname><given-names>L</given-names></name><name><surname>Chein</surname><given-names>JM</given-names></name></person-group><year iso-8601-date="2021">2021</year><article-title>Do adolescents always take more risks than adults? A within-subjects developmental study of context effects on decision making and processing</article-title><source>PLOS ONE</source><volume>16</volume><elocation-id>e0255102</elocation-id><pub-id pub-id-type="doi">10.1371/journal.pone.0255102</pub-id><pub-id pub-id-type="pmid">34339433</pub-id></element-citation></ref><ref id="bib82"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rouhani</surname><given-names>N</given-names></name><name><surname>Norman</surname><given-names>KA</given-names></name><name><surname>Niv</surname><given-names>Y</given-names></name></person-group><year iso-8601-date="2018">2018</year><article-title>Dissociable effects of surprising rewards on learning and memory</article-title><source>Journal of Experimental Psychology. Learning, Memory, and Cognition</source><volume>44</volume><fpage>1430</fpage><lpage>1443</lpage><pub-id pub-id-type="doi">10.1037/xlm0000518</pub-id><pub-id pub-id-type="pmid">29553767</pub-id></element-citation></ref><ref id="bib83"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rouhani</surname><given-names>N</given-names></name><name><surname>Niv</surname><given-names>Y</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Depressive symptoms bias the prediction-error enhancement of memory towards negative events in reinforcement learning</article-title><source>Psychopharmacology</source><volume>236</volume><fpage>2425</fpage><lpage>2435</lpage><pub-id pub-id-type="doi">10.1007/s00213-019-05322-z</pub-id></element-citation></ref><ref id="bib84"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sara</surname><given-names>SJ</given-names></name></person-group><year iso-8601-date="2009">2009</year><article-title>The locus coeruleus and noradrenergic modulation of cognition</article-title><source>Nature Reviews. Neuroscience</source><volume>10</volume><fpage>211</fpage><lpage>223</lpage><pub-id pub-id-type="doi">10.1038/nrn2573</pub-id><pub-id pub-id-type="pmid">19190638</pub-id></element-citation></ref><ref id="bib85"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Satterwhite</surname><given-names>CL</given-names></name><name><surname>Torrone</surname><given-names>E</given-names></name><name><surname>Meites</surname><given-names>E</given-names></name><name><surname>Dunne</surname><given-names>EF</given-names></name><name><surname>Mahajan</surname><given-names>R</given-names></name><name><surname>Ocfemia</surname><given-names>MCB</given-names></name><name><surname>Su</surname><given-names>J</given-names></name><name><surname>Xu</surname><given-names>F</given-names></name><name><surname>Weinstock</surname><given-names>H</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>Sexually transmitted infections among US women and men: prevalence and incidence estimates, 2008</article-title><source>Sexually Transmitted Diseases</source><volume>40</volume><fpage>187</fpage><lpage>193</lpage><pub-id pub-id-type="doi">10.1097/OLQ.0b013e318286bb53</pub-id><pub-id pub-id-type="pmid">23403598</pub-id></element-citation></ref><ref id="bib86"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Schaie</surname><given-names>KW</given-names></name></person-group><year iso-8601-date="1965">1965</year><article-title>A general model for the study of developmental problems</article-title><source>Psychological Bulletin</source><volume>64</volume><fpage>92</fpage><lpage>107</lpage><pub-id pub-id-type="doi">10.1037/h0022371</pub-id></element-citation></ref><ref id="bib87"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Shadlen</surname><given-names>MN</given-names></name><name><surname>Shohamy</surname><given-names>D</given-names></name></person-group><year iso-8601-date="2016">2016</year><article-title>Decision Making and Sequential Sampling from Memory</article-title><source>Neuron</source><volume>90</volume><fpage>927</fpage><lpage>939</lpage><pub-id pub-id-type="doi">10.1016/j.neuron.2016.04.036</pub-id><pub-id pub-id-type="pmid">27253447</pub-id></element-citation></ref><ref id="bib88"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sharot</surname><given-names>T</given-names></name><name><surname>Garrett</surname><given-names>N</given-names></name></person-group><year iso-8601-date="2016">2016</year><article-title>Forming Beliefs: Why Valence Matters</article-title><source>Trends in Cognitive Sciences</source><volume>20</volume><fpage>25</fpage><lpage>33</lpage><pub-id pub-id-type="doi">10.1016/j.tics.2015.11.002</pub-id><pub-id pub-id-type="pmid">26704856</pub-id></element-citation></ref><ref id="bib89"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Silverman</surname><given-names>MH</given-names></name><name><surname>Jedd</surname><given-names>K</given-names></name><name><surname>Luciana</surname><given-names>M</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Neural networks involved in adolescent reward processing: An activation likelihood estimation meta-analysis of functional neuroimaging studies</article-title><source>NeuroImage</source><volume>122</volume><fpage>427</fpage><lpage>439</lpage><pub-id pub-id-type="doi">10.1016/j.neuroimage.2015.07.083</pub-id><pub-id pub-id-type="pmid">26254587</pub-id></element-citation></ref><ref id="bib90"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Simonsohn</surname><given-names>U</given-names></name></person-group><year iso-8601-date="2018">2018</year><article-title>Two Lines: A Valid Alternative to the Invalid Testing of U-Shaped Relationships With Quadratic Regressions</article-title><source>Advances in Methods and Practices in Psychological Science</source><volume>1</volume><fpage>538</fpage><lpage>555</lpage><pub-id pub-id-type="doi">10.1177/2515245918805755</pub-id></element-citation></ref><ref id="bib91"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Somerville</surname><given-names>LH</given-names></name><name><surname>Sasse</surname><given-names>SF</given-names></name><name><surname>Garrad</surname><given-names>MC</given-names></name><name><surname>Drysdale</surname><given-names>AT</given-names></name><name><surname>Abi Akar</surname><given-names>N</given-names></name><name><surname>Insel</surname><given-names>C</given-names></name><name><surname>Wilson</surname><given-names>RC</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Charting the expansion of strategic exploratory behavior during adolescence</article-title><source>Journal of Experimental Psychology. General</source><volume>146</volume><fpage>155</fpage><lpage>164</lpage><pub-id pub-id-type="doi">10.1037/xge0000250</pub-id><pub-id pub-id-type="pmid">27977227</pub-id></element-citation></ref><ref id="bib92"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Steinberg</surname><given-names>L</given-names></name></person-group><year iso-8601-date="2008">2008</year><article-title>A Social Neuroscience Perspective on Adolescent Risk-Taking</article-title><source>Developmental Review</source><volume>28</volume><fpage>78</fpage><lpage>106</lpage><pub-id pub-id-type="doi">10.1016/j.dr.2007.08.002</pub-id><pub-id pub-id-type="pmid">18509515</pub-id></element-citation></ref><ref id="bib93"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Steinberg</surname><given-names>L</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>The influence of neuroscience on US Supreme Court decisions about adolescents’ criminal culpability</article-title><source>Nature Reviews. Neuroscience</source><volume>14</volume><fpage>513</fpage><lpage>518</lpage><pub-id pub-id-type="doi">10.1038/nrn3509</pub-id><pub-id pub-id-type="pmid">23756633</pub-id></element-citation></ref><ref id="bib94"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sutton</surname><given-names>RS</given-names></name><name><surname>Barto</surname><given-names>AG</given-names></name></person-group><year iso-8601-date="1998">1998</year><article-title>Reinforcement Learning: An Introduction</article-title><source>IEEE Transactions on Neural Networks</source><volume>9</volume><elocation-id>1054</elocation-id><pub-id pub-id-type="doi">10.1109/TNN.1998.712192</pub-id></element-citation></ref><ref id="bib95"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Tversky</surname><given-names>A</given-names></name><name><surname>Kahneman</surname><given-names>D</given-names></name></person-group><year iso-8601-date="1992">1992</year><article-title>Advances in prospect theory: Cumulative representation of uncertainty</article-title><source>Journal of Risk and Uncertainty</source><volume>5</volume><fpage>297</fpage><lpage>323</lpage><pub-id pub-id-type="doi">10.1007/BF00122574</pub-id></element-citation></ref><ref id="bib96"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>van den Bos</surname><given-names>W</given-names></name><name><surname>Cohen</surname><given-names>MX</given-names></name><name><surname>Kahnt</surname><given-names>T</given-names></name><name><surname>Crone</surname><given-names>EA</given-names></name></person-group><year iso-8601-date="2012">2012</year><article-title>Striatum-medial prefrontal cortex connectivity predicts developmental changes in reinforcement learning</article-title><source>Cerebral Cortex</source><volume>22</volume><fpage>1247</fpage><lpage>1255</lpage><pub-id pub-id-type="doi">10.1093/cercor/bhr198</pub-id><pub-id pub-id-type="pmid">21817091</pub-id></element-citation></ref><ref id="bib97"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>van den Bos</surname><given-names>W</given-names></name><name><surname>Rodriguez</surname><given-names>CA</given-names></name><name><surname>Schweitzer</surname><given-names>JB</given-names></name><name><surname>McClure</surname><given-names>SM</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Adolescent impatience decreases with increased frontostriatal connectivity</article-title><source>PNAS</source><volume>112</volume><fpage>E3765</fpage><lpage>E3774</lpage><pub-id pub-id-type="doi">10.1073/pnas.1423095112</pub-id><pub-id pub-id-type="pmid">26100897</pub-id></element-citation></ref><ref id="bib98"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>van Duijvenvoorde</surname><given-names>ACK</given-names></name><name><surname>Peters</surname><given-names>S</given-names></name><name><surname>Braams</surname><given-names>BR</given-names></name><name><surname>Crone</surname><given-names>EA</given-names></name></person-group><year iso-8601-date="2016">2016</year><article-title>What motivates adolescents? Neural responses to rewards and their influence on adolescents’ risk taking, learning, and cognitive control</article-title><source>Neuroscience and Biobehavioral Reviews</source><volume>70</volume><fpage>135</fpage><lpage>147</lpage><pub-id pub-id-type="doi">10.1016/j.neubiorev.2016.06.037</pub-id><pub-id pub-id-type="pmid">27353570</pub-id></element-citation></ref><ref id="bib99"><element-citation publication-type="book"><person-group person-group-type="author"><name><surname>van Duijvenvoorde</surname><given-names>ACK</given-names></name><name><surname>Blankenstein</surname><given-names>NE</given-names></name><name><surname>Crone</surname><given-names>EA</given-names></name><name><surname>Figner</surname><given-names>B</given-names></name></person-group><year iso-8601-date="2017">2017</year><chapter-title>Towards a better understanding of adolescent risk taking: Contextual moderators and model-based analysis</chapter-title><person-group person-group-type="editor"><name><surname>Toplak</surname><given-names>ME</given-names></name><name><surname>Weller</surname><given-names>JA</given-names></name></person-group><source>Individual Differences in Judgment and Decision Making from a Developmental Perspective</source><publisher-name>Routledge</publisher-name><fpage>8</fpage><lpage>27</lpage></element-citation></ref><ref id="bib100"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Van Leijenhorst</surname><given-names>L</given-names></name><name><surname>Gunther Moor</surname><given-names>B</given-names></name><name><surname>Op de Macks</surname><given-names>ZA</given-names></name><name><surname>Rombouts</surname><given-names>SARB</given-names></name><name><surname>Westenberg</surname><given-names>PM</given-names></name><name><surname>Crone</surname><given-names>EA</given-names></name></person-group><year iso-8601-date="2010">2010</year><article-title>Adolescent risky decision-making: neurocognitive development of reward and control regions</article-title><source>NeuroImage</source><volume>51</volume><fpage>345</fpage><lpage>355</lpage><pub-id pub-id-type="doi">10.1016/j.neuroimage.2010.02.038</pub-id><pub-id pub-id-type="pmid">20188198</pub-id></element-citation></ref><ref id="bib101"><element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Wechsler</surname><given-names>D</given-names></name></person-group><year iso-8601-date="2011">2011</year><source>Wechsler Abbreviated Scale of Intelligence–Second Edition</source><publisher-loc>San Antonio</publisher-loc><publisher-name>NCS Pearson</publisher-name></element-citation></ref></ref-list><app-group><app id="appendix-1"><title>Appendix 1</title><sec id="s8" sec-type="appendix"><title>Experiment 1</title><sec id="s8-1" sec-type="appendix"><title>Learning and risk taking by age group and block</title><fig id="app1fig1" position="float"><label>Appendix 1—figure 1.</label><caption><title>Choices as a function of block and age group.</title><p>(<bold>A</bold>) Mean accuracy on test trials (approximately seven per participant per block). (<bold>B</bold>) Mean risk taking for equal expected value (EV) trials (seven per participant per block). (<bold>C</bold>) Mean risk taking for unequal-EV trials (four per participant per block).</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig1.jpg"/></fig></sec><sec id="s8-2" sec-type="appendix"><title>Unequal-EV choices of probabilistic machines</title><p>On trials where participants faced risky and safe choice options with unequal EV (i.e., the 0/80 point machine vs. the safe 20 point machine), age patterns were similar to those for equal-EV trials. Specifically, there was a significant quadratic age effect on probabilistic, risky decision-making (<italic>b</italic> = 0.09, 95% CI [0.01, 0.16], <italic>t</italic>(59) = 2.21, p=0.031, <italic>f</italic><sup>2</sup> = 0.08, 95% CI [0, 0.30]). The two-lines test showed that unequal-EV risk taking significantly decreased from ages 8–17 (<italic>b</italic> = 0.04, <italic>z</italic> = –2.77, p=0.006), and marginally increased from ages 17–27 (<italic>b</italic> = 0.03, <italic>z</italic> = 1.74, p=0.081).</p><fig id="app1fig2" position="float"><label>Appendix 1—figure 2.</label><caption><title>Probabilistic choices for unequal-expected value (EV) risk trials.</title><p>Probabilistic (i.e., risky) choices by age on trials with unequal-expected value (EV) risky and safe machines, with a choice between the 0/80 probabilistic machine and the deterministic 20-point machine. Data points depict the mean percentage of trials where each participant selected the probabilistic choice option as a function of age. Regression line is from the glmer model including linear and quadratic age terms. Shaded region represents 95% CIs for estimates.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig2.jpg"/></fig></sec><sec id="s8-3" sec-type="appendix"><title>Memory performance by age, PE valence, and asymmetry index</title><fig id="app1fig3" position="float"><label>Appendix 1—figure 3.</label><caption><title>Memory performance across age.</title><p>(<bold>A</bold>) False alarm rate as a function of age. As reported in the article, false alarm rate increased with age (p=0.037). (<bold>B</bold>) D’ as a function of age. As reported in the article, there is a marginal linear decrease in d’ with age (p=0.070).</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig3.jpg"/></fig></sec><sec id="s8-4" sec-type="appendix"><title>Multilevel model fitting</title><p>The maximal multilevel model did not converge when we tested for the three-way interaction between PE valence, PE magnitude, and AI predicting memory accuracy. We systematically reduced the model until we found a model that converged (<xref ref-type="bibr" rid="bib4">Barr et al., 2013</xref>). The maximal model that converged is:</p><p>Memory Response ~ Age_Z + Age_Z^2+ Memory Trial Number + False Alarm Rate+ AI * PE Valence * PE Magnitude + (1+ PE Magnitude + Memory Trial Number || SubjectNumber)</p></sec><sec id="s8-5" sec-type="appendix"><title>Ordinal modeling of memory data</title><p>Our multilevel models of memory data collapsed across confidence ratings (e.g., ‘Definitely old’ and ‘Maybe old’), a convention widely adopted in manuscripts examining memory accuracy effects (e.g., <xref ref-type="bibr" rid="bib33">Dunsmoor et al., 2015</xref>; <xref ref-type="bibr" rid="bib65">Murty et al., 2016</xref>). As an exploratory analysis, we ran an ordinal model using the clmm function in the ordinal R package (<xref ref-type="bibr" rid="bib19">Christensen, 2019</xref>), which allowed us to test for an AI × PE Valence × PE Magnitude interaction effect using participants’ uncollapsed memory responses as the dependent variable (1 = definitely new, 2 = maybe new, 3 = maybe old, and 4 = definitely old).</p><p>Regression results are reported in Appendix 1—table 1 (see <xref ref-type="supplementary-material" rid="supp1">Supplementary file 1</xref>). Importantly, the results from the ordinal regression were not meaningfully different from results collapsed across confidence ratings. In particular, the AI × PE Valence × PE Magnitude interaction was significant in the ordinal regression. The three-way interaction is plotted in <xref ref-type="fig" rid="app1fig4">Appendix 1—figure 4</xref>. Here, probabilities of each memory response are plotted as a function of PE valence and magnitude separately for AI = –0.8 (top panels), AI = 0 (middle panels), and AI = 0.8 (bottom panels). Consistent with the results reported in the main text, the likelihood of a ‘definitely old’ response was highest for those in those with low AI for images that coincided with high-magnitude negative PEs (top-left panel) and those with high AIs for images that coincided with high-magnitude positive PEs (bottom-right panel).</p><fig id="app1fig4" position="float"><label>Appendix 1—figure 4.</label><caption><title>Ordinal regression analysis of incidental memory judgments (Experiment 1).</title><p>Results from an ordinal regression demonstrating that incidental memory accuracy for pictures presented with choice outcomes varies as a function of PE valence, PE magnitude, and asymmetry index (AI) without collapsing across response confidence levels. The probability of each memory response is plotted separately for three different AI levels (top: AI = –0.8; middle: AI = 0; bottom: AI = 0.8) as a function of PE valence, PE magnitude.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig4.jpg"/></fig></sec><sec id="s8-6" sec-type="appendix"><title>BIC distributions</title><fig id="app1fig5" position="float"><label>Appendix 1—figure 5.</label><caption><title>BIC distributions for all four models tested.</title></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig5.jpg"/></fig></sec><sec id="s8-7" sec-type="appendix"><title>RSTD vs. TD model fit as a function of asymmetry index</title><p>The RSTD model fit the data of subjects with more extreme AI values substantially better than the TD model (<xref ref-type="fig" rid="app1fig6">Appendix 1—figure 6</xref>). The difference in model fit was smaller for those with AIs close to 0, reflecting the redundancy of a model including two learning rates when the learning rates are similar.</p><fig id="app1fig6" position="float"><label>Appendix 1—figure 6.</label><caption><title>Relative BIC as a function of asymmetry index (AI).</title><p>The difference between risk-sensitive temporal difference (RSTD) and temporal difference (TD) model fit (BIC) for all participants in Experiment 1. Values below 0 indicate a better fit by the RSTD model.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig6.jpg"/></fig></sec><sec id="s8-8" sec-type="appendix"><title>RSTD vs. TD reinforcement learning model recovery</title><p>In the main text methods, we described our model recovery analysis, where we simulated 10,000 ‘subjects’ using each model and fit the simulated data using the generating model, and all alternative models. <xref ref-type="fig" rid="app1fig7">Appendix 1—figure 7</xref> shows relative RSTD and TD model fit (BIC) model fit for subjects generated by the RSTD model, as a function of AI. For simulated subjects with a more extreme AI, the RSTD model provided a substantially better fit. For those with AIs closer to 0 (i.e., when <italic>α</italic><sup>+</sup> is similar to <italic>α</italic><sup>-</sup>), RSTD and TD models perform similarly.</p><fig id="app1fig7" position="float"><label>Appendix 1—figure 7.</label><caption><title>Relative BIC as a function of asymmetry index (AI) for participants simulated by the risk-sensitive temporal difference (RSTD) model.</title><p>The difference between risk-sensitive temporal difference (RSTD) and temporal difference (TD) model fit (BIC). The difference in model fit (BIC) between the risk-sensitive temporal difference (RSTD) and temporal difference (TD) models for 10,000 subjects simulated using the RSTD model. Values below 0 indicate a better fit by the RSTD model.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig7.jpg"/></fig></sec><sec id="s8-9" sec-type="appendix"><title>RSTD model recovery as a function of asymmetry index</title><p>We tested whether parameter recovery differed as a function of individual differences in the propensity to make deterministic choices. This question was of particular interest because those who tended to make deterministic choices were less likely to choose risky machines, and therefore were less likely to experience high-magnitude PEs (for a related discussion, see the ‘Utility and subsequent memory’ section below, along with <xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11F</xref>). To this end, we tested whether parameter recovery and model fit (BIC) varied as a function of AI. Importantly, AI is highly correlated with risk taking, so this analysis allowed us to test for potential differences in parameter recoverability for participants who more frequently chose the deterministic point machines (i.e., those with low AI). To this end, we divided the simulated participants into AI quartiles and examined parameter recovery and model fit in each AI quartile. We found that the parameters were reasonably well recovered at all levels of AI (<xref ref-type="fig" rid="app1fig8">Appendix 1—figure 8A–C</xref>).</p><fig id="app1fig8" position="float"><label>Appendix 1—figure 8.</label><caption><title>Parameter recovery at different levels of Asymmetry Index (AI).</title><p>Parameter recovery for simulated participants at low (AIs ranging from –0.94 to –.0374), medium-low (AIs ranging from –0.373 to –0.07684), medium-high (AIs ranging from –0.07683 to 0.2501), and high (AIs ranging from 0.2502 to 0.97) levels of AI. (<bold>A</bold>) <italic>α</italic><sup>+</sup> recovery. (<bold>B</bold>) <italic>α<sup>-</sup></italic> recovery. (<bold>C</bold>) <italic>β</italic> recovery. (<bold>D</bold>) BIC.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig8.jpg"/></fig><p>However, parameter recoverability varied across levels of AI, somewhat counterintuitively. In particular, recovery of the <italic>α</italic>+ parameter was relatively poorer for the simulated participants in the high-AI quartile (<xref ref-type="fig" rid="app1fig8">Appendix 1—figure 8A</xref>) and <italic>α</italic>- recoverability was relatively poorer for those in the low-AI quartile (<xref ref-type="fig" rid="app1fig8">Appendix 1—figure 8B</xref>). Taken together, these patterns suggest that learning rate parameters are relatively less well recovered for individuals with higher AIs (i.e., who made more risk-seeking choices).</p><p>This differential recoverability as a function of AI stems from the interactions between subjects’ risk preferences and the set of risky choice trials presented in our task. There were two types of risky trials in our task: equal-EV (0/40 vs. 20, or 0/80 vs. 40) and unequal-EV (0/80 vs. 20). This particular combination of equal- and unequal-EV risk trials led to differential resolution in the estimation of valenced learning rates as a function of AI. Positive learning rates for risk-averse participants could be estimated more accurately because those who were very risk averse (and thus had a much larger <italic>α</italic><sup>-</sup> than <italic>α</italic><sup>+</sup>) might choose both the safe 40-point option and the safe 20-point option over the 0/80 machine, whereas those who were less risk averse might prefer the safe 40 to the 0/80, but the 0/80 over the safe 20. In contrast, those with high positive AI are likely to choose the risky option on every equal and unequal-EV trial, so the ability to distinguish precisely between different levels of <italic>α</italic>+ for those with high AI is diminished. Our model fit results provide further evidence that this lower <italic>α</italic>+ recoverability in individuals with high AIs stems from this aspect of our task structure (<xref ref-type="fig" rid="app1fig8">Appendix 1—figure 8D</xref>). Despite the model’s imprecise <italic>α</italic><sup>+</sup> estimation for these high-AI subjects, the model was able to predict behavior well (i.e., BIC was low), likely because these participants are likely to take risks on all equal and unequal-EV risk trials, regardless of their precise <italic>α</italic><sup>+</sup> level.</p><p>This increase in parameter recovery for those participants with low compared to high AI runs counter to the notion that smaller PEs may drive worse recovery in these low-AI participants. Although it is true that PEs were smaller for those with relatively lower AIs (see <xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11F</xref>), our design required these low-AI participants to experience high-magnitude PEs from risky choices on some trials. Specifically, in our task, participants encountered forced trials, where participants were required to choose specific machines, which were sometimes risky, and test trials (where one option dominated the other), and the dominating option was sometimes risky. Thus, including these forced and test trials may have facilitated our ability to recover learning rates in those with low AI, while also providing opportunities for participants to sufficiently learn and demonstrate their knowledge of machine outcomes and probabilities.</p><p>Despite these differences in parameter recoverability at different levels of AI within this experimental paradigm, there are several reasons why we do not believe that these results are problematic for interpreting the current results. First, these simulations were generated by sampling uniformly from the range of learning rate and temperature parameters observed in our empirical sample. Thus, these simulated participants can take on AI levels that are not actually represented within our empirical sample. Indeed, 81% of fit AI values observed in our empirical sample fall within the lower three quartiles of AI values for this simulation, for which parameter recoverability was higher. Moreover, the recoverability estimates in this simulation dramatically overrepresent participants with low levels of decision noise relative to our empirical sample, which distorts these estimates of recoverability to be lower than what would actually be obtained for our empirical sample of participants (e.g., when we exclude any simulated participants with decision noise <2, which captures the vast majority of participants in our sample, recoverability of <italic>r</italic> values all increase by ~.05).</p></sec><sec id="s8-10" sec-type="appendix"><title>Posterior predictive check</title><p>We ran a posterior predictive check on both RSTD and Utility models. We simulated 100 subjects for every real participant’s empirically derived parameters, using both the Utility model and the RSTD model, and took the mean proportion of risks across all 100 simulated subjects for each model. The quadratic age pattern in (simulated) risk taking for equal EV trials was significant for both RSTD- and Utility-simulated data (RSTD: <xref ref-type="fig" rid="app1fig9">Appendix 1—figure 9B</xref><italic>, b</italic> = 0.05, 95% CI [0, 0.10], <italic>t</italic>(59) = 2.16, p=0.035, <italic>f</italic><sup>2</sup> = 0.08, 95% CI [0, 0.29]; Utility: <xref ref-type="fig" rid="app1fig9">Appendix 1—figure 9D</xref><italic>, b</italic> = 0.05, 95% CI [0.01, 0.10], <italic>t</italic>(59) = 2.29, p=0.026, <italic>f</italic><sup>2</sup> = 0.09, 95% CI [0, 0.31]). Additionally, choices derived from both RSTD and Utility model simulations using each participant’s best-fit parameter estimates were highly correlated with actual choices (RSTD: <italic>r</italic> = 0.92, 95% CI [0.87, 0.95], <italic>t</italic>(60) = 18.14, p<0.001; Utility: <italic>r</italic> = 0.89, 95% CI [0.83, 0.93], <italic>t</italic>(60) = 15.25, p<.001). However, the correlation between simulated and actual choices was significantly stronger for participants simulated using the RSTD model compared to the Utility model (<italic>t</italic>(61) = 2.58, p<0.012). Thus, it appears that the RSTD model provides a better qualitative fit to the choice data compared to the Utility model.</p><fig id="app1fig9" position="float"><label>Appendix 1—figure 9.</label><caption><title>Posterior prediction check results.</title><p>Relationship between choices of the risky (probabilistic) option in real versus simulated data for (<bold>A</bold>) the risk-sensitive temporal difference (RSTD) model and (<bold>C</bold>) the Utility model. Relationship between risky choices and the corresponding real participant age for (<bold>B</bold>) RSTD-simulated and (<bold>D</bold>) Utility-simulated participants.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig9.jpg"/></fig></sec><sec id="s8-11" sec-type="appendix"><title>Age patterns in RSTD parameters</title><p>As reported in the main text, the relation between AI and age appears to be driven by quadratic age patterns in α<sup>-</sup> (<italic>b</italic> = –0.09, 95% CI [–0.15, –0.03], <italic>t</italic>(59) = –3.01, p=0.004, <italic>f</italic><sup>2</sup> = 0.15, 95% CI [0.02, 0.43]; <xref ref-type="fig" rid="app1fig10">Appendix 1—figure 10B</xref>). The relation between linear age and α<sup>+</sup> was not significant (<italic>b</italic> = –0.02, 95% CI [–0.07, 0.03], <italic>t</italic>(60) = –0.85, p=0.401, <italic>f</italic><sup>2</sup> = 0.01, 95% CI [0, 0.13]; <xref ref-type="fig" rid="app1fig10">Appendix 1—figure 10A</xref>). Additionally, the relation between age and <italic>β</italic> was not significant (<italic>b</italic> = 0.56, 95% CI [–0.22, 1.33], <italic>t</italic>(60) = 1.44, p=0.156, <italic>f</italic><sup>2</sup> = 0.03, 95% CI [0, 0.19]; <xref ref-type="fig" rid="app1fig10">Appendix 1—figure 10C</xref>).</p><fig id="app1fig10" position="float"><label>Appendix 1—figure 10.</label><caption><title>Age patterns in risk-sensitive temporal difference (RSTD) model parameters.</title><p>Age-related change in (<bold>A</bold>) α<sup>+</sup>, (<bold>B</bold>) α<sup>-</sup>, and (<bold>C</bold>) <italic>β</italic> parameter estimates from the RSTD model.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig10.jpg"/></fig><fig id="app1fig11" position="float"><label>Appendix 1—figure 11.</label><caption><title>Relationship between parameter estimates and PEs derived from the risk-sensitive temporal difference (RSTD) and Utility models.</title><p>(<bold>A</bold>) Relationship between asymmetry index (AI) and Rho. (<bold>B</bold>) PEs for all participants, colored by the mean proportion of risk taking in the task. Purple dots are PEs from risk-seeking participants, while green dots are from risk-averse participants. (<bold>C</bold>) PEs for an example risk-seeking participant. (<bold>D</bold>) PEs for an example risk-averse participant. (<bold>E</bold>) Mean PE magnitudes, z-scored within the full sample, and then averaged within-subject, ordered by increasing Rho. (<bold>F</bold>) Mean PE magnitudes, z-scored within the full sample and then averaged within-subject, ordered by increasing AI.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig11.jpg"/></fig></sec><sec id="s8-12" sec-type="appendix"><title>Utility and subsequent memory</title><p>We examined whether PEs and choice biases indexed by the Utility model could explain the memory data similar to the observed relation between subsequent memory and PEs and valence biases derived from the RSTD model reported in the main text. To this end, we first assessed the relationship between RSTD and Utility model parameters. We found that AI, derived from the RSTD parameter estimates, and <italic>ρ</italic> from the Utility model were highly correlated (<italic>r</italic> = 0.95, p<0.001; <xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11</xref>), suggesting that they provided comparable individual difference measures.</p><p>However, due to the nonlinear transformation of outcome values within the Utility model, the two models yield very different value estimates and PEs. PE magnitudes derived from the Utility model also vary widely across participants. This pattern is clear in <xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11B</xref>, in which we have plotted the PEs across all participants from both the RSTD and Utility models. The dot color represents proportion risk taking. The purple dots are PEs from risk-seeking participants, while green dots are from risk-averse participants, with lighter dots representing people who took risks on about half of trials and darker dots representing those whose risk taking deviated more from equal numbers of risky and safe choices. Because of the nonlinear Utility function, Utility PE magnitudes are much higher for risk-seeking participants (for whom <italic>ρ</italic> > 1) versus risk-averse participants (for whom <italic>ρ</italic> < 1). In contrast, PE magnitudes from the linear RSTD model are necessarily constrained between 0 and 80 and are thus more uniformly distributed across participants regardless of risk preference. <xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11</xref> displays the relationship between PEs from Utility and RSTD models from example risk-seeking (<xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11C</xref>) and risk-averse (<xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11D</xref>) participants. The participant represented in <xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11C</xref> chose the risky option on 60% of trials in which the EVs of risky and safe options were equal. This participant’s <italic>ρ</italic> estimate from the Utility model was 2.5, and the RSTD AI was 0.56. In contrast, the participant in <xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11D</xref> took risks on 24% of equal-EV risk trials and had a corresponding <italic>ρ</italic> of 0.56 and AI of –0.45. The PEs from the two models are quite different in both absolute and relative scale. For the risk-seeking participant, the nonlinear transformation of outcome values means that Utility PEs reached magnitudes over 50,000, while RSTD PEs could only reach a maximum magnitude of 80.</p><p>We next tested whether the <italic>ρ</italic> parameter and PEs derived from the Utility model could account for subsequent memory in the same manner that we observed for the RSTD model. As <italic>ρ</italic> and AI are so highly correlated, any difference in explanatory ability of the two models would necessarily stem from differences in the ability of their distinct PE dynamics to capture trial-by-trial variability in subsequent memory. In contrast to the RSTD results, we did not find a significant three-way interaction between <italic>ρ</italic>, PE magnitude, and PE valence (using the PEs derived from the Utility model; <italic>z =</italic> 1.06, p=0.291, OR = 1.18, 95% CI [0.87, 1.60]).</p><p>This absence of an effect likely stems from the large variability in PE magnitude across participants within the Utility model (<xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11E</xref>) than the RSTD model (<xref ref-type="fig" rid="app1fig11">Appendix 1—figure 11F</xref>). Under the Utility model, risk-seeking participants (high <italic>ρ</italic>) experience much larger magnitude PEs, while PEs for risk-averse participants are smaller in magnitude. Thus, within the multilevel model, high <italic>ρ</italic> participants have not only greater means but also greater variance in their PEs, leading to an inherent prediction that risk-seeking participants should experience the strongest PE-dependent memory modulation. However, in our prior analysis using the RSTD model, valence biases in the effects of PE sign and magnitude on subsequent memory were evident in both risk-seeking and risk-averse participants. This suggests that only the linear values within the RSTD model can adequately capture this pattern.</p></sec><sec id="s8-13" sec-type="appendix"><title>The effect of agency on learning</title><p>Prior studies have demonstrated that learning asymmetries can vary as a function of whether choices are free or forced, such that participants tend to exhibit a greater positive learning rate bias (<italic>α<sup>+</sup></italic> > <italic>α<sup>-</sup></italic>) for free choices and either no bias or a negative bias (<italic>α<sup>+</sup></italic>< <italic>α<sup>-</sup></italic>) in forced choices (<xref ref-type="bibr" rid="bib17">Chambon et al., 2020</xref>; <xref ref-type="bibr" rid="bib21">Cockburn et al., 2014</xref>). To understand whether our participants exhibited different learning biases in free and forced trials, we implemented an additional reinforcement learning model with four learning rates (FourLR). Here, we used separate α<sup>+</sup> and α<sup>-</sup> for forced and free choices. Although this least-parsimonious model provided a better fit to the choice data than the TD model, it performed worse than both the Utility and RSTD models at both the group and participant level.</p><p>Free- and forced-choice learning rates and asymmetry indices are plotted in <xref ref-type="fig" rid="app1fig12">Appendix 1—figure 12</xref>. Consistent with prior findings (<xref ref-type="bibr" rid="bib17">Chambon et al., 2020</xref>; <xref ref-type="bibr" rid="bib21">Cockburn et al., 2014</xref>), we found that <italic>α<sup>+</sup></italic> was significantly higher in free (median = 0.22) compared to forced (median = 0.14) choices (Wilcoxon signed-rank test: <italic>V</italic> = 1338, p=0.011). In contrast, <italic>α<sup>-</sup></italic> was not significantly different in free (median = 0.35) vs. forced (median = 0.32) choices (Wilcoxon signed-rank test: <italic>V</italic> = 794, p=0.202). We computed separate AIs for free and forced choices, and found that AI was significantly higher for free (median = –0.12) compared to forced (median = –0.39) choices (Wilcoxon signed-rank test: <italic>V</italic> = 1377, p=0.005). Although <italic>α<sup>+</sup></italic> and AI were higher in free versus forced trials, median AIs were negative for both free and forced choices. Therefore, in this study, we did not observe positive learning rate asymmetries for free choices.</p><fig id="app1fig12" position="float"><label>Appendix 1—figure 12.</label><caption><title>Learning parameters for free and forced choices.</title><p>(<bold>A</bold>) Negative learning rates, (<bold>B</bold>) positive learning rates, and (<bold>C</bold>) asymmetry indices from the FourLR model are plotted as a function of choice type (free or forced).</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig12.jpg"/></fig></sec><sec id="s8-14" sec-type="appendix"><title>The effect of agency on memory</title><p>Prior research has also found better subsequent memory for memoranda associated with free versus forced choices (<xref ref-type="bibr" rid="bib49">Katzman and Hartley, 2020</xref>; <xref ref-type="bibr" rid="bib64">Murty et al., 2015</xref>). The present study was not designed to test the effects of agency on memory. The purpose of our forced-choice trials was to ensure all participants had similar opportunities to learn probabilities and outcomes associated with each probabilistic and deterministic point machine, regardless of their risk preferences. However, to test whether memory was different for items that appeared with outcomes of free versus forced choices, we ran two additional glmer models. First, we ran the most complex model included in the original manuscript (the model predicting memory accuracy that tested for a PE valence × PE magnitude × AI interaction, and also included linear and quadratic age and memory trial number), adding a predictor that indicating whether the image appeared after a free or forced choice. Memory did not vary as a function of choice agency (<italic>z</italic> = –0.44, p=0.664, OR = 0.99, 95% CI: [0.95,1.04]).</p><p>As a final test for a potential agency benefit on memory, we tested whether the three-way PE Valence × PE Magnitude × AI interaction predicting memory performance varied as a function of whether the choice was forced or free (i.e., we tested for a four-way PE Valence × PE Magnitude × AI × choice agency interaction). The four-way interaction was also not significant (p=0.136). To further explore whether agency had any apparent qualitative effect, we plotted the (three-way interaction effect – PE Valence × PE Magnitude × AI) separately for free versus forced choice trials (<xref ref-type="fig" rid="app1fig13">Appendix 1—figure 13</xref>). Strikingly, we found that for free-choice trials (<xref ref-type="fig" rid="app1fig13">Appendix 1—figure 13A</xref>), which comprised the majority of trials, the interaction looks qualitatively similar to the overall three-way interaction effect across all trials that we report in the main text (<xref ref-type="fig" rid="fig4">Figure 4B</xref>, shown in <xref ref-type="fig" rid="app1fig13">Appendix 1—figure 13C</xref>). However, for forced trials (<xref ref-type="fig" rid="app1fig13">Appendix 1—figure 13B</xref>), memory performance for images that coincided with positive PEs did not appear to be modulated by AI. This pattern is similar to the memory effect observed in Experiment 2 (<xref ref-type="fig" rid="fig6">Figure 6B</xref>, shown in <xref ref-type="fig" rid="app1fig13">Appendix 1—figure 13D</xref>), in which participants provided explicit predictions of outcomes, but did not make choices. This qualitative similarity suggests that memory for surprising high-magnitude positive outcomes may be equivalently facilitated for all individuals in situations where they do <italic>not</italic> have agency, regardless of their idiosyncratic biases in learning.</p><fig id="app1fig13" position="float"><label>Appendix 1—figure 13.</label><caption><title>Learning biases and subsequent memory as a function of agency.</title><p>(<bold>A</bold>) PE Valence × PE Magnitude × AI for free choices. (<bold>B</bold>) PE Valence × PE Magnitude × AI for forced trials. (<bold>C</bold>) PE Valence × PE Magnitude × AI for Experiment 1 (<xref ref-type="fig" rid="fig4">Figure 4B</xref>). (<bold>D</bold>) PE Valence × PE Magnitude × AI for Experiment 2 (<xref ref-type="fig" rid="fig6">Figure 6B</xref>). Note that the interaction effect for forced choices (<bold>B</bold>) resembles that in Experiment 2 (<bold>D</bold>) where participants were not asked to make choices.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig13.jpg"/></fig></sec></sec><sec id="s9" sec-type="appendix"><title>Experiment 2</title><p>To test for generalizability of our finding that learning biases affect what information is encoded in subsequent memory, we ran a modified version of our analysis from Experiment 1 on a previously published dataset (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>). There were 383 adult participants in the dataset, each of whom completed one of the three experiments reported in the original manuscript.</p><sec id="s9-2" sec-type="appendix"><title>Pavlovian learning and memory task</title><p>In this study, participants completed a Pavlovian learning and memory task where they learned, through trial and error, about the average value of indoor and outdoor scene images. On each trial, participants viewed a scene image and provided an estimate for the average value of that image category, after which they were shown the true value of that image. After completing learning trials, participants completed a memory test for half of the scenes presented during learning and an equal number of new images.</p><p>There were three different versions of the task that varied in the number of trials and the variability of the outcome distributions. Each participant completed only one of the three versions. As the differences between the versions of the task were minimal, we modeled the data of participants of all three task variants as a single sample. A detailed explanation of the methods can be found in the original manuscript (<xref ref-type="bibr" rid="bib82">Rouhani et al., 2018</xref>).</p></sec><sec id="s9-3" sec-type="appendix"><title>Reinforcement learning model</title><p>We fit an Explicit Prediction temporal difference reinforcement learning model to the estimation data. This model was similar to our RSTD model from Experiment 1, in that it included separate learning rates for trials with positive (<italic>α</italic><sup>+</sup>) and negative (<italic>α</italic><sup>-</sup>) prediction errors with priors (Beta(1.2,1.2)) on both parameters. In this model, the value of each image category (<italic>i</italic>) on each trial (<italic>t</italic>) was estimated as <italic>V<sub>i</sub></italic>(<italic>t</italic> + 1)=<italic>V<sub>i</sub></italic>(<italic>t</italic>) + <italic>α</italic><sup>+</sup> * δ(<italic>t</italic>) when <italic>δ</italic> > 1 and <italic>V<sub>i</sub></italic>(<italic>t</italic> + 1)=<italic>V<sub>i</sub></italic>(<italic>t</italic>) + <italic>α</italic><sup>-</sup> * δ(<italic>t</italic>) when <italic>δ</italic> < 1. Each participant’s AI was calculated as in Experiment 1: AI = (<italic>α</italic><sup>+</sup> - <italic>α<sup>-</sup></italic>)/(<italic>α</italic><sup>+</sup> + <italic>α</italic><sup>-</sup>).</p></sec><sec id="s9-4" sec-type="appendix"><title>Model fitting</title><p>Participants’ estimates and trial outcomes were rescaled between 0 and 1 prior to model fitting. Image category values were initialized at 0.5. We regressed model image value estimates on participants’ actual estimates and minimized the negative log likelihood of that regression result for each participant using the optimization function fminunc in MATLAB.</p></sec><sec id="s9-5" sec-type="appendix"><title>Assessment of model fit and exclusions for poor model fit</title><p>We computed estimated value on each trial using the best-fitting parameter. We computed the correlation between the model’s value estimates and actual value estimates from participants’ task experience as a metric of the quality of fit of the model to participants’ data. The mean correlation was 0.42 (SD = 0.34). For 78 participants, there was either a negative correlation between model-derived and actual value estimates or the negative log likelihood of their data using the best-fit parameters was positive, or both, indicating that the model did not fit the data well. Because our analysis relies on trial-by-trial PEs estimated using best-fit model parameters, we removed participants from the dataset who were poorly fit by the model according to either model fit metric. However, we conducted a sensitivity analysis including all participants, below, to ensure that these exclusions did not drive any observed effects in the restricted sample. The mean correlation between estimated and actual PEs for the remaining 305 participants was 0.54 (SD = 0.22).</p></sec><sec id="s9-6" sec-type="appendix"><title>Multilevel model fitting</title><p>The maximal model did not converge. As in Experiment 1, we reduced the model until we found a model that converged (<xref ref-type="bibr" rid="bib4">Barr et al., 2013</xref>). The maximal model that converged is:</p><p>Memory Response~ Memory Trial Number + False Alarm Rate + AI * PE Valence * PE Magnitude + (1+ PE Valence + PE Magnitude + Memory Trial Number || SubjectNumber)</p><p>Below, we also report a sensitivity analysis that includes all participants, regardless of model fit. The model that converged when all participants were included is:</p><p>Memory Response~ Memory Trial Number + False Alarm Rate + AI * PE Valence * PE Magnitude + (1+ PE Magnitude + Memory Trial Number || SubjectNumber)</p></sec></sec><sec id="s10" sec-type="appendix"><title>Model parameters and asymmetry index</title><p>Distributions of learning-rate parameters from our model, as well as the distribution of our AI metric, are plotted in <xref ref-type="fig" rid="app1fig14">Appendix 1—figure 14</xref>. These distributions include all participants who were included in the analysis in the main text.</p><fig id="app1fig14" position="float"><label>Appendix 1—figure 14.</label><caption><title>Distributions of parameters from the Explicit Prediction model.</title><p>Distributions for (<bold>A</bold>) α<sup>+</sup>, (<bold>B</bold>) α<sup>-</sup>, and (<bold>C</bold>) asymmetry index (AI) in Experiment 2.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig14.jpg"/></fig></sec><sec id="s11" sec-type="appendix"><title>Sensitivity analysis with all participants</title><p>To determine whether the observed relation between AI and memory was influenced by participant exclusions, we ran a sensitivity analysis fitting the multilevel model that included all 383 participants, including those poorly fit by our RL model (<xref ref-type="fig" rid="app1fig15">Appendix 1—figure 15</xref>). The three-way interaction between AI, PE valence, and PE magnitude predicting memory accuracy was marginally significant (<italic>z</italic> = 1.80, p=0.072, OR = 1.05, 95% CI [1.00, 1.11]; <xref ref-type="fig" rid="app1fig15">Appendix 1—figure 15B</xref>). In this model, there was also a significant two-way interaction between AI and PE valence (<italic>z</italic> = 2.83, p=0.005, OR = 1.08, 95% CI [1.02, 1.14]). Both the three-way and two-way interaction effects are consistent with our hypothesis that individual’s valence biases during learning (i.e., AI) modulate the relationship between PE valence and memory: those with negative learning biases had better memory for images that coincided with negative PEs and vice versa (<xref ref-type="fig" rid="app1fig15">Appendix 1—figure 15C</xref>). As in both Experiment 1 and in the analysis reported in the main text, there was a highly significant main effect of PE magnitude (<italic>z</italic> = 5.30, p<0.001, OR = 1.18, 95% CI [1.11, 1.26]).</p><fig id="app1fig15" position="float"><label>Appendix 1—figure 15.</label><caption><title>Experiment 2 sensitivity analysis.</title><p>Generalized linear mixed-effects regression results demonstrating incidental memory accuracy for pictures presented during learning as a function of PE valence, PE magnitude, including participants poorly fit by the RL model. (<bold>A</bold>) Fixed-effects results. Whiskers represent 95% CI. (<bold>B</bold>) Estimated marginal means plot showing the marginally significant three-way interaction between asymmetry index (AI), PE valence, and PE magnitude. Shaded areas represent 95% CI for estimates. (<bold>C</bold>) Estimated marginal means for significant two-way interaction between AI and PE valence. Whiskers represent 95% CI. **p < .01, ***p < .001.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/app1-fig15.jpg"/></fig></sec></app></app-group></back><sub-article article-type="editor-report" id="sa0"><front-stub><article-id pub-id-type="doi">10.7554/eLife.64620.sa0</article-id><title-group><article-title>Editor's evaluation</article-title></title-group><contrib-group><contrib contrib-type="author"><name><surname>Schlichting</surname><given-names>Margaret L</given-names></name><role specific-use="editor">Reviewing Editor</role><aff><institution>University of Toronto</institution><country>Canada</country></aff></contrib></contrib-group></front-stub><body><p>This paper will be of interest to cognitive and behavioral neuroscientists, behavioral economists, and developmental psychologists. The authors provide novel evidence that adolescents, relative to children and young adults, are prone to making risk-averse decisions because they are more attuned to negative outcomes during learning. The paper presents rigorous computational analyses that conclusively support the major claims and advance our understanding of age-related shifts in decision making.</p></body></sub-article><sub-article article-type="decision-letter" id="sa1"><front-stub><article-id pub-id-type="doi">10.7554/eLife.64620.sa1</article-id><title-group><article-title>Decision letter</article-title></title-group><contrib-group content-type="section"><contrib contrib-type="editor"><name><surname>Schlichting</surname><given-names>Margaret L</given-names></name><role>Reviewing Editor</role><aff><institution>University of Toronto</institution><country>Canada</country></aff></contrib></contrib-group></front-stub><body><boxed-text id="box1"><p>In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.</p></boxed-text><p><bold>Decision letter after peer review:</bold></p><p>Thank you for submitting your article "Valence biases in reinforcement learning shift across adolescence and modulate subsequent memory" for consideration by <italic>eLife</italic>. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by Michael Frank as the Senior Editor. The reviewers have opted to remain anonymous.</p><p>The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.</p><p>Essential revisions:</p><p>1. All three Reviewers had concerns about the modelling results. Many of these concerns stemmed from the lack of consideration of competing accounts for the data and/or limited justification provided for the choice of models. Please (a) clarify the use of the Pavlovian reinforcement learning model in Experiment 2 (Reviewer 1), (b) relate the current work to the utility model originally presented in the Niv 2012 paper which introduced this behavioural paradigm (Reviewer 3), and (c) add fits of additional competing models. Specifically, with regards to point (c), fitting the subjective value model to account for prospect theory, and the subjective utility model, would be informative.</p><p>2. Please account for the effect of forced vs. choice trials in both reinforcement learning and memory. All Reviewers highlighted this as an important consideration, as there is evidence that this distinction could lead to differential learning, attention, and/or memory (as shown previously by this group, e.g., Katzman & Hartley 2020).</p><p>3. Please include additional data visualizations that depict the raw data, which will better equip the reader for interpreting the results (e.g., the learning curves per experimental condition, and/or per age group as suggested by Reviewer 2).</p><p>4. Please consider the role of attention versus reward prediction error (RPE) in the memory effects. Reviewers pointed out that there may be attentional differences across probabilistic versus deterministic trials that could be driving the effects rather than RPE, which if the source of memory differences would warrant a different conclusion than the current paper. In addition, whether and how attention fits into the relationship between asymmetry index and memory was unclear. The authors may be in a position to address this question by performing additional analyses on the current data (e.g., by assessing whether RPE strength alone is related to encoding within the probabilistic trials, as suggested by Reviewer 2); or more generally (in the absence of data) clarify their position or speculate on these issues.</p><p>5. Please improve the integration of Experiment 2 into the paper. More details were needed for understanding the purpose of this sample. In addition, Reviewers noted that the interaction found in Experiment 1 did not entirely replicate in Experiment 2. This warrants more consideration in the paper.</p><p>6. Please clarify the task and methods descriptions. For example, it is not clear whether the forced choices were included in the modelling. Moreover, please ensure all details needed to understand the major points of the paper are included in the main text without requiring readers to reference the methods.</p><p>7. As highlighted by Reviewer 3, it is unclear whether asymmetry biases are a trait-stable characteristic, or rather would change with age within an individual. Please speak to this point as well as other limitations associated with the cross-sectional nature of this study in the revised Discussion.</p><p>8. The Reviewers had several additional thoughtful suggestions for other work the authors might cite. I recommend the authors consider whether these suggestions might enhance the novelty, impact, and/or clarity of their paper.</p><p>9. Please provide additional details on the models (e.g., reporting parameter β, age effects, model validation, and/or more information on the recoverability analysis) as appropriate. One specific question raised by reviewers was whether learning rate parameter estimation might be more difficult in participants with more deterministic choices and thus fewer trials with non-zero RPEs. Is this empirically true, i.e., does model parameter recovery (as shown in Table 1 overall) differ across ages/ranges of behavior?</p><p><italic>Reviewer #1:</italic></p><p>Rosenbaum et al. report an investigation of developmental differences in biases to learn from positive versus negative prediction errors-that is, experiences that are either better (positive) or worse (negative) than expected. One of their key findings is that adolescents show a bias towards negative outcomes in both reinforcement learning and memory, a finding that is somewhat surprising given teens' heightened sensitivity to reward and real-world risk-taking. That is, adolescents (1) showed a general bias to make riskier choices (selecting probabilistic rather than deterministic "machines" of equal average value) and (2) show greater learning from negative than positive prediction errors (i.e., showed a negative asymmetry index [AI]). In addition, individual differences in AI were mirrored by a similar memory advantage, where participants showing greater positive AI likewise showed a bias to remember more surprisingly positive stimuli; whereas those showing a greater negative AI were biased to remember more surprisingly negative stimuli. These individual differences were unrelated to age.</p><p>This is a strong paper with an interesting set of results. The authors additionally partially replicated their individual difference findings in a separate dataset (experiment 2; including only adults but a larger sample than the main experiment 1), which is a notable strength and somewhat increases confidence in the findings. I am particularly struck by the surprising finding that adolescents are biased towards negative rather than positive outcomes. As appropriately noted in the paper-and particularly given the adolescents did self-report a relatively higher amount of real-world risk taking than other ages, suggesting that it is not the case that the teens in the study are atypically risk-averse in their everyday lives-the degree to which the reinforcement learning task accurately captures something akin to everyday risk taking warrants further consideration in the field.</p><p>In addition to these strengths, the paper has several weaknesses as follows:</p><p>1. I am left unsure as to the value added by the reinforcement learning model and whether the asymmetry index reflects something unique about learning as compared with a general bias towards deterministic choices in the adolescent period. That is, if the authors wish to make claims about asymmetry of learning rate specifically (α) it would be important to know if this is dissociable from the choice or sampling bias they observe. Empirically, is it the case that there would be a similar set of findings if one focused on the percent probabilistic choices value per participant (data in Figure 2) rather than the asymmetry index (Figure 3)? If these are somewhat redundant or overlapping metrics, I would encourage the authors to clarify this in the paper and perhaps only focus on one of them in the main text, so as not to imply these are separate findings.</p><p>2. Related to my above point, those individuals who make fewer probabilistic choices (disproportionately adolescents) have fewer opportunities to learn from prediction errors that are either positive or negative. That is, in my understanding, there will be no prediction error from the deterministic machines which the adolescents primarily select. Their bias to select deterministic machines seems fairly extreme; it appears as though they shy away from the probabilistic choices across the board, even when the value of the probabilistic choice on average would greatly exceed that of the deterministic choice (e.g., as shown in Figure S2). As such, I am wondering whether theoretically the authors can clarify the relationship between choice and learning rate; and analytically whether the bias towards deterministic choices could impact the model fitting procedures.</p><p>3. As an additional interpretive question, I had trouble linking the asymmetry index to the memory performance and understanding how the authors believed these metrics to be related. Is there a general increase in interest or attention or salience when a prediction error aligning with the learner's biases (e.g., a highly positive prediction error if I'm someone who learns best from those types of trials – that is, showing a positive AI) happens, therefore indirectly impacting memory formation in the process? Or, is this thought to be a separate mechanism that occurs (the learning from positive prediction errors vs. "prioritization" in memory of those positively valenced experiences)? It seems to me as though both are related to learning/encoding, and thus this individual difference may not be terribly surprising, but I was unsure about the authors' preferred interpretation.</p><p>4. While I appreciated the inclusion of experiment 2, I felt that it was not particularly well integrated into the paper. For example, why did the authors opt to use data with only adults rather than a developmental sample (and does this constrain interpretation in any way)? In addition, it is important to highlight that the results do not fully replicate the main findings. In particular, there is no difference in the relationship between the magnitude of prediction error and memory among positively valenced trials according to AI, which was observed in the main developmental experiment 1. This discrepancy warrants more attention in the paper. (Could this be about the sample characteristics? Task differences? and so on.)</p><p>5. It was not clear at the conceptual level how the Pavlovian reinforcement learning model fit in experiment 2 is different from the main RSTD used in experiment 1, and/or why these paradigms required the use of different models. Additional description on this point would be helpful.</p><p>6. I would appreciate more context for the recoverability results. In particular, the ability to recover which model generated the data for simulated participants (RSTD vs. TD) seemed fairly low. I understand the point about low-asymmetry participants generated by the RSTD model being more parsimoniously fit by the simpler TD model, so that direction of error does not bother me so much. However, I am puzzled by the 87% correct for those simulated participants generated by the TD model. This would have to mean, I think, that the simple TD behavior was being incorrectly attributed to the more complex two-alpha model? I was hoping the authors could provide additional context or reassurance that this qualifies as "good" performance.</p><p>7. There were some details not sufficiently described in the main paper for the reader to understand without referencing the methods or supplement. For example: How did the forced versus choice trials work? What is "test trial performance" – the test trials are not described until the supplement and it was confusing to me because the only "test" mentioned in the main paper at this point was the memory test. How were the probabilistic vs. deterministic choices incorporated into the mixed-effects modeling?</p><p>8. How were the different responses on the memory test considered? There were four choices according to the figure but it is described as though it is simply "old" vs. "new" responses. Please clarify how this was handled in analysis as well as the reason for inclusion of these four options.</p><p>9. I apologized if I missed this point in the paper: I understand that the models were fit to individual data and most participants were better fit by RSTD than TD. However, the authors also discuss the RSTD model winning overall and all subsequent analyses on the individual differences require the two separate α values. So, I am confused as to whether (a) all participants were ultimately fit with the RSTD so all could be included in the analysis, despite some being better fit by TD or (b) only participants who were best fit by RSTD were included in subsequent analyses (or of course (c) something else)? I think this would be OK because my assumption would be that the participants would simply have two α values (positive and negative) that would be similar if their behavior is better explained by the TD model, but I just wanted to clarify.</p><p><italic>Reviewer #2:</italic></p><p>The authors investigate valence biases in reinforcement learning (RL) and memory, and how they change across adolescence. In a medium size developmental study where outcomes are learned rather than instructed (n = 62, ages 8-27), the results show a surprising choice pattern: adolescents have a stronger bias to select a choice with a deterministic 40 outcome, over a probabilistic 80/0 (50%), compared to younger and older participants. The authors interpret this as a more risk-averse performance, and operationalize it as a difference in RL learning rate. Memory test phase results show that individual images observed with a high absolute reward prediction error (RPE) are more likely to be recalled, and that individuals' valence biases predict a valence asymmetry of this RPE effect. The latter result is partially replicated in a reanalysis of a previously published, non-developmental study.</p><p>This is a well-written, easy to read article, that adds a new data point to the complex and contradictory literature on valence, risk, and adolescence, without resolving it. The strengths of the article are (1) its interesting experimental protocol and pure behavioral results (2) the existence of a replication of the complex memory finding.</p><p>There are also a number of weaknesses.</p><p>First the behavioral results are presented in a very processed manner, limiting the ability to re-interpret the results.</p><p>Second, the computational modeling is fairly limited and lacking in competing accounts, which also limits the interpretation of the results. As such, it seems that at this point, the results do not fully support the conclusions – in particular, it is not clear that the interpretation in terms of asymmetric learning rates is warranted yet.</p><p>Comments for the authors:</p><p>1. Presentation of the results. All figures presented are quite removed from the raw data. It would be helpful to provide more unprocessed results – for example, the learning curves per experimental condition, and/or per age group. This would also provide a more sensitive target for model validation than the ones currently presented in Figure S4. It is much harder for the reader to interpret the results when only very processed data is shown.</p><p>This is a well-written, easy to read article, that adds a new data point to the complex and contradictory literature on valence, risk, and adolescence, without resolving it. I see a number of issues with the presentation of the results, the modeling and interpretation of the results.</p><p>2. Modeling. The authors use two very simple RL models to capture the performance (a classic δ rule model, and the same with two learning rates). There are a few relevant aspects to the modeling that are either not considered or not reported, but that are important for interpreting the results.</p><p>a. Please indicate Q-value initialization and reward rescaling as part of the model description, rather than as part of the model fitting procedure. The initialization has theoretical impacts, so should be part of the model.</p><p>b. Prospect theory indicates that a value of 80 is subjectively worth less that 2* the subjective value of 40. As such, the claim that "expected value" and "risk" are controlled for is debatable: if a true 40 feels like a 50 in comparison to an 80, then the two "same objective expected value stimuli" have different subjective expected values, without a need to invoke risk. The authors should test a competing model where the learning rate is fixed, but the obtained reward is modified according to prospect theory (i.e. subjective reward = 80*((reward/80)^p), where p is a free parameter). This model should also be able to capture the presented processed results (existence of an apparent "risk-aversion"), but should have slightly different temporal dynamics, and would lead to a completely different interpretation of the results.</p><p>c. Please report parameter β, age effects. Also provide more model validation.</p><p>d. Have the author investigated the effect of forced vs. free choice trials, both on RL and memory? There is evidence that this leads to differential learning processes, and potentially differential attention, which could impact both the learning and memory findings.</p><p>3. Memory findings:</p><p>a. Can the authors comment on the role of attention? Presumably, the participants pay more attention to the outcome of probabilistic than deterministic choices. Could this be a factor in encoding the image, instead of the actual RPE strength? Is there some analysis in probabilistic trials that could be done to show convincingly that the actual size of the RPE matters? In the current analysis, it seems confounded with condition.</p><p>b. While the overall statistical pattern is replicated (Figure 4 and 6A), the realization of the triple interaction looks different in the two experiments (Figure 4 and 6B). In the replication, the asymmetry seems to not matter for positive RPEs, and to have a stronger effect for negative RPEs. For the new study, the patterns seem symmetrical between positive and negative RPEs. Can the authors comment on this?</p><p><italic>Reviewer #3:</italic></p><p>This work provides novel insights about how good and bad feedback differentially influence risky decision making with age from childhood to young adulthood. Further the authors examine how valenced learning biases (updating from positive or negative feedback) influence subsequent memory. The authors tested individuals age 8 to 27 using a risk-sensitive learning task. The task design allowed for the authors to measure learning asymmetries following choices with equivalent expected value but different levels of risk. Learning feedback was accompanied by trial unique images, which allowed for an examination of how learning biases modulate memory. They found that adolescents made more risk-averse decisions, and they over-weighted negative prediction errors during learning. Memory analyses revealed that individual differences in asymmetry bias influenced subsequent memory, an effect that was replicated in an independent sample. The results strongly support the conclusions put forth in this paper. These findings provide timely contributions to both developmental cognitive neuroscience and contemporary research in memory and decision making. The current work provides important revisions to theoretical models in both fields.</p><p>Strengths</p><p>Prevailing theories of cognitive development posit that adolescents are uniquely attuned to rewards. Yet, little research has investigated developmental differences in sensitivity to negative feedback. The present study advances our understanding of how valence asymmetries (learning from positive vs. negative feedback) during learning bias decision making and subsequent memory across development. The authors use an experimental paradigm that assesses risk-sensitive reinforcement learning. This approach is well-suited to identify individual biases in using positive and negative feedback to guide decision making. The results provide novel, yet surprising, evidence that adolescents are more risk-averse due to enhanced sensitivity to negative feedback. These findings offer an important revision to the current theoretical models of adolescent behavior.</p><p>The subsequent memory findings advance our understanding of how prediction errors during learning modulate memory encoding. Prior work has produced mixed results as to whether the valence or magnitude of prediction error influences memory. Here, the authors illustrate that individual differences in learning biases can account for when outcome valence modulates memory. The authors report that asymmetric biases in updating from positive vs. negative prediction errors during learning modulate subsequent memory. However, this effect depends on individual differences in learning asymmetries. Therefore, reward enhances subsequent memory for individuals who are more sensitive to positive prediction errors, however punishment enhances subsequent memory in individuals who are more sensitive to negative prediction errors. A major strength of this paper is that the authors reproduce this memory finding by conducting novel analyses in an existing dataset that was collected in an adult sample.</p><p>A major strength of this paper is the analytical approach. The authors implement rigorous computational modeling procedures. They test multiple models of learning and run formal model comparisons to identify the best fitting model. The authors also run simulations and present parameter and model recovery. To test age-related differences, the authors fit both linear and nonlinear age terms. Moreover, to confirm the strength of nonlinear effects, they conducted break-point analyses using segmented regression, as recommended by the Simohnson two-line approach.</p><p>Weaknesses</p><p>This paper presents data from a cross-sectional sample. This raises questions as to whether learning asymmetries are a stable individual characteristic, or whether these biases exhibit within-person changes with age. Nonetheless, the results of this paper provide important advances to our understanding of age-related differences in learning, decision making, and memory that can form the basis of future longitudinal studies.</p><p>Comments for the authors:</p><p>This manuscript is exceptionally well-written, and the authors present the methods and findings very clearly. I commend the authors approach to computational modeling and nonlinear age analyses. The present findings provide exciting and novel insights about how adolescents approach risky decision making, which in turn has consequences for memory formation. This is a strong paper, and I believe that the current conclusions warrant publication in <italic>eLife</italic>. However, I also believe that the inclusion of some additional analyses and clarifications, which I offer below, will further strengthen this manuscript.</p><p>In the introduction, the authors explain how prior research in developmental samples cannot disentangle learning asymmetries because performance in prior tasks improved if individuals relied upon updating from positive prediction errors. To clarify this point, and to emphasize the novelty of the current study, it would be helpful if the authors provide a more detailed explanation as to how the present design differs from the paradigm used in Van den Bos 2012.</p><p>In the present paper, the authors run model comparison for a basic TD model and a risk-sensitive reinforcement learning model. However, risk sensitivity may be influenced by nonlinear utility functions. It would be informative if the authors also discussed the utility model, as presented in the Niv 2012 paper, which first introduced the present behavioral paradigm. If the authors did not fit this model, please provide an explanation as to why this model was not tested in the current sample.</p><p>Prior work from this group has shown that choice and agency can influence memory (e.g. Katzman & Hartley 2020). Therefore, for the memory data, it would be helpful if the authors accounted for the different trial types (choice trials vs. forced trials) in the analyses.</p><p>Due to the cross-sectional nature of the sample, it is unclear if asymmetry biases are a trait-stable characteristic, or whether this bias changes with age within an individual. It would be helpful for the authors to address this in the discussion.</p></body></sub-article><sub-article article-type="reply" id="sa2"><front-stub><article-id pub-id-type="doi">10.7554/eLife.64620.sa2</article-id><title-group><article-title>Author response</article-title></title-group></front-stub><body><disp-quote content-type="editor-comment"><p>Essential revisions:</p><p>1. All three Reviewers had concerns about the modelling results. Many of these concerns stemmed from the lack of consideration of competing accounts for the data and/or limited justification provided for the choice of models. Please (a) clarify the use of the Pavlovian reinforcement learning model in Experiment 2 (Reviewer 1), (b) relate the current work to the utility model originally presented in the Niv 2012 paper which introduced this behavioural paradigm (Reviewer 3), and (c) add fits of additional competing models. Specifically, with regards to point (c), fitting the subjective value model to account for prospect theory, and the subjective utility model, would be informative.</p></disp-quote><p>We thank the reviewers for these helpful suggestions to clarify our modeling results. Below we address each of the three points (a-c) individually in turn.</p><p>a) A critical distinction between Experiments 1 and 2 is that, while participants in Experiment 1 made choices between point machines that influenced the outcomes they observed, participants in Experiment 2 provided explicit numerical predictions of the likely outcome value for each stimulus image, and the observed outcomes were drawn from a fixed distribution. We referred to the Experiment 2 model as “Pavlovian” because participants made no choices, and thus our model was fit to participants’ trial-by-trial predictions rather than to choices. However, we now recognize that the terminology was confusing and we now refer to the model as the “Explicit Prediction” model.</p><p>On the implementation level, the RSTD model and the Explicit Prediction model differ in that the RSTD model requires an extra parameter, <italic>β</italic>. This extra parameter allows us to use the softmax function to convert the relative estimated values of the two machines into a probability of choosing each machine presented in Experiment 1, which is then compared to participants’ actual choices during maximum likelihood estimation. In contrast, in Experiment 2 participants explicitly report their value predictions (and do not make choices), so the model’s free parameters can be fit by minimizing the difference between the model’s value estimates and participants’ explicit predictions.</p><p>To clarify these differences, in the manuscript, we have changed all references to the “Pavlovian model” to the “Explicit Prediction model”, and we have augmented our description of the distinction between the Explicit Prediction and RSTD models as follows:</p><p>Results:</p><p>“To quantify valence biases in this task, we fit an “Explicit Prediction” RL model that was similar to the RSTD model used in Experiment 1, but was fit to participants’ trial-by-trial predictions rather than to choices. […] Results are reported in Figure 6.”</p><p>Methods:</p><p>“In order to derive each participant’s AI, we fit an “Explicit Prediction” RL model to the participants’ estimation data (see Appendix 1 for more details on our model specification and fitting procedure). […] In contrast, in Experiment 2, participants explicitly reported their value predictions (and did not make choices), so the model’s free parameters were fit by minimizing the difference between the model’s value estimates and participants’ explicit predictions.”</p><p>b) As noted by the reviewers, in the Niv et al., 2012 paper that informed the current study, an additional “Utility” model that converted outcome values into subjective utilities was also fit to the data. We now include a Utility model in the manuscript. This model is necessarily different from the Utility model presented in Niv et al. (2012) due to a critical difference between our experimental paradigms. Specifically, in Niv et al. (2012), there was only one pair of equal-EV risky/safe machines (safe 20 points or 50% 0 points, 50% 40 points). With this version of the paradigm, the authors assumed that “<italic>U</italic>(0)=0, <italic>U</italic>(20)=20, and <italic>U</italic>(40)=<italic>a</italic>*20,” with smaller values of their utility parameter, <italic>a,</italic> consistent with a concave utility curve, and larger values of <italic>a</italic> consistent with a convex utility curve.</p><p>Our version of the task introduced a second pair of equal-EV risky and safe machines (safe 40 points or 50% 0 points, 50% 80 points). Because subjective utilities are nonlinear transformations of value, we could not apply the Utility model from <italic>Niv</italic> et al. (2012) to our present paradigm. Instead, we fit a model in which all outcomes were first transformed into utilities with an exponential subjective utility function (<italic>U</italic>(value)=value<italic><sup>ρ</sup></italic>; Pratt 1964) within the same value update equation used in the TD model.</p><p>We conducted simulations with the Utility model to assess parameter and model recoverability. As with the RSTD model, we simulated 10,000 participants with <italic>α</italic>, <italic>ρ</italic> and <italic>β</italic> drawn randomly from the distribution of fit parameters observed in our empirical data. We found that all parameters were comparably recoverable to those from the RSTD model (Utility: <italic>α</italic>: <italic>r</italic> = .75, <italic>ρ</italic>: <italic>r</italic> = .88, <italic>β</italic>: <italic>r</italic> = .88; RSTD: <italic>α<sup>+</sup></italic>: <italic>r</italic> = .79, <italic>α<sup>-</sup></italic>: <italic>r</italic> = .88, <italic>β</italic>: <italic>r</italic> = .90). Additionally, model recovery analyses indicated that the RSTD and Utility models were reasonably identifiable. 76% of simulated participants’ data generated by the Utility model were best fit by the Utility model, and 65% of simulated participants’ data generated by the RSTD model were best fit by the RSTD model.</p><p>Assessment of which model provided the best fit to participants’ data was inconclusive. <xref ref-type="fig" rid="sa2fig1">Author response image 1</xref> displays BICs for each participant for the RSTD and Utility models. At the group level, the <italic>median</italic> BIC for the Utility model was numerically lower than the RSTD model (Utility: 131.06, RSTD: 131.93). At the single participant level, 36 participants were best fit by the Utility model whereas 26 participants were best fit by the RSTD model. The median within-subjects ΔBIC between the two models was 0.33 while the mean ΔBIC was 1.30, both less than the metric of ΔBIC > 6 that is taken as evidence of a superior fit (Raftery 1995). Taken together, relative BIC metrics at the group and subject level did not provide strong evidence (i.e., ΔBIC > 6) in favor of either model.</p><fig id="sa2fig1" position="float"><label>Author response image 1.</label><caption><title>BICs for each participant from RSTD and Utility models.</title></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/sa2-fig1.jpg"/></fig><p>To examine whether one model provided a better qualitative fit to the data, we next conducted a posterior predictive check. Here, we tested the extent to which the choice predictions made by each model correlated with the actual choice behavior of participants. We simulated 100 subjects for every real participant’s empirically derived parameters, using both the Utility model and the RSTD model, and took the mean proportion of risks across all 100 simulated subjects for each model. Choices derived from RSTD model simulations using each participants best-fit parameter estimates exhibited a significantly stronger correlation with actual choices (<italic>r</italic> = .92) than those simulated using the Utility model (<italic>r</italic> = .89; <italic>t</italic>(61) = 2.58, <italic>p</italic> <.012). Thus, it appears that the RSTD model provides a better qualitative fit to the choice data compared to the Utility model.</p><p>We next asked whether one model might provide a better account of the observed relation between reinforcement learning biases and subsequent memory reported in our manuscript. We previously found that subsequent memory for images encountered during learning related to individual valence biases in learning. That is, we observed a significant 3-way interaction effect of PE valence, PE magnitude and the asymmetry index (AI; or the relative size of <italic>α<sup>+</sup></italic> and <italic>α<sup>-</sup></italic> derived from the RSTD model) on memory accuracy.</p><p>In order to examine whether choice biases indexed by the Utility model could similarly explain the memory data, we first assessed the relationship between RSTD and Utility model parameters. We found that the AI, derived from the RSTD parameter estimates, and <italic>ρ</italic> from the Utility model were highly correlated (<italic>r</italic> = .95, <italic>p</italic> <.001; Appendix 1—figure 11A), suggesting that they provided comparable individual difference measures.</p><p>However, due to the nonlinear transformation of outcome values within the Utility model, the two models yield very different value estimates and Pes. PE magnitudes derived from the Utility model also vary widely across participants. This pattern is clear in Appendix 1—figure 11B, in which we have plotted the PEs across all participants from both the RSTD and Utility models. The dot color represents proportion risk taking. The purple dots are PEs from risk-seeking participants, while green dots are from risk-averse participants, with lighter dots representing people who took risks on about half of trials and darker dots representing those whose risk taking deviated more from equal numbers of risky and safe choices. Because of the nonlinear Utility function, Utility PE magnitudes are much higher for risk-seeking participants (for whom <italic>ρ</italic> > 1) versus risk-averse participants (for whom <italic>ρ</italic> < 1). In contrast, PE magnitudes from the linear RSTD model are necessarily constrained between 0 and 80 and are thus more uniformly distributed across participants regardless of risk preference. Appendix 1—figure 11C and D display the relationship between PEs from Utility and RSTD models from example risk-seeking (Appendix 1—figure 11C) and risk-averse (Appendix 1—figure 11D) participants. The participant represented in Appendix 1—figure 11C chose the risky option on 60% of trials in which the EV of risky and safe options were equal. This participant’s <italic>ρ</italic> estimate from the Utility model was 2.5, and the RSTD AI was.56. In contrast the participant in Appendix 1—figure 11D took risks on 24% of equal-EV risk trials and had a corresponding <italic>ρ</italic> of.56 and AI of -.45. The PEs from the two models are quite different in both absolute and relative scale. For the risk-seeking participant, the nonlinear transformation of outcome values means that Utility PEs reached magnitudes over 50,000, while RSTD PEs could only reach a maximum magnitude of 80.</p><p>We next tested whether the <italic>ρ</italic> parameter and PEs derived from the Utility model could account for subsequent memory in the same manner that we observed for the RSTD model. As <italic>ρ</italic> and AI are so highly correlated, any difference in explanatory ability of the two models would necessarily stem from differences in the ability of their distinct PE dynamics to capture trial-by-trial variability in subsequent memory. In contrast to the RSTD results, we did not find a significant 3-way interaction between <italic>ρ</italic>, PE magnitude and PE valence (using the PEs derived from the Utility model <italic>z =</italic> 1.06, <italic>p</italic> = .291, <italic>OR</italic> = 1.18, 95% CI [0.87, 1.60]).</p><p>This absence of an effect likely stems from the large variability in PE magnitude across participants within the Utility model (Appendix 1—figure 11D) than the RSTD model (Appendix 1—figure 11E). Under the Utility model, risk-seeking participants (high <italic>ρ)</italic> experience much larger magnitude PEs, while PEs for risk-averse participants are smaller in magnitude. Thus, within the multilevel model, high <italic>ρ</italic> participants have not only greater means but also greater variance in their PEs, leading to an inherent prediction that risk-seeking participants should experience the strongest PE-dependent memory modulation. However, in our prior analysis using the RSTD model, valence biases in the effects of PE sign and magnitude on subsequent memory were evident in both risk-seeking and risk-averse participants. This suggests that only the linear outcome values within the RSTD model can adequately capture this pattern.</p><p>To summarize, although we found that the Utility and RSTD models provided a similar degree of fit to the learning data based on BIC, the difference in fit between the models was equivocal (median ΔBIC = .33, mean ΔBIC = 1.30; both less than ΔBIC of 6; Raftery 1995). Moreover, the RSTD model provided a better fit to the learning data (based on our posterior predictive check) and was uniquely able to account for individual differences in the memory data (based on our multilevel model results). Given the better predictive fit of the RSTD model to both the learning and memory data, we now present the comparison of both models within the revised manuscript, but we continue to focus the framing of the study and our central results in the main text on the RSTD model. However, we have also expanded the part of the discussion in which we discuss the relation between the conceptual constructs of asymmetric learning rates and nonlinear utilities as accounts of risk preferences.</p><p>In the main text Methods, we introduce the Utility model as an alternative model fit to the data and present the model comparison results. The details of the posterior predictive check, and model and parameter recoverability have been added to the supplement. We have also expanded our discussion of the conceptual relations between the dual learning rate and subjective utility frameworks.</p><p>Methods:</p><p>“Utility Model.</p><p>As a further point of comparison with the TD, RSTD, and FourLR models, we estimated a utility model that employed the same value update equation as the TD model, <italic>Q<sub>M</sub></italic>(<italic>t</italic>+1) = <italic>Q<sub>M</sub></italic>(<italic>t</italic>) + <italic>α</italic> * δ(<italic>t</italic>). […] In contrast, ρ>1 corresponds to a convex utility function that yields risk-seeking behavior.</p><p>Methods:</p><p>“Parameter and Model Recovery</p><p>For each model, we simulated data for 10,000 subjects with values of each parameter drawn randomly and uniformly from the range of possible parameter values. […] We also found that RSTD model parameters were reasonably well recovered across the range of AI observed in our empirical sample (see Appendix 1, Appendix 1—figure 8).”</p><p>Results:</p><p>“To better understand the learning processes underlying individuals’ decision making, we compared the fit of four Reinforcement Learning (RL) models to participants’ choice behavior. […] Because the RSTD model fit choice data approximately as well as the Utility model, provided a significantly better qualitative fit to the choice data, and yielded an index of valence biases in learning, we focused our remaining analyses on the RSTD model (see Appendix 1 for additional model comparison analyses, and for an examination of the relation between the Utility model and subsequent memory data).”</p><p>Discussion:</p><p>“Traditional behavioral economic models of choice suggest that risk preferences stem from a nonlinear transformation of objective value into subjective utility (Bernoulli, 1954; Kahneman & Tversky, 1979), with decreases in the marginal utility produced by each unit of objective value (i.e., a concave utility curve) producing risk aversion. […] However, a potential parsimonious account is that a risk-sensitive learning algorithm could represent a biologically plausible process for the construction of risk preferences (Dabney et al., 2020), in which distortions of value are produced through differential subjective weighting of good and bad choice outcomes (Mihatsch & Neuneier, 2002; Niv et al., 2012).”</p><p>Appendix 1:</p><p>“We ran a posterior predictive check on both the RSTD and Utility models. We simulated 100 subjects for every real participant’s empirically derived parameters, using both the Utility model and the RSTD model, and took the mean proportion of risks across all 100 simulated subjects for each model. […] Thus, it appears that the RSTD model provides a better qualitative fit to the choice data compared to the Utility model.”</p><p>Additionally, we included Appendix 1—figures 11A-F, along with the explanation preceding those figures in Appendix 1.</p><p>c) In addition to the TD, RSTD, and Utility models, we compared one additional model in response to Essential Revision 2 below concerning potential differences in learning from forced versus free choices. This model included Four learning rates (henceforth FourLR): <italic>α<sup>+</sup></italic> and <italic>α<sup>-</sup></italic> for free choices and <italic>α<sup>+</sup></italic> and <italic>α<sup>-</sup></italic> for forced choices. Although the FourLR model performed better than the TD model (see (p. 31, lines 1230-1232, Appendix 1—figure 5).; FourLR Median BIC: 141.25; TD Median BIC: 145.35), FourLR median BIC was worse than both RSTD and Utility models (Utility: 131.06, RSTD: 131.93). On the individual participant level the RSTD provided a better fit than FourLR for 57 out of 62 participants (Median ΔBIC = 8.59). The Utility model fit better than the FourLR model for 55 out of 62 participants (Median ΔBIC = 8.41). Moreover, although parameter recovery was reasonable for FourLR (Table 1), model recovery was poor (Table 2). That is, when we simulated participants with the FourLR model and fit the data using the alternative models, FourLR fit better than RSTD for only 31% of participants and Utility for only 39% of participants, suggesting that our experimental paradigm was not able to reliably distinguish the value computation process formalized within the FourLR model from those of these more parsimonious models. Since the RSTD and Utility models both performed better than the FourLR model in model comparison, we focus on RSTD and Utility in the main text. However, we added a brief description of the FourLR model to the main text and included additional analyses of FourLR parameters in Appendix 1 (see response to Essential Revision 2 for more information).</p><p>Finally, the suggestion was raised that we should fit both a Prospect Theory model and a subjective utility model. However, given the set of choices participants faced in our task, these two models would have equivalent explanatory power. All outcomes in our task were in the gain domain, so a loss aversion parameter could not be estimated. Moreover, we did not vary outcome probabilities of the stochastic choices in the task, and thus nonlinear probability weighting also could not be assessed. In the Utility model, the exponential value transformation by <italic>ρ</italic> is no different from Prospect Theory’s exponential value transformation in the gain domain. Given this formal equivalence in the context of our task, we did not compare any additional Prospect Theory model.</p><disp-quote content-type="editor-comment"><p>2. Please account for the effect of forced vs. choice trials in both reinforcement learning and memory. All Reviewers highlighted this as an important consideration, as there is evidence that this distinction could lead to differential learning, attention, and/or memory (as shown previously by this group, e.g., Katzman & Hartley 2020).</p></disp-quote><p>As noted by the reviewers, past studies (including work from our group) have demonstrated differential learning, attention, and memory when making free versus forced choices (Katzman and Hartley 2020; Cockburn et al. 2014; Chambon et al. 2020). Below we address the additional analyses and corresponding changes to the manuscript that we have made to address the myriad potential ways in which making free versus forced choices may have influenced our results.</p><p>Prior studies have demonstrated that learning asymmetries can vary as a function of whether choices are free or forced, such that participants tend to exhibit a greater positive learning rate bias (<italic>α<sup>+</sup></italic> > <italic>α<sup>-</sup></italic>) for free choices and either no bias or a negative bias (<italic>α<sup>+</sup></italic>< <italic>α<sup>-</sup></italic>) in forced choices (Cockburn et al. 2014; Chambon et al. 2020). To understand whether our participants exhibited different learning biases in free and forced trials, we implemented an additional reinforcement learning model with four learning rates (FourLR, briefly referenced above in response to Essential Revision 1C). Here, we used separate <italic>α</italic><sup>+</sup> and <italic>α</italic><sup>-</sup> for forced and free choices. Although this least-parsimonious model provided a better fit to the choice data than the TD model, the FourLR model performed worse than both the Utility and RSTD models at both the group and participant level (see Appendix 1—figure 5 in response to Essential Revision 1C) and it performed poorly in model recovery analyses.</p><p>Despite the model’s poor performance at fitting choice data relative to more parsimonious models, parameter recovery from the model was reasonable. Thus, we conducted an exploratory analysis of whether learning rates differed as a function of choice agency. Free- and forced-choice learning rates and asymmetry indices are plotted in Appendix 1—figure 12<italic>.</italic> Consistent with prior findings (Cockburn et al. 2014; Chambon et al. 2020), we found that <italic>α<sup>+</sup></italic> was significantly higher in free (Median = .22) compared to forced (Median = .14) choices (Wilcoxon Signed-rank test: <italic>V</italic> = 1338, <italic>p</italic> = .011). In contrast, <italic>α<sup>-</sup></italic> was not significantly different in free (Median = .35) vs. forced (Median = .32) choices (Wilcoxon Signed-rank test: <italic>V</italic> = 794, <italic>p</italic> = .202). We computed separate AIs for free and forced choices, and found AI was significantly higher for free (Median = -.12) compared to forced (Median = -.39) choices (Wilcoxon Signed-rank test: <italic>V</italic> = 1377, <italic>p</italic> = .005). Although <italic>α<sup>+</sup></italic> and AI were higher in free versus forced trials, median AIs were negative for both free and forced choices. Therefore, in this study, we did not observe positive learning rate asymmetries for free choices.</p><p>In the main text, we now briefly mention the FourLR model in the Results (please see revisions in Essential Revision 1B in addition to revisions below), and Methods sections. These changes are detailed here:</p><p>Results:</p><p>“Prior work has found positive valence biases tend to be positive in free choices, but neutral or negative in forced choices (Chambon et al. 2020; Cockburn et al. 2014). […] While the <italic>α</italic>+ and AI were both higher for free compared to forced trials, median AIs were negative for both free and forced choices (see Appendix 1 for full results; Appendix 1—figure 12).”</p><p>Methods:</p><p>“FourLR Model<italic>.</italic> In our task, participants made both free and forced choices. Past research suggests that valence biases in learning may differ as a function of choice agency (Cockburn et al. 2014; Chambon et al. 2020). To test this possibility, we assessed the fit of a Four Learning Rate (FourLR) model, which was the same as the RSTD model except that it included four learning rates instead of two, with separate <italic>α</italic><sup>+</sup> and <italic>α</italic><sup>-</sup> parameters for free and forced choices.”</p><p>The preceding explanation is included in Appendix 1 (Appendix 1—figure 12).</p><p>As the reviewers pointed out, prior research has also found better subsequent memory for memoranda associated with free versus forced choices (Murty et al. 2015; Katzman and Hartley 2020). The present study was not designed to test the effects of agency on memory. The purpose of our forced-choice trials was to ensure all participants had similar opportunities to learn probabilities and outcomes associated with each probabilistic and deterministic point machine, regardless of their risk preferences. However, to test whether memory was different for items that appeared with outcomes of free versus forced choices, we ran two additional glmer models. First, we ran the most complex model included in the original manuscript (the model predicting memory accuracy that tested for a PE valence x PE magnitude x AI interaction, and also included linear and quadratic age and memory trial number), adding a predictor that indicating whether the image appeared after a free or forced choice. Memory did not vary as a function of choice agency (<italic>z</italic> = -0.43, <italic>p</italic> = .667, <italic>OR</italic> = 0.99, 95% CI: [0.95-1.04]).</p><p>As a final test for a potential agency benefit on memory, we tested whether the 3-way PE Valence x PE Magnitude x AI interaction predicting memory performance varied as a function of whether the choice was forced or free (i.e., we tested for a 4-way PE Valence x PE Magnitude x AI x choice agency interaction). The 4-way interaction was also not significant (<italic>z</italic> = -1.39, <italic>p</italic> = .165, <italic>OR</italic> = 0.96, 95% CI: [0.91, 1.02]). To further explore whether agency had any apparent qualitative effect, we plotted the 3-way interaction effect — PE Valence x PE Magnitude x AI) separately for free versus forced choice trials (Appendix 1—figure 13). Strikingly, we found that for free-choice trials (Appendix 1—figure 13A), which comprised the majority of trials, the interaction looks qualitatively similar to the overall 3-way interaction effect across all trials that we report in the main text <italic>(</italic>Figure 4B in the manuscript, shown in Appendix 1—figure 13C). However, for forced trials (Appendix 1—figure 13B), memory performance for images that coincided with positive Pes did not appear to be modulated by AI. This pattern is similar to the memory effect observed in Experiment 2 (Figure 6B in the manuscript, shown in Appendix 1—figure 13D, in which participants provided explicit predictions of outcomes, but did not make choices. This qualitative similarity suggests that memory for surprising high-magnitude positive outcomes may be equivalently facilitated for all individuals in situations where they do <italic>not</italic> have agency, regardless of their idiosyncratic biases in learning.</p><p>We added text to the Results and Discussion sections describing prior findings on agency and memory in relation to our study. We also added the multilevel models described above to Appendix 1, with the free vs. forced choice predictor as a main effect as well as the model with the 4-way PE Valence x PE Magnitude x AI x choice agency interaction:</p><p>Results:</p><p>“Finally, we tested for effects of agency – whether an image coincided with the outcome of a free or forced choice – on memory performance. We did not find a significant main effect of agency on memory, and agency did not significantly modulate the AI x PE magnitude x PE valence interaction effect (see Appendix 1 for full results).”</p><p>We have now added a paragraph to the discussion giving a fuller account of the idea that learning and memory may differ as a function of whether choices are self-determined or imposed. In this paragraph, we also discuss observed parallels between the patterns of effects observed on forced choice trials, and those from Experiment 2 in which participants were not able to make choices, and speculate that these patterns may reflect common effects of lack of agency on learning and memory. The corresponding revised sections are excerpted below:</p><p>Discussion:</p><p>“In Experiment 1 of the present study, participants observed the outcomes of both free and forced choices. […] Thus, while our study was not explicitly designed to test for such effects, this preliminary evidence suggests that choice agency may modulate the relation between valence biases in learning and corresponding biases in long-term memory, a hypothesis that should be directly assessed in future studies.”</p><disp-quote content-type="editor-comment"><p>3. Please include additional data visualizations that depict the raw data, which will better equip the reader for interpreting the results (e.g., the learning curves per experimental condition, and/or per age group as suggested by Reviewer 2).</p></disp-quote><p>We added several new figures that depict the raw data, which we hope will aid in interpreting our results.</p><p>First, we now include a depiction of mean accuracy on test trials (where one option dominated the other, and thus there was a “correct” choice) across the task, separately by age group (Appendix 1—figure 1A). This plot clearly reveals that there were no age differences in learning trajectory or asymptotic accuracy.</p><p>Reviewer 2 suggested plotting learning curves by experimental condition. We want to clarify that our experiment does not involve different conditions. Rather, participants encounter one of a number of pairs of possible options. In some pairs, as in those plotted Appendix 1—figure 1A, there is a correct option, allowing us to index accuracy and assess learning. However, many of our pairs did not involve a “correct” choice, so there is no way to index accuracy on these trials. That is, in risk trials, one option was risky and one was safe, but neither option strictly dominated the other. For these trials, we plotted mean risk taking with age and block separately for equal EV trials (20 vs. 0/40 or 40 vs. 0/80; Appendix 1—figure 1B) and unequal EV trials (20 vs. 0/80; Appendix 1—figure 1C).</p><p>In addition to plotting learning curves, we added several plots that illustrate patterns of memory performance beyond the logistic regression analysis that is the primary focus within the manuscript. We include Appendix 1—figure 3, which depicts the significant relationship between false alarms and age, and the marginally significant relationship between d’ and age, patterns that are reported in the manuscript.</p><disp-quote content-type="editor-comment"><p>4. Please consider the role of attention versus reward prediction error (RPE) in the memory effects. Reviewers pointed out that there may be attentional differences across probabilistic versus deterministic trials that could be driving the effects rather than RPE, which if the source of memory differences would warrant a different conclusion than the current paper. In addition, whether and how attention fits into the relationship between asymmetry index and memory was unclear. The authors may be in a position to address this question by performing additional analyses on the current data (e.g., by assessing whether RPE strength alone is related to encoding within the probabilistic trials, as suggested by Reviewer 2); or more generally (in the absence of data) clarify their position or speculate on these issues.</p></disp-quote><p>As our study did not include attentional measures, we did not focus on the role of attention in the discussion of our results in the initial manuscript. However, given a large literature demonstrating the critical role of attention in reinforcement learning (Pearce and Hall 1980; Dayan et al. 2000; Holland and Schiffino 2016; Radulescu et al. 2019) and memory formation (Chun and Turk-Browne 2007), we agree completely with the reviewers’ intuitions that attention likely played a critical role in the effects we observed.</p><p>Prominent theoretical accounts have proposed that attention should be preferentially allocated to stimuli that are more uncertain (Pearce and Hall 1980; Dayan et al. 2000). Computational formalizations of such attentional modulation of learning posit that greater attention might be reflected by increases in learning rates. Within these models, attention to a stimulus is proportional to the aggregate prediction errors it is associated with. Thus, the notion that greater attention might be paid to the outcomes of probabilistic versus deterministic machines in our task has strong theoretical support. However, rather than prediction errors and attention representing distinct potential mechanisms that might influence learning and memory, these models conceptualize prediction errors as direct drivers of attention allocation.</p><p>In our study, memory was better for items that coincided with probabilistic compared to deterministic outcomes. We think this finding quite likely reflects greater attention to the episodic features associated with outcomes of uncertain choices. Importantly however, our memory findings cannot be solely explained via an uncertainty-driven attention account. As suggested by reviewers, we tested whether differences in memory for outcomes of deterministic versus probabilistic trials were driving the AI x PE magnitude x PE valence effect observed in our Experiment 1 by re-running the regression only within the subset of trials in which participants made probabilistic choices. Our results did not change — we observed both a main effect of PE magnitude (<italic>z</italic> = 2.22, <italic>p</italic> = .026, <italic>OR</italic> = 1.11, 95% CI [1.01,1.23], <italic>N</italic> = 62) and a significant PE valence x PE magnitude x AI interaction (<italic>z</italic> = 2.34, <italic>p</italic> = .019, <italic>OR</italic> = 1.11, 95% CI [1.02,1.21], <italic>N</italic> = 62). While the absence of attentional measures in the current study precludes decisive inferences about potential attentional mechanisms, this analysis suggests that our current results may reflect differential attention to valenced outcomes that varies systematically across individuals in a manner that can be accounted for by asymmetries in their learning rates. Such valence biases in attention have been widely observed in clinical populations (Mogg and Bradley 2016; Bar-Haim et al. 2007), and may also be individually variable within non-clinical populations.</p><p>We have now revised our manuscript to include greater discussion of this putative role of attention in our learning and memory findings:</p><p>Introduction:</p><p>“Outcomes that violate our expectations might also be particularly valuable to remember. […] Moreover, while few studies have explored the development of these interactive learning systems, a recent empirical study observing an effect of prediction errors on recognition memory in adolescents, but not adults (Davidow et al., 2016), suggests that the influence of reinforcement learning signals on memory may be differentially tuned across development.”</p><p>Results:</p><p>“We next tested whether the decision context in which images were presented influenced memory encoding. […] This result suggests that pictures were better remembered when they followed the choice of a machine that consistently generated reward prediction errors, which may reflect preferential allocation of attention toward outcomes of uncertain choices (Pearce and Hall 1980; Dayan et al. 2000).”</p><p>Results:</p><p>“To test whether differences in memory for outcomes of deterministic versus probabilistic trials might have driven the observed AI x PE magnitude x PE valence interaction effect, we re-ran the regression model only within the subset of trials in which participants made probabilistic choices. Our results did not change — we observed both a main effect of PE magnitude (<italic>z</italic> = 2.22, <italic>p</italic> = .026, <italic>OR</italic> = 1.11, 95% CI [1.01,1.23], <italic>N</italic> = 62) and a significant PE valence x PE magnitude x AI interaction (<italic>z</italic> = 2.34, <italic>p</italic> = .019, <italic>OR</italic> = 1.11, 95% CI [1.02,1.21], <italic>N</italic> = 62).”</p><p>Discussion:</p><p>“Attention likely played a critical role in the observed learning and memory effects. […] Such valence biases in attention have been widely observed in clinical disorders (Mogg and Bradley 2016; Bar-Haim et al. 2007), and may also be individually variable within non-clinical populations.”</p><disp-quote content-type="editor-comment"><p>5. Please improve the integration of Experiment 2 into the paper. More details were needed for understanding the purpose of this sample. In addition, Reviewers noted that the interaction found in Experiment 1 did not entirely replicate in Experiment 2. This warrants more consideration in the paper.</p></disp-quote><p>In our revised manuscript, we have attempted to better integrate Experiment 2 into the manuscript. Specifically, we added additional text clarifying that our motivation was to replicate the effect of valence biases in learning on subsequent memory in an independent and larger sample. Additionally, we wondered whether the observed effect was sufficiently robust that it would be evident when participants were not explicitly making choices (i.e., in a task in which participants made predictions about expected outcomes, but could not choose them) and when learning biases reflected idiosyncratic individual differences across a sample of adults, rather than age-related variation.</p><p>We also revised our results and Discussion sections to directly acknowledge the qualitative differences in the interaction patterns observed in Experiments 1 and 2. Specifically, we observed a qualitative similarity between the interaction in Experiment 2, in which participants made explicit predictions but not free choices, and the analysis of the forced-choice trials only in Experiment 1 (see response to Essential Revision 2 above for more details). In both analyses, differences in memory performance as a function of AI were primarily apparent for images coinciding with negative PEs — those who learned more from negative PEs also had better episodic memory for images that coincided with increasingly large negative PEs, while all participants appeared to have stronger memory for images coinciding with larger positive PEs, independent of AI. Based on this qualitative correspondence between the patterns of results, we now introduce a speculative interpretation that the effects of PE on memory may differ in agentic and non-agentic learning contexts.</p><p>To better integrate Experiment 2 into the manuscript, we amended the manuscript as follows:</p><p>Introduction:</p><p>“To determine whether this hypothesized correspondence between valence biases in learning and memory generalized across experimental tasks and samples of different ages, in Experiment 2, we conducted a re-analysis of data from a previous study (Rouhani et al., 2018). […] Here, we tested whether a valence-dependent effect of PE on memory might be evident after accounting for idiosyncratic valence biases in learning.”</p><p>Results:</p><p>“Next, we assessed the generalizability of the observed effect of valence biases in learning on memory by conducting a reanalysis of a previously published independent dataset from a study that used a different experimental task in an adult sample (Rouhani et al., 2018). […] Here, we examined whether signed valence-specific effects might be evident when we account for individual differences in valence biases in learning.”</p><p>Results:</p><p>“Consistent with the results reported in the original manuscript (Rouhani et al., 2018), as well as the findings in Experiment 1, there was a strong main effect of unsigned PE (i.e., PE magnitude) on memory (<italic>z</italic> = 5.09, <italic>p</italic> <.001, <italic>OR</italic> = 1.19, 95% CI [1.12, 1.28]). […] One possibility is that PE magnitude and PE valence enhance memory through separate mechanisms, with a universal positive effect of unsigned PEs but a contextually (depending on choice agency) and individually variable effect of PE valence.”</p><p>Discussion:</p><p>“Studies have also found that subsequent memory is facilitated for images associated with free, relative to forced, choices (Murty et al. 2015; Katzman and Hartley 2020). […] Thus, while our study was not explicitly designed to test for such effects, this preliminary evidence suggests that choice agency may modulate the relation between valence biases in learning and corresponding biases in long-term memory, a hypothesis that should be directly assessed in future studies.”</p><disp-quote content-type="editor-comment"><p>6. Please clarify the task and methods descriptions. For example, it is not clear whether the forced choices were included in the modelling. Moreover, please ensure all details needed to understand the major points of the paper are included in the main text without requiring readers to reference the methods.</p></disp-quote><p>We now include additional text to clarify important methodological details within the Results section. We also modified the text in the Methods section to ensure that our descriptions of the methods were clear and complete. As part of this revision, we clarified that the outcomes of the forced trials were included in the value updating step within the model, but these trials were not included in the modeling stage in which learned values are passed through a softmax to determine choice probabilities, as there was only a single choice option on these trials.</p><p>Results:</p><p>“Participants (<italic>N</italic> = 62) ages 8-27 <italic>(M</italic> = 17.63, <italic>SD</italic> = 5.76) completed a risk-sensitive reinforcement learning (RL) task (Niv et al., 2012). […] A subsequent memory test allowed us to explore the interaction between choice outcomes and memory encoding across age (Figure 1C).”</p><p>Results:</p><p>“Next, we explored whether valence biases in learning could account for individual variability in subsequent memory. […] We also tested for effects of linear and quadratic age, false alarm rate, as a measure of participants’ tendency to generally deem items as old, and trial number in the memory task, to account for fatigue as the task progressed (Figure 4A). ”</p><p>Methods:</p><p>“In all models, Q-values were converted to probabilities of choosing each option in a trial using the softmax rule, P<sub>M1</sub> = e<sup>β*Q(t)M1</sup>/(e<sup>β*Q(t)M1</sup>+ e<sup>β*Q(t)M2</sup>), where P<sub>M1</sub> is the predicted probability of choosing Machine 1, with the inverse temperature parameter <italic>β</italic> capturing how sensitive an individual’s choices are to the difference in value between the two machines. Notably, outcomes of the forced trials were included in the value updating step for each model. However, forced trials were not included in the modeling stage in which learned values are passed through the softmax function to determine choice probabilities, as there was only a single choice option on these trials.”</p><disp-quote content-type="editor-comment"><p>7. As highlighted by Reviewer 3, it is unclear whether asymmetry biases are a trait-stable characteristic, or rather would change with age within an individual. Please speak to this point as well as other limitations associated with the cross-sectional nature of this study in the revised Discussion.</p></disp-quote><p>We agree with the editor and reviewer’s suggestion that the manuscript would benefit from a fuller discussion of how the age differences in valence asymmetry biases that we observed in our cross-sectional sample might be interpreted as reflecting stable or malleable individual differences, as well as the limitations of the inferences that can be made from cross-sectional results.</p><p>Past studies have demonstrated that valence asymmetries in reinforcement learning are malleable within a given individual, exhibiting sensitivity to the statistics of the environment (e.g., the informativeness of positive versus negative outcomes (Pulcu and Browning 2017)) as well as endogenous manipulations such as the pharmacological manipulation of neuromodulatory systems (Michely et al. 2020). In Experiment 1, we aimed to quantify age-related variation in valence biases in learning and memory in a task in which there was no specific level of bias that yielded optimal performance. One of our central motivations for adopting this design was the theoretical suggestion that individual valence biases in learning should be malleable across contexts—that they should adapt to the reward statistics of the environment in a manner that maximizes reward (Cazé and Meer 2013). While prior developmental studies have reported age differences in valence biases across adolescence (Christakou et al. 2013; Van Den Bos et al. 2012; Hauser et al. 2015; Jones et al. 2014; Master et al. 2020), the interpretation of these findings confounded. They may reflect age-differences in an adaptive reward maximization process, rather than performance-independent, age-associated biases (Nussenbaum and Hartley 2019).</p><p>Here, we found that independent of this optimality confound, valence asymmetries in our cross-sectional sample varied nonlinearly as a function of age. These findings raise the possible suggestion that, for a given individual, learning rate asymmetries might become more negative from childhood into adolescence, and more positive from adolescence into young adulthood. However, such predicted patterns of change cannot be validly inferred from cross-sectional studies, which are confounded by potential effects of cohort (Schaie 1965). Future longitudinal studies would be needed to definitively establish whether such age-related changes can be observed within an individual over developmental time.</p><p>We have made the following corresponding modifications to the manuscript:</p><p>Discussion:</p><p>“The present findings raise the suggestion that, for a given individual, valence asymmetries in value-based learning might become more negative from childhood into adolescence, and more positive from adolescence into young adulthood. […] Future longitudinal studies will be needed to definitively establish whether valence biases in learning exhibit systematic age-related changes within an individual over developmental time.”</p><disp-quote content-type="editor-comment"><p>8. The Reviewers had several additional thoughtful suggestions for other work the authors might cite. I recommend the authors consider whether these suggestions might enhance the novelty, impact, and/or clarity of their paper.</p></disp-quote><p>We thank the reviewers for their thoughtful suggestions on additional citations. We added these citations as follows:</p><p>In response to Reviewer 2, we noted that Master et al. (2019) examined reinforcement learning across adolescence, and demonstrated increases in negative learning from childhood into adolescence:</p><p>Introduction:</p><p>“Several past studies have characterized developmental changes in learning from valenced outcomes (Christakou et al. 2013; Hauser et al. 2015; Jones et al. 2014; Master et al. 2020; Moutoussis et al. 2018; Van Den Bos et al. 2012).”</p><p>Discussion:</p><p>“Moreover, heightened reactivity to negatively valenced stimuli has also been observed in adolescents, relative to children (Master et al. 2020) and adults (Galván & McGlennen, 2013). While a relatively small number of studies have used reinforcement learning models to characterize age-related changes in valence-specific value updating (Christakou et al. 2013; Hauser et al. 2015; Jones et al. 2014; Master et al. 2020; Moutoussis et al. 2018; Van Den Bos et al. 2012), age patterns reported in these studies vary substantially and none observed the same pattern of valence asymmetries present in our data.”</p><p>As suggested by Reviewer 3, we briefly described the paradigm in Van den Bos et al. (2012) to demonstrate the novel contribution of the current study:</p><p>“Several past studies have characterized developmental changes in learning from valenced outcomes (Christakou et al. 2013; Hauser et al. 2015; Jones et al. 2014; Master et al. 2020; Moutoussis et al. 2018; Van Den Bos et al. 2012). […] Thus, choice behavior in these studies might reflect both potential age differences in the optimality of reinforcement learning, as well as context-independent differences in the weighting of positive versus negative prediction errors (Cazé & van der Meer, 2013; Nussenbaum & Hartley, 2019).”</p><p>As described in response to Essential Revision 2, we now include analyses of memory as a function of free compared to forced choices, inspired in part by our prior work, along with relevant citations (Katzman & Hartley, 2020; Murty et al., 2015):</p><p>Discussion:</p><p>“Prior studies have demonstrated differential effects of free versus forced choices on both learning and memory (Katzman and Hartley 2020; Cockburn et al. 2014; Chambon et al. 2020), which may reflect greater allocation of attention to contexts in which individuals have agency. […] Studies have also found that subsequent memory is facilitated for images associated with free, relative to forced, choices (Katzman and Hartley 2020; Murty et al. 2015).”</p><disp-quote content-type="editor-comment"><p>9. Please provide additional details on the models (e.g., reporting parameter β, age effects, model validation, and/or more information on the recoverability analysis) as appropriate. One specific question raised by reviewers was whether learning rate parameter estimation might be more difficult in participants with more deterministic choices and thus fewer trials with non-zero RPEs. Is this empirically true, i.e., does model parameter recovery (as shown in Table 1 overall) differ across ages/ranges of behavior?</p></disp-quote><p>Our revised manuscript and supplement now include additional information about differences in parameter estimates and recoverability across age and behavioral patterns. First, as suggested by the reviewers, we assessed whether <italic>β</italic> differed with age in the RSTD model. We found that <italic>β</italic> increased with age numerically, but there was no statistically significant relation with age (Appendix 1—figure 10C; <italic>r</italic> = .18, <italic>p</italic> = .156).</p><p>Next, we tested whether parameter recovery differed as a function of individual differences in the propensity to make deterministic choices. Specifically, we tested whether parameter recovery and model fit (BIC) varied as a function of AI. Importantly, because AI is so highly correlated with risk taking (as discussed in response to Reviewer 1, Essential Revision 1), this analysis allows us to address the reviewers’ questions about potential differences in parameter recoverability for participants who more frequently chose the deterministic point machines (i.e., those with low AI). To this end, we divided the simulated participants into AI quartiles and examined parameter recovery and model fit in each AI quartile. We found that the parameters were reasonably well recovered at all levels of AI (Appendix 1—figure 8A, B, and C.).</p><p>However, parameter recoverability varied across levels of AI, in a manner that runs somewhat counter to the reviewers’ intuitions. In particular, recovery of the <italic>α</italic><sup>+</sup> parameter was relatively poorer for the simulated participants in the High AI quartile (Appendix 1—figure 8A) and <italic>α</italic><sup>-</sup> recoverability was relatively poorer for those in the Low-AI quartile (Appendix 1—figure 8A). Taken together, these patterns suggest that learning rate parameters are relatively less well recovered for individuals with higher AIs (i.e., who made more risk-seeking choices).</p><p>This differential recoverability as a function of AI stems from the interactions between subjects’ risk preferences and the set of risky choice trials presented in our task. There were two types of risky trials in our task: Equal-EV, (0/40 vs. 20, or 0/80 vs. 40) and Unequal-EV (0/80 vs. 20). This particular combination of Equal- and Unequal-EV risk trials led to differential resolution in the estimation of valenced learning rates as a function of AI. Learning rates for risk-averse participants could be estimated more accurately because those who were very risk averse (and thus had a much larger <italic>α</italic><sup>-</sup> than <italic>α</italic><sup>+</sup>) might choose both the safe 40 point option and the safe 20 point option over the 0/80 machine, whereas those who were less risk averse might prefer the safe 40 to the 0/80, but the 0/80 over the safe 20. In contrast, those with high positive AI are likely to choose the risky option on every Equal and Unequal-EV trial, so the ability to distinguish precisely between different levels of <italic>α</italic><sup>+</sup> for those with high AI is diminished. Our model fit results provide further evidence that this lower <italic>α</italic><sup>+</sup> recoverability in individuals with high AIs stems from this aspect of our task structure (Appendix 1—figure 8D). Despite the model’s imprecise <italic>α</italic><sup>+</sup> estimation for these high-AI subjects, the model was able to predict behavior well (i.e., BIC was low), likely because these participants are likely to take risks on all Equal and Unequal-EV risk trials, regardless of their precise <italic>α</italic><sup>+</sup> level.</p><p>This increase in parameter recovery for those participants with low compared to high AI runs counter to the reviewers’ intuitions that smaller PEs may drive worse recovery in these Low-AI participants. Although it is true that PEs were smaller for those with relatively lower AIs (see Appendix 1—figure 11F in response to Essential Revision 1), our design required these low-AI participants to experience high-magnitude PEs from risky choices on some trials. Specifically, in our task, participants encountered forced trials, where participants were required to choose specific machines, which were sometimes risky, and Test trials, where one option dominated the other, and the dominating option was sometimes risky. Thus, including these Forced and Test trials may have boosted our ability to recover learning rates in those with Low AI, while also providing opportunities for participants to sufficiently learn and demonstrate their knowledge of machine outcomes and probabilities.</p><p>Despite these differences in parameter recoverability at different levels of AI within this experimental paradigm, there are several reasons why we don’t believe that these results are problematic for interpreting the current results. First, these simulations were generated by sampling uniformly from the range of learning rate and temperature parameters observed in our empirical sample. Thus, these simulated participants can take on AI levels that are not actually represented within our empirical sample. Indeed, 81% of fit AI values observed in our empirical sample fall within the lower three quartiles of AI values for this simulation, for which parameter recoverability was higher. Moreover, the recoverability estimates in this simulation dramatically overrepresent participants with low levels of decision noise relative to our empirical sample, which distorts these estimates of recoverability to be lower than what would actually be obtained for our empirical sample of participants (e.g., when we exclude any simulated participants with decision noise < 2, which captures the vast majority of participants in our sample, recoverability of <italic>r</italic> values all increase by ~.05).</p><p>To address this comment in the manuscript, we now report the analysis of age differences in the <italic>β</italic> parameter to the Results section and have added these new recoverability analyses to Appendix 1, and we reference these analyses within the Methods section.</p><p>Results:</p><p>“We computed an asymmetry index (AI) for each participant, which reflects the relative size of <italic>α</italic><sup>+</sup> and <italic>α<sup>-</sup></italic>, from the RSTD model. […] Finally, there were no significant linear or quadratic age patterns in the <italic>β</italic> parameter (<italic>p</italic>s >.15, see Appendix 1 for full results; Appendix 1—figure 10C).”</p><p>Methods:</p><p>“We also found that RSTD model parameters were reasonably well recovered across the range of AI observed in our empirical sample (see Appendix 1, Appendix 1—figure 8).”</p><p>Appendix 1:</p><p>“As reported in the main text, the relation between AI and age appears to be driven by quadratic age patterns in <italic>α</italic><sup>-</sup> (<italic>b</italic> = -.09, 95% CI [-0.16, -0.03], <italic>t</italic>(59) = -3.01, <italic>p</italic> = .004, <italic>f</italic><sup>2</sup> = 0.15, 95% CI [0.02, 0.43]; Appendix 1—figure 10B). […] Additionally, the relation between age <italic>β</italic> was not significant (<italic>b</italic> = .56, 95% CI [-0.22, 1.33], <italic>t</italic>(60) = 1.44, <italic>p</italic> = .156, <italic>f</italic><sup>2</sup> = 0.03, 95% CI [0, 0.19]; Appendix 1—figure 10C).”</p><p>Finally, we included the detailed description of recovery as a function of AI in Appendix 1 (p. 33-34, lines 1259-1325, Appendix 1—figure 8).</p><disp-quote content-type="editor-comment"><p>Reviewer #1:</p><p>[…] 1. I am left unsure as to the value added by the reinforcement learning model and whether the asymmetry index reflects something unique about learning as compared with a general bias towards deterministic choices in the adolescent period. That is, if the authors wish to make claims about asymmetry of learning rate specifically (α) it would be important to know if this is dissociable from the choice or sampling bias they observe. Empirically, is it the case that there would be a similar set of findings if one focused on the percent probabilistic choices value per participant (data in Figure 2) rather than the asymmetry index (Figure 3)? If these are somewhat redundant or overlapping metrics, I would encourage the authors to clarify this in the paper and perhaps only focus on one of them in the main text, so as not to imply these are separate findings.</p></disp-quote><p>We thank the reviewer for this insightful comment. It is true that asymmetry index is highly correlated with percent probabilistic choices. However, we feel that the RL models are valuable in that they enable us to test hypotheses about putative decision processes that drive risk taking. This point is perhaps even clearer in the revised manuscript, in which we now include two additional models of the learning process within our model comparison. Moreover, although AI and percent risky choice were highly correlated, the models provided additional value in enabling us to test the hypothesis that learning biases might be related to subsequent memory. The formal RL models are necessary for computing the individually varying trial-by-trial PEs for each participant that we use as predictors for the memory analysis.</p><p>To further demonstrate the explanatory value of PEs derived from RL models, we ran an analysis analogous to the multilevel model described throughout our paper, but without measures derived from RL models. We tested for a 3-way interaction in subsequent memory between a participant’s percent risk taking, the outcome magnitude (similar to PE magnitude, coded as the absolute value of the outcome magnitude minus 40), and whether the outcome value was greater than or equal to 40 versus less than 40 (similar to PE valence). Although there was a significant effect of outcome magnitude, such that images coinciding with more extreme outcomes (0 and 80 points) were better remembered than medium point outcomes, similar to the PE magnitude effect in memory, the 3-way interaction was not significant (<italic>p</italic> = .638). We included a plot of the 3-way interaction in <xref ref-type="fig" rid="sa2fig2">Author response image 2</xref>, which shows a different pattern from the 3-way interaction effect observed using PEs and AIs derived from the RSTD model. Collectively, our findings suggest that the RL models are helpful in understanding the mechanisms underlying choice, and yield measures (PEs and AIs) that can better explain individually varying valence biases in both learning and memory.</p><fig id="sa2fig2" position="float"><label>Author response image 2.</label><caption><title>Non-significant results from a 3-way interaction in subsequent memory data using individual difference and outcome value measures not derived from a reinforcement learning model.</title></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="elife-64620.xml.media/sa2-fig2.jpg"/></fig><disp-quote content-type="editor-comment"><p>2. Related to my above point, those individuals who make fewer probabilistic choices (disproportionately adolescents) have fewer opportunities to learn from prediction errors that are either positive or negative. That is, in my understanding, there will be no prediction error from the deterministic machines which the adolescents primarily select. Their bias to select deterministic machines seems fairly extreme; it appears as though they shy away from the probabilistic choices across the board, even when the value of the probabilistic choice on average would greatly exceed that of the deterministic choice (e.g., as shown in Figure S2). As such, I am wondering whether theoretically the authors can clarify the relationship between choice and learning rate; and analytically whether the bias towards deterministic choices could impact the model fitting procedures.</p></disp-quote><p>We appreciate this valuable comment. We addressed this comment in response to Essential Revision 9. Briefly, it is true that those with more negative learning biases also make fewer probabilistic choices on “risk” trials – that is, trials in which they face a choice between a risky and a safe option, neither of which dominates the other. However, these participants have opportunities to experience large PEs on trials where probabilistic machines are the only option (i.e. “forced” choices) and on some “test” trials where a probabilistic machine is the dominating option.</p><p>We analytically tested whether biases toward deterministic choices might have impeded model fitting for risk-averse participants. Counterintuitively, we found somewhat <italic>better</italic> parameter recovery for those biased toward safe choices. Please see Essential Revision 9 for a detailed summary of this analysis.</p><disp-quote content-type="editor-comment"><p>3. As an additional interpretive question, I had trouble linking the asymmetry index to the memory performance and understanding how the authors believed these metrics to be related. Is there a general increase in interest or attention or salience when a prediction error aligning with the learner's biases (e.g., a highly positive prediction error if I'm someone who learns best from those types of trials – that is, showing a positive AI) happens, therefore indirectly impacting memory formation in the process? Or, is this thought to be a separate mechanism that occurs (the learning from positive prediction errors vs. "prioritization" in memory of those positively valenced experiences)? It seems to me as though both are related to learning/encoding, and thus this individual difference may not be terribly surprising, but I was unsure about the authors' preferred interpretation.</p></disp-quote><p>We appreciate this insightful comment. We agree that individual differences in value computation and memory performance may likely reflect individually varying valence biases in attentional allocation. Such valence biases in attention have been widely observed in clinical disorders (Mogg and Bradley 2016; Bar-Haim et al. 2007), and may also be evident within non-clinical populations.</p><p>Please see our response to Essential Revision 4, where we now describe in greater detail hypotheses regarding the role of attention in our findings, and our manuscript revisions that discuss such potential attentional mechanisms.</p><disp-quote content-type="editor-comment"><p>4. While I appreciated the inclusion of experiment 2, I felt that it was not particularly well integrated into the paper. For example, why did the authors opt to use data with only adults rather than a developmental sample (and does this constrain interpretation in any way)? In addition, it is important to highlight that the results do not fully replicate the main findings. In particular, there is no difference in the relationship between the magnitude of prediction error and memory among positively valenced trials according to AI, which was observed in the main developmental experiment 1. This discrepancy warrants more attention in the paper. (Could this be about the sample characteristics? Task differences? and so on.)</p></disp-quote><p>We thank the reviewer for this helpful comment. In brief, our high-level motivation for Experiment 2 was to replicate the effect of valence biases in learning on subsequent memory in an independent and larger sample. While the use of this particular dataset was admittedly opportunistic (this was the only dataset amenable to RL with trial-unique memoranda that we were able to gain access to), we were interested in testing whether the observed effect was sufficiently robust that it would be evident when participants were not explicitly making choices (i.e., in a task in which participants made predictions about expected outcomes but could not choose them) and when learning biases reflected idiosyncratic individual differences across a sample of adults, rather than age-related variation. Please see our response to Essential Revision 5, where we describe our improved integration of Experiment 2 into the manuscript.</p><p>Additionally, we now more explicitly acknowledge that Experiment 2 does not fully replicate Experiment 1. In addressing Essential Revision 2 about agency and the observed individual differences in memory, we discovered that the memory patterns in forced trials of Experiment 1 (i.e., trials where participants were only given one option and therefore did not make an autonomous choice) are similar to the pattern observed in Experiment 2, where participants made predictions rather than choices. We therefore hypothesize that the discrepancy between memory patterns in Experiments 1 and 2 stem from differences in task demands, specifically with respect to making agentic choices. Please see our response to Essential Revision 2, where we speculate on how agentic and non-agentic choices may yield different memory patterns, a point we now raise in the discussion.</p><disp-quote content-type="editor-comment"><p>5. It was not clear at the conceptual level how the Pavlovian reinforcement learning model fit in experiment 2 is different from the main RSTD used in experiment 1, and/or why these paradigms required the use of different models. Additional description on this point would be helpful.</p></disp-quote><p>In Essential Revision 1, we detailed the differences between the task in Experiment 1, where participants made explicit choices on most trials, and that of Experiment 2, where participants made explicit outcome value predictions on each trial. Because the behavioral measures differed across the 2 experiments (one model was fit to choices while the other was fit to value estimates), we needed to specify slightly different models for the tasks.</p><disp-quote content-type="editor-comment"><p>6. I would appreciate more context for the recoverability results. In particular, the ability to recover which model generated the data for simulated participants (RSTD vs. TD) seemed fairly low. I understand the point about low-asymmetry participants generated by the RSTD model being more parsimoniously fit by the simpler TD model, so that direction of error does not bother me so much. However, I am puzzled by the 87% correct for those simulated participants generated by the TD model. This would have to mean, I think, that the simple TD behavior was being incorrectly attributed to the more complex two-alpha model? I was hoping the authors could provide additional context or reassurance that this qualifies as "good" performance.</p></disp-quote><p>We thank the reviewer for raising this concern. As the reviewer notes, while it is clear why a proportion of the simulated participants generated with the RSTD model were better fit by the TD model (i.e., those with low asymmetry), it was indeed somewhat puzzling why only 87% of those participants simulated with the more parsimonious TD model were better fit by the more complex two-alpha model. To better understand why this was the case, we closely examined the simulated and estimated parameters to see whether there was some systematic bias present in a subset of the TD-simulations that enabled the single learning-rate model to effectively reproduce choice behavior that would more commonly be generated by the RSTD model. We found no such systematic effects. Instead, these participants exhibited some small degree of risk-seeing or risk-averse choice bias and correspondingly, had a small AIC difference between the two models that, while essentially equivocal, favored the RSTD model. We took this to be potential evidence of overfitting. We reasoned that if the choice behavior generated by models in this task were indeed differentiable but the AIC metric simply did not apply a sufficient penalty for the additional free parameter, that adopting a more conservative metric (i.e., one that requires a greater improvement in fit in order to justify the addition of a parameter) would correct this poor model recoverability performance. Indeed, when we used the Bayesian Information Criterion metric instead, we found that the proportion of TD-simulated participants best fit by the TD model was now 98%. To correct for potential overfitting in our results, we have now moved to use this more conservative BIC metric in all of our model comparisons. Importantly, none of the conclusions of our model comparisons are affected by this change. We are grateful to the reviewer for pushing us to better understand and correct this issue.</p><disp-quote content-type="editor-comment"><p>7. There were some details not sufficiently described in the main paper for the reader to understand without referencing the methods or supplement. For example: How did the forced versus choice trials work? What is "test trial performance" – the test trials are not described until the supplement and it was confusing to me because the only "test" mentioned in the main paper at this point was the memory test. How were the probabilistic vs. deterministic choices incorporated into the mixed-effects modeling?</p></disp-quote><p>We thank the reviewer for raising this point. Please see our response to Essential Revision 6, where we describe revisions to the Results section. Specifically, we modified our description of the choice task within the main text to include the trial types (risk trials, test trials, forced trials) that participants encountered. We also modified our description of the mixed-effects modeling to clarify that whether a choice was probabilistic or deterministic was not included in the model. Rather, probabilistic and deterministic choices are reflected in the PE magnitude predictor variable, where, in general, more extreme PE magnitudes reflect outcomes of risky choices, and over time, deterministic choices do not elicit PEs at all.</p><disp-quote content-type="editor-comment"><p>8. How were the different responses on the memory test considered? There were four choices according to the figure but it is described as though it is simply "old" vs. "new" responses. Please clarify how this was handled in analysis as well as the reason for inclusion of these four options.</p></disp-quote><p>We appreciate this helpful comment. Our focus in the present manuscript was on memory accuracy rather than confidence, so we collapsed across confidence ratings (e.g., “Definitely old” and “Maybe old”), a convention widely adopted in manuscripts examining memory accuracy effects (e.g., Dunsmoor et al. 2015; Murty et al. 2016).</p><p>However, based on the reviewer’s suggestion, we now include an ordinal regression analysis in Appendix 1 that tests for an AI x PE Valence x PE Magnitude interaction but with participants’ uncollapsed responses (1 = Definitely New, 2 = Maybe New, 3 = Maybe Old, and 4 = Definitely Old).</p><p>Regression results are reported in Appendix 1—table 1. Importantly, the results from the ordinal regression were not meaningfully different from results collapsed across confidence ratings. In particular, the AI x PE Valence x PE Magnitude interaction was highly significant in the ordinal regression. The 3-way interaction is plotted in Appendix 1—figure 4<italic>.</italic> Here, probabilities of each memory response are plotted as a function of PE valence and magnitude separately for AI = -.8 (top panels), AI = 0 (middle panels) and AI = .8 (bottom panels). Consistent with the results from the model that collapsed across confidence, the likelihood of a “definitely old” response was highest for those in those with low AI for images that coincided with high-magnitude negative PEs (top left panel) and those with high AIs for images that coincided with high-magnitude positive PEs (bottom right panel).</p><p>We now include these ordinal regression results in Appendix 1 and briefly mention the analysis in the main text:</p><p>Results:</p><p>“We had no a priori hypothesis about how any effects of valence bias on memory might interact with participants’ confidence in their “old” and “new” judgments. […] Notably, neither linear (z = .30, p = .767, OR = 1.02, 95% CI [0.89, 1.17]) nor quadratic age (z = -0.15, p = .881, OR = 0.99, 95% CI [0.85, 1.16]) were significant predictors of memory, suggesting that AI parsimoniously accounted for individual differences in memory.</p><p>Appendix 1:</p><p>“Our multilevel models of memory data collapsed across confidence ratings (e.g., “Definitely old” and “Maybe old”), a convention widely adopted in manuscripts examining memory accuracy effects (e.g., Dunsmoor et al. 2015; Murty et al. 2016). […] Consistent with the results reported in the main text, the likelihood of a “definitely old” response was highest for those in those with low AI for images that coincided with high-magnitude negative PEs (top left panel) and those with high AIs for images that coincided with high-magnitude positive PEs (bottom right panel).”</p><disp-quote content-type="editor-comment"><p>9. I apologized if I missed this point in the paper: I understand that the models were fit to individual data and most participants were better fit by RSTD than TD. However, the authors also discuss the RSTD model winning overall and all subsequent analyses on the individual differences require the two separate α values. So, I am confused as to whether (a) all participants were ultimately fit with the RSTD so all could be included in the analysis, despite some being better fit by TD or (b) only participants who were best fit by RSTD were included in subsequent analyses (or of course (c) something else)? I think this would be OK because my assumption would be that the participants would simply have two α values (positive and negative) that would be similar if their behavior is better explained by the TD model, but I just wanted to clarify.</p></disp-quote><p>All participants’ choice data were fit to TD and RSTD (and now Utility and FourLR) models. Even when patients were best fit by the TD model, the learning rates from the RSTD model were used for subsequent analyses. As the reviewer suggested, participants better fit by the TD than the RSTD model had similar <italic>α</italic><sup>+</sup> and <italic>α</italic><sup>-</sup> values (i.e., their AI was close to 0). This pattern is demonstrated in Appendix 1—figure 6, which displays AI as a function of the relative BIC for RSTD vs. TD models.</p><disp-quote content-type="editor-comment"><p>Reviewer #2:</p><p>[…] First the behavioral results are presented in a very processed manner, limiting the ability to re-interpret the results.</p><p>Second, the computational modeling is fairly limited and lacking in competing accounts, which also limits the interpretation of the results. As such, it seems that at this point, the results do not fully support the conclusions – in particular, it is not clear that the interpretation in terms of asymmetric learning rates is warranted yet.</p><p>Comments for the authors:</p><p>1. Presentation of the results. All figures presented are quite removed from the raw data. It would be helpful to provide more unprocessed results – for example, the learning curves per experimental condition, and/or per age group. This would also provide a more sensitive target for model validation than the ones currently presented in Figure S4. It is much harder for the reader to interpret the results when only very processed data is shown.</p></disp-quote><p>We appreciate this helpful comment. As described in Essential Revision 3, we added several additional figures to Appendix 1, including learning curves by age group as suggested by the reviewer.</p><disp-quote content-type="editor-comment"><p>This is a well-written, easy to read article, that adds a new data point to the complex and contradictory literature on valence, risk, and adolescence, without resolving it. I see a number of issues with the presentation of the results, the modeling and interpretation of the results.</p><p>2. Modeling. The authors use two very simple RL models to capture the performance (a classic δ rule model, and the same with two learning rates). There are a few relevant aspects to the modeling that are either not considered or not reported, but that are important for interpreting the results.</p><p>a. Please indicate Q-value initialization and reward rescaling as part of the model description, rather than as part of the model fitting procedure. The initialization has theoretical impacts, so should be part of the model.</p></disp-quote><p>As suggested by the reviewer, we now indicate that rewards were rescaled to range from 0 to 1, and Q-values were initialized at.5 (equivalent to 40 points), in our description of the model.</p><disp-quote content-type="editor-comment"><p>b. Prospect theory indicates that a value of 80 is subjectively worth less that 2* the subjective value of 40. As such, the claim that "expected value" and "risk" are controlled for is debatable: if a true 40 feels like a 50 in comparison to an 80, then the two "same objective expected value stimuli" have different subjective expected values, without a need to invoke risk. The authors should test a competing model where the learning rate is fixed, but the obtained reward is modified according to prospect theory (i.e. subjective reward = 80*((reward/80)^p), where p is a free parameter). This model should also be able to capture the presented processed results (existence of an apparent "risk-aversion"), but should have slightly different temporal dynamics, and would lead to a completely different interpretation of the results.</p></disp-quote><p>We thank the reviewer for this valuable comment. As suggested, we ran a new model where subjective reward = 80*((reward/80)^p). We call this the Utility model because Prospect theory does not differ from the Utility model in the context of our task, which only includes positive outcomes. Briefly, we did not find clear differences in quantitative model fit between Utility and RSTD models. However, we did find that posterior predictions of choice data from the RSTD model were more accurate than from the Utility model, and that valence biases and PEs from the RSTD model could qualitatively explain subsequent memory patterns, while the Utility model could not. Please see our response to Essential Revision 1B for a description of our Utility model and related results.</p><disp-quote content-type="editor-comment"><p>c. Please report parameter β, age effects. Also provide more model validation.</p></disp-quote><p>Please see our response to Essential Revision 9 for age effects in the β parameter, along with additional model validation.</p><disp-quote content-type="editor-comment"><p>d. Have the author investigated the effect of forced vs. free choice trials, both on RL and memory? There is evidence that this leads to differential learning processes, and potentially differential attention, which could impact both the learning and memory findings.</p></disp-quote><p>We thank the reviewer for this helpful suggestion. In response to this comment, we investigated how agency in decision making may have influenced both RL and memory by testing whether learning and memory patterns varied for free compared to forced choices. A detailed discussion of these analyses can be found in Essential Revision 2. Briefly, in a model that included separate positive and negative learning rates for free and forced choices (i.e., the FourLR model) positive learning rates were numerically greater in free compared to forced choices, but average learning asymmetries for free choices were not positive. Additionally, individual differences in memory for items presented during forced vs. free choices qualitatively differed (although there was no significant effect of choice agency on memory performance), such that memory patterns for forced choices were similar to those observed in Experiment 2, where participants also did not make choices. We have also added to the discussion further consideration of the potential role of choice agency in this study.</p><disp-quote content-type="editor-comment"><p>3. Memory findings:</p><p>a. Can the authors comment on the role of attention? Presumably, the participants pay more attention to the outcome of probabilistic than deterministic choices. Could this be a factor in encoding the image, instead of the actual RPE strength? Is there some analysis in probabilistic trials that could be done to show convincingly that the actual size of the RPE matters? In the current analysis, it seems confounded with condition.</p></disp-quote><p>We thank the reviewer for this helpful comment. We agree that attention likely plays a mechanistic role in our observed findings, and now include a discussion of the role of attention in the manuscript. Additionally, we found evidence that PE strength matters within the probabilistic trials — that is, our PE valence x PE magnitude x AI interaction remained significant when we removed images that were presented alongside deterministic outcomes. Please see our response to Essential Revision 4 for more information.</p><disp-quote content-type="editor-comment"><p>b. While the overall statistical pattern is replicated (Figure 4 and 6A), the realization of the triple interaction looks different in the two experiments (Figure 4 and 6B). In the replication, the asymmetry seems to not matter for positive RPEs, and to have a stronger effect for negative RPEs. For the new study, the patterns seem symmetrical between positive and negative RPEs. Can the authors comment on this?</p></disp-quote><p>We now explicitly address the statistical patterns observed in Experiments 1 and 2, and include analyses that aid in interpreting the differences in Appendix 1. Specifically, the pattern observed in Experiment 2, where participants made predictions about stimuli rather than choices, mirrors the pattern seen within forced trials in Experiment 1. Therefore, the difference in these patterns of memory performance may relate to the differential degree of choice agency across both trial types (free vs. forced) and experiments. Please see Essential Revisions 2 and 5 for additional information.</p><disp-quote content-type="editor-comment"><p>Reviewer #3:</p><p>[…] This paper presents data from a cross-sectional sample. This raises questions as to whether learning asymmetries are a stable individual characteristic, or whether these biases exhibit within-person changes with age. Nonetheless, the results of this paper provide important advances to our understanding of age-related differences in learning, decision making, and memory that can form the basis of future longitudinal studies.</p></disp-quote><p>We thank the reviewer for their helpful comments. We now note the weaknesses related to our cross-sectional sample in the Discussion section, and speculate that the biases shift within individuals across age. Please see our response to Essential Revision 7 for a detailed description of these revisions.</p><disp-quote content-type="editor-comment"><p>Comments for the authors:</p><p>This manuscript is exceptionally well-written, and the authors present the methods and findings very clearly. I commend the authors approach to computational modeling and nonlinear age analyses. The present findings provide exciting and novel insights about how adolescents approach risky decision making, which in turn has consequences for memory formation. This is a strong paper, and I believe that the current conclusions warrant publication in eLife. However, I also believe that the inclusion of some additional analyses and clarifications, which I offer below, will further strengthen this manuscript.</p><p>In the introduction, the authors explain how prior research in developmental samples cannot disentangle learning asymmetries because performance in prior tasks improved if individuals relied upon updating from positive prediction errors. To clarify this point, and to emphasize the novelty of the current study, it would be helpful if the authors provide a more detailed explanation as to how the present design differs from the paradigm used in Van den Bos 2012.</p></disp-quote><p>We thank the reviewer for this valuable comment. As suggested, we included a description of the key differences between our paradigm and that employed in Van den Bos et al., (2012). Please see Essential Revision 8 for our revised text.</p><disp-quote content-type="editor-comment"><p>In the present paper, the authors run model comparison for a basic TD model and a risk-sensitive reinforcement learning model. However, risk sensitivity may be influenced by nonlinear utility functions. It would be informative if the authors also discussed the utility model, as presented in the Niv 2012 paper, which first introduced the present behavioral paradigm. If the authors did not fit this model, please provide an explanation as to why this model was not tested in the current sample.</p></disp-quote><p>As suggested by the reviewer, we tested a Utility model as an alternative to the TD and RSTD models. Briefly, we did not find clear differences in quantitative model fit between Utility and RSTD models. However, we did find that posterior predictions of choice data from the RSTD model were more accurate than from the Utility model, and that valence biases and PEs from the RSTD model could qualitatively explain subsequent memory patterns, while the Utility model could not. Please see our response to Essential Revision 1B for a description of our Utility model and related results.</p><disp-quote content-type="editor-comment"><p>Prior work from this group has shown that choice and agency can influence memory (e.g. Katzman & Hartley 2020). Therefore, for the memory data, it would be helpful if the authors accounted for the different trial types (choice trials vs. forced trials) in the analyses.</p></disp-quote><p>We thank the reviewer for this comment. We now include analyses of memory as a function of choice agency. Individual differences in memory for items presented during forced vs. free choices qualitatively differed (although there was no significant effect of choice agency on memory performance), such that memory patterns for forced choices were similar to those observed in Experiment 2, where participants also did not make choices. We have also added to the discussion further consideration of the potential role of choice agency in this study. Please see Essential Revision 2 for additional details.</p><disp-quote content-type="editor-comment"><p>Due to the cross-sectional nature of the sample, it is unclear if asymmetry biases are a trait-stable characteristic, or whether this bias changes with age within an individual. It woul be helpful for the authors to address this in the discussion.</p></disp-quote><p>We agree that our cross-sectional design is a key limitation of our study. We now acknowledge this limitation in the discussion and suggest that future longitudinal studies examine how valence biases in learning and memory shift within an individual with age (please see Essential Revision 7 for more details).</p></body></sub-article></article>