A TSI Community-Consensus Composite Greg Kopp, Kim Kokkonen, and Brent Dagdagan LASP / Univ. of Colorado SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 1 Traditional TSI Composites SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 2 Issues with Traditional Composites • Created by individuals (PIs) • Binary (and biased) selection of instrument data used – Discontinuities at boundaries • Controversial corrections applied to data records • Normalizations incorrect • Lack uncertainties SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 3 Improvements in “Community-Consensus” Composite • Recent improvements to absolute accuracy in the newer TSI measurements are incorporated – SORCE, PREMOS, TCTE, and TSIS help transfer improved ground-based calibrations to space • Weight data from all available instruments • Use unbiased statistically-driven approach rather than favored instrument • Include time-dependent uncertainties to indicate temporal regions where contributing data may be suspect • Smooth transitions and gaps scale-wise SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 4 Latest “Community Consensus” TSI Composite Includes efforts of former ISSI team and current SIST team SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 5 ISSI Team Laid Groundwork 1. Agreed upon the absolute value to use for the composite TSI record 2. Agreed upon an unbiased computational methodology to create this new composite Team: Greg Kopp (PI), Will Ball, Steven Dewitte, Thierry Dudok de Wit, André Fehlmann, Wolfgang Finsterle, Claus Fröhlich, Sabri Mekaoui, Werner Schmutz, Richard Willson, Pia Zacharias SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 6 SIST Effort 1. Demonstrate, implement, and improve the computational methodology to create a new community-consensus TSI composite including time-dependent uncertainties with (partial) continued involvement from ISSI team 2. Distribute the composite to public and produce a publication detailing the methodology 3. Establish a system to update this TSI composite regularly as new data are available Summary: Provide data users with a single TSI composite including, for the first time, timedependent uncertainties, a non-binary selection of contributing instruments, and an unbiased weighting of those instruments SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 7 SIST Team Collaborators SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 8 Specific Results from SIST Effort • A single TSI composite having daily values over space-borne measurement era with consensus from experts of the TSI instruments and data – Current composites are from individual researchers, not groups representing all instruments, so show bias in selection of instrument data • Time-dependent uncertainties for values in the composite – Current composites do not include uncertainties (let alone time-dependent ones) • Consolidated estimates of time-dependent uncertainties in the current and historical individual TSI instrument records – Proposed approach provides a relatively unbiased assessment of all data records • Establish computational algorithms to enable regular updates as new data and new instruments become available • Creation of a website providing the resulting composite to users • A publication describing the composite and the inputs SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 9 TSI-Composite Methodology Has Been Published • TSI-community based for openness • Uses all available instrument data • Scale-wise weightings smoothly fill gaps • Uses an unbiased statistical approach • Normalized to most accurate instruments • Has time-dependent uncertainties Geophysical Research Letters RESEARCH LETTER 10.1002/2016GL071866 Key Points: • We present a new approach for merging different solar irradiance time series into a single composite • We provide a new and fully traceable composite of the total solar irradiance • We quantify uncertainties in the total solar irradiance composite and demonstrate a 1∕f scaling in them Supporting Information: • Supporting Information S1 Correspondence to: T. Dudok de Wit, ddwit@cnrs-orleans.fr Citation: Dudok de Wit, T., G. Kopp, C. Fröhlich, and M. Schöll (2017), Methodology to create a new total solar irradiance record: Making a composite out of multiple data records, Geophys. Res. Lett., 44, doi:10.1002/2016GL071866. Received 7 NOV 2016 Accepted 24 JAN 2017 Accepted article online 27 JAN 2017 Methodology to create a new total solar irradiance record: Making a composite out of multiple data records Thierry Dudok de Wit1 , Greg Kopp2,3 , Claus Fröhlich4 , and Micha Schöll1,5 1 LPC2E, CNRS and University of Orléans, Orléans, France, 2 Laboratory for Atmospheric and Space Physics, University of Colorado Boulder, Boulder, Colorado, USA, 3 Max-Planck-Institut für Sonnensystemforschung, Göttingen, Germany, 4 Dählenwaldstrasse 30, Davos Wolfgang, Switzerland, 5 Physikalisch Meteorologisches Observatorium Davos and World Radiation Center, Davos Dorf, Switzerland Abstract Many observational records critically rely on our ability to merge different (and not necessarily overlapping) observations into a single composite. We provide a novel and fully traceable approach for doing so, which relies on a multiscale maximum likelihood estimator. This approach overcomes the problem of data gaps in a natural way and uses data-driven estimates of the uncertainties. We apply it to the total solar irradiance (TSI) composite, which is currently being revised and is critical to our understanding of solar radiative forcing. While the final composite is pending decisions on what corrections to apply to the original observations, we find that the new composite is in closest agreement with the PMOD composite and the NRLTSI2 model. In addition, we evaluate long-term uncertainties in the TSI, which reveal a 1∕f scaling. 1. Introduction Combining different (and only partly overlapping) time series of the same physical quantity into a single composite is both a scientific and a statistical challenge that arises in many contexts, in particular in paleoclimatic reconstructions [Mann et al., 2008]. In space sciences, observations are often constrained by the finite lifetimes of satellites, making composites the key to investigation over long timescales. A timely and demanding application is the reconstruction of the total solar irradiance (TSI), which is the spatially and spectrally integrated radiant output from the Sun at a mean Sun-Earth distance of 1 AU [Kopp, 2014]. The TSI has been continuously measured since November 1978 by over a dozen instruments and is paramount to understanding the Earth’s global energy budget [Trenberth et al., 2009]. Weak secular variations in the TSI are hotly debated, as they may have large implications on our understanding of the role of the Sun in climate change [Ermolli et al., 2013]. SIST Meeting Lanham, MD 8 May 2018 TSI Composite The nominal value of the TSI, averaged over Solar Cycle 23 (which lasted from 1996 to 2008), is 1361.0 ± 0.5 p. 10 (W/m2 ), with a weak peak-to-peak solar cycle modulation of 0.08% that is in phaseGreg with Kopp the 11-year cycle TSI-Composite Methodology Demonstrated • TSI composite improved with reduced biases and better instrument-transition overlaps – Methodology demonstrated, but final composite needs refining • Agree on amount of “early increase” correction (if any) to apply • Estimate initial uncertainties TSI instrument and composite data are available at: • Update regularly http://spot.colorado.edu/~koppg/TSI SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 11 New TSI-Composite Methodology • Estimate noise for each instrument based on high-frequency daily values – Predictive-model noise-estimating method agrees well with independent results from Kopp, SWSC, 2014 • Apply wavelet transform for scale-wise analysis • Compute weighted average of all instrument data scale-wise based on frequencydependent noise model – Extrapolate uncertainties scale-wise based on 1/f noise model – Surrounding values smooth gaps and discontinuities scale-wise • Invert net wavelet transform • Estimate uncertainties – Monte Carlo using 1/f noise model SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 12 Methodology – Graphical Instrument Std. Dev. R ACRIM1 NIMBUS7 ACRIM2 ACRIM3 SORCE PREMOS TCTE 0.1139 0.1391 0.3491 0.0326 0.0290 0.0326 0.0591 0.9705 0.9849 0.8560 0.9983 0.9977 0.9953 0.9902 NIMBUS7 ACRIM1 ACRIM2 VIRGO ACRIM3 SORCE PREMOS TCTE 0.1099 0.2286 0.2618 0.0412 0.0324 0.0286 0.0326 0.0536 0.9831 0.9254 0.9054 0.9966 0.9983 0.9977 0.9954 0.9918 SIST Meeting Lanham, MD 8 May 2018 Direct Reference NIMBUS7 ACRIM2 VIRGO VIRGO VIRGO VIRGO VIRGO Std. Dev. R 0.1078 0.1439 0.0348 0.0424 0.0410 0.0474 0.0416 0.9727 0.9816 0.9975 0.9967 0.9946 0.9901 0.9951 TSI Composite Overlap Begin Date 16-Feb-1980 5-Oct-1991 28-Jan-1996 5-Apr-2000 25-Feb-2003 27-Jul-2010 16-Dec-2013 Overlap Ratio of Std. Dev. to End Date Std. Devs. Common 14-Jul-1989 1.0561 0.4214 12-Jan-1993 0.9666 0.3991 1-Mar-2001 10.0277 0.4128 17-Sep-2013 0.7694 0.0317 20-Sep-2017 0.7058 0.0291 11-Feb-2014 0.6885 0.0283 20-Sep-2017 1.4201 0.0585 16-Nov-1978 16-Feb-1980 5-Oct-1991 28-Jan-1996 5-Apr-2000 25-Feb-2003 27-Jul-2010 13-Dec-2013 24-Jan-1993 14-Jul-1989 1-Mar-2001 20-Sep-2017 common 17-Sep-2013 2-Apr-2018 10-Feb-2014 2-Apr-2018 0.0412 Greg Kopp - p. 13 Measurement Differences Show 1/f Power Scaling • Dispersion is not indicative of linear trends or of white noise • Use as noise model of each instrument for scale-dependent weightings based on highfrequency predictive-model correlations SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 14 Wedge Trends in Differences Are Misleading • Linear trends in instrument differences are not what is observed • Linearly-increasing uncertainties overestimate actual uncertainties in time (eventually) SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 15 Comparisons During Solar Minimum Indicate Instrument Noise SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 16 Predictive Model Used to Estimate High-Frequency Uncertainties • Based on predictive model using daily values – Is time dependent because of solar variability SIST Meeting Lanham, MD 8 May 2018 "! # = % '& " #& ≠ # & TSI Composite Greg Kopp - p. 17 Predictive Model Used to Estimate High-Frequency Uncertainties Instrument Std. Dev. R ACRIM1 NIMBUS7 ACRIM2 ACRIM3 SORCE PREMOS TCTE 0.1139 0.1391 0.3491 0.0326 0.0290 0.0326 0.0591 0.9705 0.9849 0.8560 0.9983 0.9977 0.9953 0.9902 NIMBUS7 ACRIM1 ACRIM2 VIRGO ACRIM3 SORCE PREMOS TCTE 0.1099 0.2286 0.2618 0.0412 0.0324 0.0286 0.0326 0.0536 0.9831 0.9254 0.9054 0.9966 0.9983 0.9977 0.9954 0.9918 SIST Meeting Lanham, MD 8 May 2018 Direct Reference NIMBUS7 ACRIM2 VIRGO VIRGO VIRGO VIRGO VIRGO Std. Dev. R 0.1078 0.1439 0.0348 0.0424 0.0410 0.0474 0.0416 0.9727 0.9816 0.9975 0.9967 0.9946 0.9901 0.9951 TSI Composite Overlap Begin Date 16-Feb-1980 5-Oct-1991 28-Jan-1996 5-Apr-2000 25-Feb-2003 27-Jul-2010 16-Dec-2013 Overlap Ratio of Std. Dev. to End Date Std. Devs. Common 14-Jul-1989 1.0561 0.4214 12-Jan-1993 0.9666 0.3991 1-Mar-2001 10.0277 0.4128 17-Sep-2013 0.7694 0.0317 20-Sep-2017 0.7058 0.0291 11-Feb-2014 0.6885 0.0283 20-Sep-2017 1.4201 0.0585 16-Nov-1978 16-Feb-1980 5-Oct-1991 28-Jan-1996 5-Apr-2000 25-Feb-2003 27-Jul-2010 13-Dec-2013 24-Jan-1993 14-Jul-1989 1-Mar-2001 20-Sep-2017 common 17-Sep-2013 2-Apr-2018 10-Feb-2014 2-Apr-2018 0.0412 Greg Kopp - p. 18 Relative Uncertainties Between Times Need to Be Expressed in 2D SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 19 Composite Plot and Uncertainties in 2D SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 20 LASP Has Methodology in Place for Continued Updates SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 21 Absolute Value Determined at Solar Minimum • Determined absolute value based on latest measurements – Used data from ACRIM3, PREMOS, TIM, VIRGO (incl. DIARAD) – Selected temporal region of overlap • 2008 solar minimum – Computed mean over region weighted by estimated instrument uncertainties • Normalize composite to resulting 1360.54 W/m2 – Using solar minimum period from 20 Sept. 2008 through 5 May 2009 • Include TCTE/TIM via comparisons similar to those for PREMOS (needs updating for latest data) SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 22 Issues with Originally-Planned Approach for Absolute Value • “Latest” measurements vary with time – ACRIM3, PREMOS, SORCE/TIM, and VIRGO will not always be the most current measurements • They already aren’t – Need to accommodate newer instruments as available • 2008 solar-minimum temporal region of overlap isn’t measured by newer instruments • Does not use all available instruments Agreed to use “Day 1” mean with weightings by estimated instrument uncertainties Normalize composite to resulting weighted mean SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 23 Adjust to Absolute Value Based on “Day 1” Measurements • Use data from all instruments – Weighted average of “Day 1” values – Average first few measurements SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 24 VIRGO and PREMOS Had Large “Early Increases” 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 Total Solar Irradiance (Wm−2) 1370 Note the “early increase” in VIRGO • Both the VIRGO and the PREMOS had ~600 ppm increase at very beginning of mission 1368 1366 1364 1362 Version 6_005_1611 1360 PREMOS “early increase” 2 DIARAD−R − 4.5 Wm DIARAD−L PMO6V−A − 0.3 Wm22 PMO6V−B − 1.7 Wm 1358 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 VIRGO Level 1 Data (all 4 channels) >4000-ppm correction needed SIST Meeting Lanham, MD 8 May 2018 TSI Composite PREMOS degradation fitted by linear function initially then by 8th order polynomial >4000-ppm correction needed Greg Kopp - p. 25 Current TSI-Measurement Record An “early increase”? SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 26 Decided to Use “Early Increase” Corrections data courtesy of Claus Fröhlich SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 27 Data for Composite Must be Publicly Available • Transparency to community is important for acceptance SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 28 “Community Consensus” TSI Composite • “Community consensus” composite refinements – – – – TSI teams recently agreed on using data corrected for early increases Agreed to scale to weighted “Day 1” absolute value of all instruments Refined initial uncertainties LASP is prepared to update regularly SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 29 Comparisons of Composites SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 30 Comparisons of Models and Community-Consensus Composite SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 31 Future Efforts • Improvements to composite itself – – – – – Modify initial weightings based on known instrument artifacts Refine uncertainties for early instruments relative to later ones Consider appropriateness of applying 1/f spectral variation to all instruments Improve method of adding/losing instruments ”Sanity check” • Provide regular updates as new data or instruments become available • Publish and serve resulting composite to research community SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 32 Future Improvements / Ideas • Predictive Model: Instead of applying from only the preceding points, apply to surrounding points • Noise Determination: Use SVD of high-frequency components to determine common modes between two or more instruments. These are presumably solar in origin. The remainder is instrument noise. • Data gaps: Look for common-mode between instruments from SVD. Time-extend using another instrument’s wavelets for times when desired instrument lacks data. • Consider ERBS: Is 1/f noise model appropriate for this instrument? SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 33 And Then What Else Is Needed? • Models to extend to historical times are getting more sophisticated – But downward trend of SATIRE relative to measurements and NRLTSI in recent decades is concerning – CMIP6 clarity on model versions used • The sunspot-number reconstruction may make this all irrelevant... SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 34 New Sunspot-Number Reconstruction(s) • Community reanalysis of sunspot-number records lead to new series – Cle6e & Lefèvre, “The New Sunspot Number: Assembling All CorrecDons,” Solar Physics, 291, 2016 SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 35 New Sunspot-Group-Number Reconstruction(s) • Community reanalysis of sunspot-number records leads to new series – Svalgaard & Schatten, “Reconstruction of the Sunspot Group Number: The Backbone Method,” Solar Physics, 291, 2016 SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 36 Sensitivity of TSI Models to Sunspot Record(s) Kopp, G., Krivova, N., Lean, J., and Wu, C.J., “The Impact of the Revised Sunspot Record on Solar Irradiance Reconstructions,” Solar Physics, 2016, doi: 10.1007/s11207-016-0853-x SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 37 “TSI Reconstruc?ons Based on Updated TSI Composite and Sunspot Records” • Update the 400-year sunspot record used in TSI models for historical reconstructions • Re-compute flux-transport results to improve historical solar-variability estimates • Improve the TSI-measurement composite, providing a reference for TSI models Team Member Expertise Odele Coddington NRLTSI modeler; historical solar-irradiance extensions Thierry Dudok de Wit TSI-composite methodology creator; ISSI sunspot-team member Greg Kopp TSI instrument scientist; TSI-composite team leader; ISSI sunspot-team member Natalie Krivova SATIRE modeler; historical solar-irradiance extensions Judith Lean NRLTSI modeler; historical solar-irradiance extensions Lisa Upton Flux-transport and dynamo modeler Chi-Ju Wu SATIRE modeler; historical solar-irradiance extensions SIST Meeting Lanham, MD 8 May 2018 TSI Composite Greg Kopp - p. 38