data assimilation

{{short description|Technique for combining information from a computer model with information from observations}}

{{Multiple issues|

{{Confusing|reason=complicated jargon is used throughout, lacking explanation|date=October 2013}}

}}

Data assimilation refers to a large group of methods that update information from numerical computer models with information from observations. Data assimilation is used to update model states, model trajectories over time, model parameters, and combinations thereof. What distinguishes data assimilation from other estimation methods is that the computer model is a dynamical model, i.e. the model describes how model variables change over time, and its firm mathematical foundation in Bayesian Inference. As such, it generalizes inverse methods and has close connections with machine learning.

Data assimilation initially developed in the field of numerical weather prediction. Numerical weather prediction models are equations describing the evolution of the atmosphere, typically coded into a computer program. When these models are used for forecasting the model output quickly deviates from the real atmosphere. Hence, we use observations of the atmosphere to keep the model on track. Data assimilation provides a very large number of practical ways to bring these observations into the models.

Simply inserting point-wise measurements into the numerical models did not provide a satisfactory solution. Real world measurements contain errors both due to the quality of the instrument and how accurately the position of the measurement is known. These errors can cause instabilities in the models that eliminate any level of skill in a forecast. Thus, more sophisticated methods were needed in order to initialize a model using all available data while making sure to maintain stability in the numerical model. Such data typically includes the measurements as well as a previous forecast valid at the same time the measurements are made. If applied iteratively, this process begins to accumulate information from past observations into all subsequent forecasts.

Because data assimilation developed out of the field of numerical weather prediction, it initially gained popularity amongst the geosciences. In fact, one of the most cited publication in all of the geosciences is an application of data assimilation to reconstruct the observed history of the atmosphere.{{cite journal |doi=10.1175/1520-0477(1996)077<0437:TNYRP>2.0.CO;2 |title=The NCEP/NCAR 40-Year Reanalysis Project |journal= Bulletin of the American Meteorological Society|volume=77 |issue=March |pages=437–471 |year=1996 |last1=Kalnay |first1=Eugenia |last2=and coauthors |s2cid=124135431 |bibcode=1996BAMS...77..437K |issn=1520-0477 |doi-access=free }}

Details of the data assimilation process

Classically, data assimilation has been applied to chaotic dynamical systems that are too difficult to predict using simple extrapolation methods. The cause of this difficulty is that small changes in initial conditions can lead to large changes in prediction accuracy. This is sometimes known as the butterfly effect – the sensitive dependence on initial conditions in which a small change in one state of a deterministic nonlinear system can result in large differences in a later state.

At any update time, data assimilation usually takes a forecast (also known as the first guess, or background information) and applies a correction to the forecast based on a set of observed data and estimated errors that are present in both the observations and the forecast itself. The difference between the forecast and the observations at that time is called the departure or the innovation (as it provides new information to the data assimilation process). A weighting factor is applied to the innovation to determine how much of a correction should be made to the forecast based on the new information from the observations. The best estimate of the state of the system based on the correction to the forecast determined by a weighting factor times the innovation is called the analysis. In one dimension, computing the analysis could be as simple as forming a weighted average of a forecasted and observed value. In multiple dimensions the problem becomes more difficult. Much of the work in data assimilation is focused on adequately estimating the appropriate weighting factor based on intricate knowledge of the errors in the system.

The measurements are usually made of a real-world system, rather than of the model's incomplete representation of that system, and so a special function called the observation operator (usually depicted by h() for a nonlinear operator or "H" for its linearization) is needed to map the modeled variable to a form that can be directly compared with the observation.

Data assimilation as statistical estimation

One of the common mathematical philosophical perspectives is to view data assimilation as a Bayesian estimation problem. From this perspective, the analysis step is an application of Bayes' theorem and the overall assimilation procedure is an example of recursive Bayesian estimation. However, the probabilistic analysis is usually simplified to a computationally feasible form. Advancing the probability distribution in time would be done exactly in the general case by the Fokker–Planck equation, but that is not feasible for high-dimensional systems; so, various approximations operating on simplified representations of the probability distributions are used instead. Often the probability distributions are assumed Gaussian so that they can be represented by their mean and covariance, which gives rise to the Kalman filter.

Many methods represent the probability distributions only by the mean and input some pre-calculated covariance. An example of a direct (or sequential) method to compute this is called optimal statistical interpolation, or simply optimal interpolation (OI). An alternative approach is to iteratively solve a cost function that solves an identical problem. These are called "variational methods", such as 3D-Var and 4D-Var. Typical minimization algorithms are the conjugate gradient method or the generalized minimal residual method. The ensemble Kalman filter is sequential method that uses a Monte Carlo approach to estimate both the mean and the covariance of a Gaussian probability distribution by an ensemble of simulations. More recently, hybrid combinations of ensemble approaches and variational methods have become more popular (e.g. they are used for operational forecasts both at the European Centre for Medium-Range Weather Forecasts (ECMWF) and at the NOAA National Centers for Environmental Prediction (NCEP).

Data assimilation as a model update

Data assimilation can also be achieved within a model update loop, where we will iterate an initial model (or initial guess) in an optimisation loop to constrain the model to the observed data. Many optimisation approaches exist and all of them can be set up to update the model, for instance, evolutionary algorithm have proven to be efficient as free of hypothesis, but computationally expensive.

Weather forecasting applications

In numerical weather prediction applications, data assimilation is most widely known as a method for combining observations of meteorological variables such as temperature and atmospheric pressure with prior forecasts in order to initialize numerical forecast models.

=Necessity=

The atmosphere is a fluid. The idea of numerical weather prediction is to sample the state of the fluid at a given time and use the equations of fluid dynamics and thermodynamics to estimate the state of the fluid at some time in the future. The process of entering observation data into the model to generate initial conditions is called initialization. On land, terrain maps available at resolutions down to {{convert|1|km|mi|1|sp=us}} globally are used to help model atmospheric circulations within regions of rugged topography, in order to better depict features such as downslope winds, mountain waves and related cloudiness that affects incoming solar radiation.{{cite book|url=https://books.google.com/books?id=lMXSpRwKNO8C&pg=PA56|title=Parameterization schemes: keys to understanding numerical weather prediction models|author=Stensrud, David J.|page=56|year=2007|publisher=Cambridge University Press|isbn=978-0-521-86540-1}} The main inputs from country-based weather services are observations from devices (called radiosondes) in weather balloons that measure various atmospheric parameters and transmits them to a fixed receiver, as well as from weather satellites. The World Meteorological Organization acts to standardize the instrumentation, observing practices and timing of these observations worldwide. Stations either report hourly in METAR reports,{{cite web|title=Key to METAR Surface Weather Observations|url=http://www.ncdc.noaa.gov/oa/climate/conversion/swometardecoder.html|publisher=National Oceanic and Atmospheric Administration|access-date=2011-02-11|author=National Climatic Data Center|date=2008-08-20|author-link=National Climatic Data Center}} or every six hours in SYNOP reports.{{cite web|title=SYNOP Data Format (FM-12): Surface Synoptic Observations|publisher=UNISYS|archive-url=https://web.archive.org/web/20071230100059/http://weather.unisys.com/wxp/Appendices/Formats/SYNOP.html|archive-date=2007-12-30|date=2008-05-25|url=http://weather.unisys.com/wxp/Appendices/Formats/SYNOP.html}} These observations are irregularly spaced, so they are processed by data assimilation and objective analysis methods, which perform quality control and obtain values at locations usable by the model's mathematical algorithms.{{cite journal |doi=10.1146/annurev.fl.27.010195.001211 |title=Numerical Weather Prediction |journal=Annual Review of Fluid Mechanics |volume=27 |pages=195–225 |year=1995 |last1=Krishnamurti |first1=T N |bibcode=1995AnRFM..27..195K |s2cid=122230747 }} Some global models use finite differences, in which the world is represented as discrete points on a regularly spaced grid of latitude and longitude;{{cite book|chapter-url=https://books.google.com/books?id=BQ_7vh5SrHQC&pg=PA223 |pages=223–30 |last1=Chaudhari |first1=H. S. |first2=K. M. |last2=Lee |first3=J. H. |last3=Oh |chapter=Weather prediction and computational aspects of icosahedral-hexagonal gridpoint model GME |title=Parallel computational fluid dynamics: parallel computings and its applications : proceedings of the Parallel CFD 2006 Conference, Busan city, Korea (May 15–18, 2006) |editor1-first=Jang-Hyuk |editor1-last=Kwon |editor2-first=Jacques |editor2-last=Periaux |editor3-first=Pat |editor3-last=Fox |editor4-first=N. |editor4-last=Satofuka |editor5-first=A. |editor5-last=Ecer |year=2007 |access-date=2011-01-06|publisher=Elsevier|isbn=978-0-444-53035-6}} other models use spectral methods that solve for a range of wavelengths. The data are then used in the model as the starting point for a forecast.{{cite web|title=The WRF Variational Data Assimilation System (WRF-Var)|publisher=University Corporation for Atmospheric Research|archive-url=https://web.archive.org/web/20070814044336/http://www.mmm.ucar.edu/wrf/WG4/wrfvar/wrfvar-tutorial.htm|archive-date=2007-08-14|date=2007-08-14|url=http://www.mmm.ucar.edu/wrf/WG4/wrfvar/wrfvar-tutorial.htm}}

A variety of methods are used to gather observational data for use in numerical models. Sites launch radiosondes in weather balloons which rise through the troposphere and well into the stratosphere.{{cite web|last=Gaffen|first=Dian J.|title=Radiosonde Observations and Their Use in SPARC-Related Investigations|archive-url=https://web.archive.org/web/20070607142822/http://www.aero.jussieu.fr/~sparc/News12/Radiosondes.html|archive-date=2007-06-07|date=2007-06-07|url=http://www.aero.jussieu.fr/~sparc/News12/Radiosondes.html}} Information from weather satellites is used where traditional data sources are not available. Commerce provides pilot reports along aircraft routes{{cite journal |doi=10.1175/2008BAMS2332.1 |title=Systematic Differences in Aircraft and Radiosonde Temperatures |journal=Bulletin of the American Meteorological Society |volume=89 |issue=11 |page=1689 |year=2008 |last1=Ballish |first1=Bradley A |last2=Kumar |first2=V. Krishna |bibcode=2008BAMS...89.1689B |doi-access=free }} and ship reports along shipping routes.{{cite web|author=National Data Buoy Center|url=http://www.vos.noaa.gov/vos_scheme.shtml|title=The WMO Voluntary Observing Ships (VOS) Scheme|access-date=2011-02-15|date=2009-01-28|publisher=National Oceanic and Atmospheric Administration}} Research projects use reconnaissance aircraft to fly in and around weather systems of interest, such as tropical cyclones.{{cite web|year=2011|author=403rd Wing|url=http://www.hurricanehunters.com|title=The Hurricane Hunters|publisher=53rd Weather Reconnaissance Squadron|access-date=2006-03-30}}{{cite news|author=Lee, Christopher|title=Drone, Sensors May Open Path Into Eye of Storm|url=https://www.washingtonpost.com/wp-dyn/content/article/2007/10/07/AR2007100700971_pf.html|newspaper=The Washington Post|access-date=2008-02-22|date=2007-10-08}} Reconnaissance aircraft are also flown over the open oceans during the cold season into systems which cause significant uncertainty in forecast guidance, or are expected to be of high impact from three to seven days into the future over the downstream continent.{{cite web|url=http://www.noaanews.noaa.gov/stories2010/20100112_plane.html|title=NOAA Dispatches High-Tech Research Plane to Improve Winter Storm Forecasts|date=2010-11-12|access-date=2010-12-22|author=National Oceanic and Atmospheric Administration|author-link=National Oceanic and Atmospheric Administration}} Sea ice began to be initialized in forecast models in 1971.{{cite book|url=https://books.google.com/books?id=lMXSpRwKNO8C&pg=PA137|author=Stensrud, David J.|page=137|title=Parameterization schemes: keys to understanding numerical weather prediction models|publisher=Cambridge University Press|year=2007|isbn=978-0-521-86540-1}} Efforts to involve sea surface temperature in model initialization began in 1972 due to its role in modulating weather in higher latitudes of the Pacific.{{cite book|url=https://books.google.com/books?id=SV04AAAAIAAJ&pg=PA38|pages=49–50|title=The Global Climate|author=Houghton, John Theodore|publisher=Cambridge University Press archive|year=1985|isbn=978-0-521-31256-1}}

=History=

Image:Lewis Fry Richardson.png

In 1922, Lewis Fry Richardson published the first attempt at forecasting the weather numerically. Using a hydrostatic variation of Bjerknes's primitive equations,{{cite journal |doi=10.1016/j.jcp.2007.02.034 |title=The origins of computer weather prediction and climate modeling |journal=Journal of Computational Physics |volume=227 |issue=7 |pages=3431–3444 |year=2008 |last1=Lynch |first1=Peter |bibcode=2008JCoPh.227.3431L |url=https://zenodo.org/record/1326528 }} Richardson produced by hand a 6-hour forecast for the state of the atmosphere over two points in central Europe, taking at least six weeks to do so.{{cite book|last=Lynch|first=Peter|title=The Emergence of Numerical Weather Prediction|year=2006|publisher=Cambridge University Press|isbn=978-0-521-85729-1|pages=1–27|chapter=Weather Prediction by Numerical Process}} His forecast calculated that the change in surface pressure would be {{convert|145|mbar|inHg|lk=on}}, an unrealistic value incorrect by two orders of magnitude. The large error was caused by an imbalance in the pressure and wind velocity fields used as the initial conditions in his analysis, indicating the need for a data assimilation scheme.

Originally "subjective analysis" had been used in which numerical weather prediction (NWP) forecasts had been adjusted by meteorologists using their operational expertise. Then "objective analysis" (e.g. Cressman algorithm) was introduced for automated data assimilation. These objective methods used simple interpolation approaches, and thus{{why|date=April 2021}} were 3DDA (three-dimensional data assimilation) methods.

Later, 4DDA (four-dimensional data assimilation) methods, called "nudging", were developed, such as in the MM5 model. They are based on the simple idea of Newtonian relaxation (the 2nd axiom of Newton). They introduce into the right part of dynamical equations of the model a term that is proportional to the difference of the calculated meteorological variable and the observed value. This term that has a negative sign keeps the calculated state vector closer to the observations. Nudging can be interpreted as a variant of the Kalman-Bucy filter (a continuous time version of the Kalman filter) with the gain matrix prescribed rather than obtained from covariances.{{citation needed|date=January 2018}}

A major development was achieved by L. Gandin (1963) who introduced the "statistical interpolation" (or "optimal interpolation") method, which developed earlier ideas of Kolmogorov. This is a 3DDA method and is a type of regression analysis which utilizes information about the spatial distributions of covariance functions of the errors of the "first guess" field (previous forecast) and "true field". These functions are never known. However, the different approximations were assumed.{{citation needed|date=January 2018}}

The optimal interpolation algorithm is the reduced version of the Kalman filtering (KF) algorithm and in which the covariance matrices are not calculated from the dynamical equations but are pre-determined in advance.

Attempts to introduce the KF algorithms as a 4DDA tool for NWP models came later. However, this was (and remains) a difficult task because the full version requires solution of the enormous number of additional equations (~N*N~10**12, where N=Nx*Ny*Nz is the size of the state vector, Nx~100, Ny~100, Nz~100 – the dimensions of the computational grid). To overcome this difficulty, approximate or suboptimal Kalman filters were developed. These include the Ensemble Kalman filter and the Reduced-Rank Kalman filters (RRSQRT).Todling, Ricardo, and Stephen E. Cohn. [https://journals.ametsoc.org/view/journals/mwre/122/11/1520-0493_1994_122_2530_ssfada_2_0_co_2.xml "Suboptimal schemes for atmospheric data assimilation based on the Kalman filter."] Monthly Weather Review 122, no. 11 (1994): 2530-2557.

Another significant advance in the development of the 4DDA methods was utilizing the optimal control theory (variational approach) in the works of Le Dimet and Talagrand (1986), based on the previous works of J.-L. Lions and G. Marchuk, the latter being the first to apply that theory in the environmental modeling. The significant advantage of the variational approaches is that the meteorological fields satisfy the dynamical equations of the NWP model and at the same time they minimize the functional, characterizing their difference from observations. Thus, the problem of constrained minimization is solved. The 3DDA variational methods were developed for the first time by Sasaki (1958).

As was shown by Lorenc (1986), all the above-mentioned 4DDA methods are in some limit equivalent, i.e. under some assumptions they minimize the same cost function. However, in practical applications these assumptions are never fulfilled, the different methods perform differently and generally it is not clear what approach (Kalman filtering or variational) is better. The fundamental questions also arise in application of the advanced DA techniques such as convergence of the computational method to the global minimum of the functional to be minimised. For instance, cost function or the set in which the solution is sought can be not convex. The 4DDA method which is currently most successful{{cite web | url=http://ams.confex.com/ams/91Annual/webprogram/Paper181664.html | title=Abstract: Mesoscale ensemble 4DVAR and its comparison with EnKF and 4DVAR (91st American Meteorological Society Annual Meeting)| date=27 January 2011}}{{cite journal |last1=Yang |first1=Eun-Gyeong |last2=Kim |first2=Hyun Mee |title=A comparison of variational, ensemble-based, and hybrid data assimilation methods over East Asia for two one-month periods |journal=Atmospheric Research |date=February 2021 |volume=249 |pages=105257 |doi=10.1016/j.atmosres.2020.105257 |bibcode=2021AtmRe.24905257Y |s2cid=224864029 |url=http://hfip.psu.edu/EDA2010/MZhang.pdf |access-date=9 November 2022 |language=en}} is hybrid incremental 4D-Var, where an ensemble is used to augment the climatological background error covariances at the start of the data assimilation time window, but the background error covariances are evolved during the time window by a simplified version of the NWP forecast model. This data assimilation method is used operationally at forecast centres such as the Met Office.{{cite web |url=https://www.ecmwf.int/sites/default/files/elibrary/2011/14950-hybrid-variationalensemble-data-assimilation.pdf |title=Hybrid Variational/Ensemble Data Assimilation |first1=Dale |last1=Barker |first2=Andrew |last2=Lorenc |first3=Adam |last3=Clayton |date=September 2011 }}{{cite web|url=http://www.metoffice.gov.uk/research/modelling-systems/unified-model/weather-forecasting|title = Numerical weather prediction models}}

=Cost function=

The process of creating the analysis in data assimilation often involves minimization of a cost function. A typical cost function would be the sum of the squared deviations of the analysis values from the observations weighted by the accuracy of the observations, plus the sum of the squared deviations of the forecast fields and the analyzed fields weighted by the accuracy of the forecast. This has the effect of making sure that the analysis does not drift too far away from observations and forecasts that are known to usually be reliable.{{citation needed|date=January 2018}}

==3D-Var==

$J(\mathbf{x}) = (\mathbf{x}-\mathbf{x}_{b})^{\mathrm{T}}\mathbf{B}^{-1}(\mathbf{x}-\mathbf{x}_{b}) + (\mathbf{y}-\mathit{H}[\mathbf{x}])^{\mathrm{T}}\mathbf{R}^{-1}(\mathbf{y}-\mathit{H}[\mathbf{x}]),$

where $\mathbf{B}$ denotes the background error covariance, $\mathbf{R}$ the observational error covariance.

$\nabla J(\mathbf{x}) = 2\mathbf{B}^{-1}(\mathbf{x}-\mathbf{x}_{b}) - 2\mathit{H}^T\mathbf{R}^{-1}(\mathbf{y}-\mathit{H}[\mathbf{x}])$

==4D-Var==

$J(\mathbf{x}) = (\mathbf{x}-\mathbf{x}_{b})^{\mathrm{T}}\mathbf{B}^{-1}(\mathbf{x}-\mathbf{x}_{b}) + \sum_{i=0}^{n}(\mathbf{y}_{i}-\mathit{H}_{i}[\mathbf{x}_{i}])^{\mathrm{T}}\mathbf{R}_{i}^{-1}(\mathbf{y}_{i}-\mathit{H}_{i}[\mathbf{x}_{i}])$

provided that $\mathit{H}$ is a linear operator (matrix).

=Future development=

Factors driving the rapid development of data assimilation methods for NWP models include:

Utilizing the observations currently offers promising improvement in forecast skill at a variety of spatial scales (from global to highly local) and time scales.
The number of different kinds of available observations (sodars, radars, satellite) is rapidly growing.

Other applications

= Monitoring water and energy transfers =

File:ReSeDAssimilationDiagram.png

Data assimilation has been used, in the 1980s and 1990s, in several HAPEX (Hydrologic and Atmospheric Pilot Experiment) projects for monitoring energy transfers between the soil, vegetation and atmosphere. For instance:

- [https://www.eol.ucar.edu/field_projects/hapex-mobilhy HAPEX-MobilHy],{{cite journal |doi=10.1175/1520-0477(1986)067<0138:HAHAEF>2.0.CO;2 |year=1986 |volume=67 |issue=2 |page=138 |title=HAPEX—MOBLIHY: A Hydrologic Atmospheric Experiment for the Study of Water Budget and Evaporation Flux at the Climatic Scale |journal=Bulletin of the American Meteorological Society |last1=André |first1=Jean-Claude |last2=Goutorbe |first2=Jean-Paul |last3=Perrier |first3=Alain |bibcode=1986BAMS...67..138A |doi-access=free }} HAPEX-Sahel,{{cite journal |doi=10.1016/S0022-1694(96)03308-2 |title=An overview of HAPEX-Sahel: A study in climate and desertification |journal=Journal of Hydrology |volume=188-189 |pages=4–17 |year=1997 |last1=Goutorbe |first1=J.P |last2=Lebel |first2=T |last3=Dolman |first3=A.J |last4=Gash |first4=J.H.C |last5=Kabat |first5=P |last6=Kerr |first6=Y.H |last7=Monteny |first7=B |last8=Prince |first8=S.D |last9=Stricker |first9=J.N.M |last10=Tinga |first10=A |last11=Wallace |first11=J.S |bibcode=1997JHyd..188....4G }}

- the "Alpilles-ReSeDA" (Remote Sensing Data Assimilation) experiment,{{cite journal|title=Assimilation of Multi-Sensor and Multi-Temporal Remote Sensing Data, to Monitor Vegetation and Soil: the Alpilles-ReSeDA project.|vauthors=Prevot L, Baret F, Chanzy A, Olioso A, Wigneron JP, Autret H, Baudin F, Bessemoulin P, Bethenod O, Blamont D, Blavoux B, Bonnefond JM, Boubkraoui S, Bouman BA, Braud I, Bruguier N, Calvet JC, Caselles V, Chauki H, Clevers JG, Coll C, Company A, Courault D, Dedieu G, Degenne P, Delecolle R, Denis H, Desprats JF, Ducros Y, Dyer D, Fies JC, Fischer A, Francois C, Gaudu JC, Gonzalez E, Goujet R, Gu XF, Guerif M, Hanocq JF, Hautecoeur O, Haverkamp R, Hobbs S, Jacob F, Jeansoulin R, Jongschaap RE, Kerr Y, King C, Laborie P, Lagouarde JP, Laques AE |display-authors=etal |publisher=IGARSS'98, International Geoscience and Remote Sensing Symposium|date= July 1998|place= Seattle, WA, USA|url=http://w3.avignon.inra.fr/reseda/base/watermed/documents/igarss98_e0403.pdf |access-date=8 July 2019}}{{cite book |doi=10.1109/IGARSS.1998.702226 |chapter=Comparison of the albedo derived from MOS-B and WIFS with NOAA-AVHRR |title=IGARSS '98. Sensing and Managing the Environment. 1998 IEEE International Geoscience and Remote Sensing. Symposium Proceedings. (Cat. No.98CH36174) |pages=2402–4 |year=1998 |last1=Eibl |first1=B |last2=Mauser |first2=W |last3=Moulin |first3=S |last4=Noilhan |first4=J |last5=Ottle |first5=C |last6=Paloscia |first6=S |last7=Pampaloni |first7=P |last8=Podvin |first8=T |last9=Quaracino |first9=F |last10=Roujean |first10=J.L |last11=Rozier |first11=C |last12=Ruisi |first12=R |last13=Susini |first13=C |last14=Taconet |first14=O |last15=Tallet |first15=N |last16=Thony |first16=J.L |last17=Travi |first17=Y |last18=Van Leewen |first18=H |last19=Vauclin |first19=M |last20=Vidal-Madjar |first20=D |last21=Vonder |first21=O.W |isbn=978-0-7803-4403-7 |s2cid=55492076 |url=https://hal.inrae.fr/hal-02770351/file/134612_20120110123553345_1.pdf }} a European project in the [https://cordis.europa.eu/programme/rcn/471/en FP4-ENV] program{{cite web |title=ReSeDA |url= https://cordis.europa.eu/project/rcn/35579/factsheet/en |website=cordis.europa.eu |access-date=8 July 2019}} which took place in the Alpilles region, South-East of France (1996–97). The Flow-chart diagram (right), excerpted from the final report of that project,{{cite web|date=June 2000|publisher=Institut national de la recherche agronomique|place=Avignon|page=59|last1=Baret |first1=Frederic |title=ReSeDA: Assimilation of Multi-Sensor & Multi-Temporal Remote Sensing Data to Monitor Soil & Vegetation Functioning|url= http://w3.avignon.inra.fr/reseda/base/documents/reseda-report/00reseda-report.pdf |access-date=8 July 2019|type=final report, European contract number ENV4CT960326}} shows how to infer variables of interest such as canopy state, radiative fluxes, environmental budget, production in quantity and quality, from remote sensing data and ancillary information. In that diagram, the small blue-green arrows indicate the direct way the models actually run.{{citation needed|date=January 2018}}{{cite conference|title=Spatial Aspects in the Alpilles-ReSeDA Project|date= 19–21 March 1998|pages=93–102|vauthors= Olioso A, Prevot L, Baret F, Chanzy A, Braud I, Autret H, Baudin F, Bessemoulin P, Bethenod O, Blamont D, Blavoux B, Bonnefond JM, Boubkraoui S, Bouman BA, Bruguier N, Calvet JC, Caselles V, Chauki H, Clevers JW, Coll C, Company A, Courault D, Dedieu G, Degenne P, Delecolle R, Denis H, Desprats JF, Ducros Y, Dyer D, Fies JC, Fischer A, Francois C, Gaudu JC, Gonzalez E, Gouget R, Gu XF, Guerif M, Hanocq JF, Hautecoeur O, Haverkamp R, Hobbs S, Jacob F, Jeansoulin R, Jongschaap RE, Kerr Y, King C, Laborie P, Lagouarde JP, Laques AE, Larcena D, Laurent G, Laurent JP, Leroy M, McAneney J, Macelloni G, Moulin S, Noilhan J, Ottle C, Paloscia S, Pampaloni P, Podvin T, Quaracino F, Roujean JL, Rozier C, Ruisi R, Susini C, Taconet O, Tallet N, Thony JL, Travi Y, van Leewen H, Vauclin M, Vidal-Madjar D, Vonder OW, Weiss M, Wigneron JP |location= University of Montreal, Montréal, Québec, Canada |conference= International Workshop on Scaling and Modelling in Forestry: Applications in Remote Sensing and GIS |editor= D. Marceau |url= http://w3.avignon.inra.fr/reseda/base/watermed/documents/can1.pdf |access-date=8 July 2019}}

= Other forecasting applications =

Data assimilation methods are currently also used in other environmental forecasting problems, e.g. in hydrological and hydrogeological forecasting.{{Cite journal |last1=Chen |first1=Shang-Ying |last2=Wei |first2=Jian-Yu |last3=Hsu |first3=Kuo-Chin |date=2023-10-01 |title=Data assimilation for real-time subsurface flow modeling with dynamically adaptive meshless node adjustments |url=https://doi.org/10.1007/s00366-023-01897-6 |journal=Engineering with Computers |volume=40 |issue=3 |pages=1893–1925 |language=en |doi=10.1007/s00366-023-01897-6 |issn=1435-5663|url-access=subscription }} Bayesian networks may also be used in a data assimilation approach to assess natural hazards such as landslides.{{cite journal|last1= Cardenas |first1=IC|title= On the use of Bayesian networks as a meta-modelling approach to analyse uncertainties in slope stability analysis|journal =Georisk: Assessment and Management of Risk for Engineered Systems and Geohazards|date=2019|volume=13|issue=1|pages=53–65|doi=10.1080/17499518.2018.1498524|bibcode=2019GAMRE..13...53C |s2cid=216590427}}

Given the abundance of spacecraft data for other planets in the Solar System, data assimilation is now also applied beyond the Earth to obtain re-analyses of the atmospheric state of extraterrestrial planets. Mars is the only extraterrestrial planet to which data assimilation has been applied so far. Available spacecraft data include, in particular, retrievals of temperature and dust/water/ice optical thicknesses from the Thermal Emission Spectrometer onboard NASA's Mars Global Surveyor and the Mars Climate Sounder onboard NASA's Mars Reconnaissance Orbiter. Two methods of data assimilation have been applied to these datasets: an Analysis Correction scheme {{cite web |url=http://www.atm.ox.ac.uk/group/gpfd/research.html#marsgcm |title=Oxford Physics: Atmospheric, Oceanic and Planetary Physics: SRC: Research |access-date=2011-08-19 |archive-url=https://web.archive.org/web/20110928051446/http://www.atm.ox.ac.uk/group/gpfd/research.html#marsgcm |archive-date=2011-09-28 |url-status=dead |date=July 2019}} and two Ensemble Kalman Filter schemes,{{cite web | url=http://www.eps.jhu.edu/~mjhoffman/pages/research.html | title=Matthew J. Hoffman - Research }}{{cite web|url=http://www.marsclimatecenter.com |title=marsclimatecenter.com |publisher=marsclimatecenter.com |date= |accessdate=2022-04-19}} both using a global circulation model of the martian atmosphere as forward model. The Mars Analysis Correction Data Assimilation (MACDA) dataset is publicly available from the British Atmospheric Data Centre.http://badc.nerc.ac.uk/home/{{full citation needed|date=January 2018}}

Data assimilation is a part of the challenge for every forecasting problem.

Dealing with biased data is a serious challenge in data assimilation. Further development of methods to deal with biases will be of particular use. If there are several instruments observing the same variable then intercomparing them using probability distribution functions can be instructive.{{citation needed|date=January 2018}}

The numerical forecast models are becoming of higher resolution due to the increase of computational power, with operational atmospheric models now running with horizontal resolutions of order of 1 km (e.g. at the German National Meteorological Service, the Deutscher Wetterdienst (DWD) and Met Office in the UK). This increase in horizontal resolutions is starting to allow to resolve more chaotic features of the non-linear models, e.g. to resolve convection on the grid scale, or clouds, in the atmospheric models. This increasing non-linearity in the models and observation operators poses a new problem in the data assimilation. The existing data assimilation methods such as many variants of ensemble Kalman filters and variational methods, well established with linear or near-linear models, are being assessed on non-linear models.

Many new methods are being developed, e.g. particle filters for high-dimensional problems, and hybrid data assimilation methods.{{Cite journal|last1=Vetra-Carvalho|first1=Sanita|last2=P. J. van Leeuwen|last3=L. Nerger|last4=A. Barth|last5=A.M. Umer|last6=P. Brasseur|last7=P. Kirchgessner|last8=J-M. Beckers|date=2018|title=State-of-the-art stochastic data assimilation methods for high-dimensional non-Gaussian problems|journal=Tellus A|volume=70|issue=1|page=1445364|doi=10.1080/16000870.2018.1445364|bibcode=2018TellA..7045364V|doi-access=free|hdl=10754/630565|hdl-access=free}}

Other uses include trajectory estimation for the Apollo program, GPS, and atmospheric chemistry.

References

External links

Examples of how variational assimilation is implemented weather forecasting at:

{{cite book |title=Data Assimilation |work=ECMWF |url=https://www.ecmwf.int/en/elibrary/9231-part-ii-data-assimilation |publisher=ECMWF |year=2010 |series=IFS Documentation }}
{{cite web |title=Data Assimilation |work=Met Office |url=https://www.metoffice.gov.uk/research/weather/data-assimilation }}

Other examples of assimilation:

[https://web.archive.org/web/20060717223237/http://www.cdacentral.info/ CDACentral (an example analysis from Chemical Data Assimilation)]
[https://web.archive.org/web/20070223072036/http://www.pdfcentral.info/ PDFCentral (using PDFs to examine biases and representativeness)]
[http://www.openda.org/joomla/index.php OpenDA – Open Source Data Assimilation package] {{Webarchive|url=https://web.archive.org/web/20110112140216/http://www.openda.org/joomla/index.php |date=2011-01-12 }}
[http://pdaf.awi.de PDAF – open-source Parallel Data Assimilation Framework]
[http://www.data-assimilation.net/ SANGOMA New Data Assimilation techniques]

Category:Weather forecasting

Category:Numerical climate and weather models

Category:Estimation theory

Category:Control theory

Category:Bayesian statistics

Category:Climate and weather statistics

Category:Statistical forecasting