Empirical dynamic modeling

Mathematical models have tremendous power to describe observations of real-world systems. They are routinely used to test hypothesis, explain mechanisms and predict future outcomes. However, real-world systems are often nonlinear and multidimensional, in some instances rendering explicit equation-based modeling problematic. Empirical models, which infer patterns and associations from the data instead of using hypothesized equations, represent a natural and flexible framework for modeling complex dynamics.

Donald DeAngelis and Simeon Yurek illustrated that canonical statistical models are ill-posed when applied to nonlinear dynamical systems.[https://www.pnas.org/content/112/13/3856]Donald L. DeAngelis, Simeon Yurek, 2015, Equation-free modeling unravels the behavior of complex ecological systems. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) 3856-3857; DOI: 10.1073/pnas.1503154112 A hallmark of nonlinear dynamics is state-dependence: system states are related to previous states governing transition from one state to another. EDM operates in this space, the multidimensional state-space of system dynamics rather than on one-dimensional observational time series. EDM does not presume relationships among states, for example, a functional dependence, but projects future states from localised, neighboring states. EDM is thus a state-space, nearest-neighbors paradigm where system dynamics are inferred from states derived from observational time series. This provides a model-free representation of the system naturally encompassing nonlinear dynamics.

A cornerstone of EDM is recognition that time series observed from a dynamical system can be transformed into higher-dimensional state-spaces by time-delay embedding with Takens's theorem. The state-space models are evaluated based on in-sample fidelity to observations, conventionally with Pearson correlation between predictions and observations.

EDM is continuing to evolve. As of 2022, the main algorithms are Simplex projection,[https://www.nature.com/articles/344734a0] Sugihara G. and May R., 1990. Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series. Nature, 344:734–741 Sequential locally weighted global linear maps (S-Map) projection,[https://royalsocietypublishing.org/doi/abs/10.1098/rsta.1994.0106] Sugihara G., 1994. Nonlinear forecasting for the classification of natural time series. Philosophical Transactions: Physical Sciences and Engineering, 348 (1688) : 477–495 Multivariate embedding in Simplex or S-Map,

Convergent cross mapping (CCM),[https://www.science.org/doi/10.1126/science.1227079] Sugihara G., May R., Ye H., et al. 2012. Detecting Causality in Complex Ecosystems. Science 338:496-500

and Multiview Embeding,[https://www.science.org/doi/10.1126/science.aag0863] Ye H., and G. Sugihara, 2016. Information leverage in interconnected ecosystems: Overcoming the curse of dimensionality. Science 353:922–925 described below.

Nearest neighbors are found according to:

$\backslash text\{NN\}(y,\; X,\; k)\; =\; \backslash |\; X\_\{N\_i\}^\{E\}\; -\; y\backslash |\; \backslash leq\; \backslash |\; X\_\{N\_j\}^\{E\}\; -\; y\backslash |\; \backslash text\{\; if\; \}\; 1\; \backslash leq\; i\; \backslash leq\; j\; \backslash leq\; k$

Simplex projection[https://doi.org/10.1007/BFb0091924] Takens, F. (1981). Detecting strange attractors in turbulence. In D. A. Rand & L. S. Young (Eds.), Dynamical Systems and Turbulence (pp. 366–381). Springer.[https://doi.org/https://doi.org/10.1016/0167-2789(89)90074-2] Casdagli, M. (1989). Nonlinear prediction of chaotic time series. Physica D: Nonlinear Phenomena, 35(3), 335–356.[https://doi.org/10.1016/S0167-2789(98)00089-X] Judd, K., & Mees, A. (1998). Embedding as a modeling problem. Physica D: Nonlinear Phenomena, 120(3), 273–286. is a nearest neighbor projection. It locates the $k$ nearest neighbors to the location in the state-space from which a prediction is desired. To minimize the number of free parameters $k$ is typically set to $E+1$ defining an $E+1$ dimensional simplex in the state-space. The prediction is computed as the average of the weighted phase-space simplex projected $Tp$ points ahead. Each neighbor is weighted proportional to their distance to the projection origin vector in the state-space.

1. Find $k$ nearest neighbor: $N\_k\; \backslash gets\; \backslash text\{NN\}(y,\; X,\; k)$
2. Define the distance scale: $d\; \backslash gets\; \backslash |\; X\_\{N\_1\}^\{E\}\; -\; y\backslash |$
3. Compute weights: For{$i=1,\backslash dots,k$} : $w\_i\; \backslash gets\; \backslash exp\; (-\backslash |\; X\_\{N\_i\}^\{E\}\; -\; y\backslash |\; /\; d\; )$
4. Average of state-space simplex: $\backslash hat\{y\}\; \backslash gets\; \backslash sum\_\{i\; =\; 1\}^\{k\}\; \backslash left(w\_iX\_\{N\_i+T\_p\}\backslash right)\; /\; \backslash sum\_\{i\; =\; 1\}^\{k\}\; w\_i$

S-Map extends the state-space prediction in Simplex from an average of the $E+1$ nearest neighbors to a linear regression fit to all neighbors, but localised with an exponential decay kernel. The exponential localisation function is $F(\backslash theta)\; =\; \backslash text\{exp\}(-\backslash theta\; d/D)$, where $d$ is the neighbor distance and $D$ the mean distance. In this way, depending on the value of $\backslash theta$, neighbors close to the prediction origin point have a higher weight than those further from it, such that a local linear approximation to the nonlinear system is reasonable. This localisation ability allows one to identify an optimal local scale, in-effect quantifying the degree of state dependence, and hence nonlinearity of the system.

Another feature of S-Map is that for a properly fit model, the regression coefficients between variables have been shown to approximate the gradient (directional derivative) of variables along the manifold.[http://dx.doi.org/10.1098/rspb.2015.2258]Deyle ER. et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258 These Jacobians represent the time-varying interaction strengths between system variables.

1. Find $k$ nearest neighbor: $N\; \backslash gets\; \backslash text\{NN\}(y,\; X,\; k)$
2. Sum of distances: $D\; \backslash gets\; \backslash frac\{1\}\{k\}\; \backslash sum\_\{i=1\}^k\; \backslash |\; X\_\{N\_i\}^\{E\}\; -\; y\backslash |$
3. Compute weights: For{$i=1,\backslash dots,k$} : $w\_i\; \backslash gets\; \backslash exp\; (-\backslash theta\; \backslash |\; X\_\{N\_i\}^\{E\}\; -\; y\backslash |\; /\; D\; )$
4. Reweighting matrix: $W\; \backslash gets\; \backslash text\{diag\}(w\_i)$
5. Design matrix: $A\; \backslash gets$

\begin{bmatrix}

1 & X_{N_1} & X_{N_1- 1} & \dots & X_{N_1 - E + 1} \\

1 & X_{N_2} & X_{N_2- 1} & \dots & X_{N_2 - E + 1} \\

\vdots & \vdots & \vdots & \ddots & \vdots \\

1 & X_{N_k} & X_{N_k- 1} & \dots & X_{N_k - E + 1}

\end{bmatrix}

1. Weighted design matrix: $A\; \backslash gets\; WA$
2. Response vector at $Tp$: $b\; \backslash gets$

\begin{bmatrix}

X_{N_1 + T_p} \\

X_{N_2 + T_p} \\

\vdots \\

X_{N_k + T_p}

\end{bmatrix}

1. Weighted response vector: $b\; \backslash gets\; Wb$
2. Least squares solution (SVD): $\backslash hat\{c\}\; \backslash gets\; \backslash text\{argmin\}\_\{c\}\backslash |\; Ac\; -\; b\; \backslash |\_2^2$
3. Local linear model $\backslash hat\{c\}$ is prediction: $\backslash hat\{y\}\; \backslash gets\; \backslash hat\{c\}\_0\; +\; \backslash sum\_\{i=1\}^E\backslash hat\{c\}\_iy\_i$

Multivariate Embedding[https://doi.org/10.1007/BF01053745] Sauer, T., Yorke, J. A., & Casdagli, M. (1991). Embedology. Journal of Statistical Physics, 65(3), 579–616 recognizes that time-delay embeddings are not the only valid state-space construction. In Simplex and S-Map one can generate a state-space from observational vectors, or time-delay embeddings of a single observational time series, or both.

Convergent cross mapping (CCM) leverages a corollary to the Generalized Takens Theorem that it should be possible to cross predict or cross map between variables observed from the same system. Suppose that in some dynamical system involving variables $X$ and $Y$, $X$ causes $Y$. Since $X$ and $Y$ belong to the same dynamical system, their reconstructions (via embeddings) $M\_\{x\}$, and $M\_\{y\}$, also map to the same system.

The causal variable $X$ leaves a signature on the affected variable $Y$, and consequently, the reconstructed states based on $Y$ can be used to cross predict values of $X$. CCM leverages this property to infer causality by predicting $X$ using the $M\_\{y\}$ library of points (or vice versa for the other direction of causality), while assessing improvements in cross map predictability as larger and larger random samplings of $M\_\{y\}$ are used. If the prediction skill of $X$ increases and saturates as the entire $M\_\{y\}$ is used, this provides evidence that $X$ is casually influencing $Y$.

Multiview Embedding is a Dimensionality reduction technique where a large number of state-space time series vectors are combitorially assessed towards maximal model predictability.

Extensions to EDM techniques include:

• Generalized Theorems for Nonlinear State Space Reconstruction
• Extended Convergent Cross Mapping
• Dynamic stability
• S-Map regularization[https://doi.org/10.1111/2041-210X.13150]Cenci S, Sugihara G, Saavedra S, 2019. Regularized S-map for inference and forecasting with noisy ecological time series, METHODS IN ECOLOGY AND EVOLUTION, 10 (5), 650-660
• Visual analytics with EDM[https://www.computer.org/csdl/journal/tg/2021/02/09216532/1nJsFIg64us] Hiroaki Natsukawa, et al. 2021. A Visual Analytics Approach for Ecosystem Dynamics based on Empirical Dynamic Modeling. IEEE Transactions on Visualization and Computer Graphics. Feb. 2021, 506-516, vol. 27

DOI: 10.1109/TVCG.2020.3028956

• Convergent Cross Sorting[https://doi.org/10.1038/s41598-021-98864-2] Breston, L., Leonardis, E.J., Quinn, L.K. et al. 2021. Convergent cross sorting for estimating dynamic coupling. Sci Rep 11, 20374 (2021). doi:10.1038/s41598-021-98864-2
• Expert system with EDM hybrid[https://doi.org/10.1073/pnas.2102466119] Deyle E. R. et al. A hybrid empirical and parametric approach for managing ecosystem complexity: Water quality in Lake Geneva under nonstationary futures. PNAS Vol. 119, No. 26 (2022).
• Sliding windows based on the extended convergent cross-mapping[https://doi.org/10.1007/s11071-021-06362-x] Ge, X., Lin, A. Dynamic causality analysis using overlapped sliding windows based on the extended convergent cross-mapping. Nonlinear Dyn 104, 1753–1765 (2021). https://doi.org/10.1007/s11071-021-06362-x
• Empirical Mode Modeling
• Variable step sizes with bundle embedding[https://www.sciencedirect.com/science/article/abs/pii/S0304380022000680] Bethany Johnson, Stephan B. Munch. 2022. An empirical dynamic modeling framework for missing or irregular samples. Ecological Modelling, Volume 468, June 2022, 109948.
• Multiview distance regularised S-map[https://doi.org/10.1111/ele.13897] Chang, C.-W., Miki, T., Ushio, M., et al. (2021) Reconstructing large interaction networks from empirical time series data. Ecology Letters, 24, 2763– 2774. https://doi.org/10.1111/ele.13897

• System dynamics
• Complex dynamics
• Nonlinear dimensionality reduction

{{Reflist}}

• {{cite journal

|journal = Ecol Res

|year = 2017

|title = Empirical dynamic modeling for beginners

|pages = 785–796

|author = Chang, CW., Ushio, M. & Hsieh, Ch.

|volume = 32

|issue = 6

|doi = 10.1007/s11284-017-1469-9

|s2cid = 4641225

|doi-access = free

|bibcode = 2017EcoR...32..785C

|hdl = 2433/235326

|hdl-access = free

}}

• {{cite journal

|journal = ICES Journal of Marine Science

|year = 2020

|title = Frequently asked questions about nonlinear dynamics and empirical dynamic modelling

|pages = 1463–1479

|author = Stephan B Munch, Antoine Brias, George Sugihara, Tanya L Rogers

|volume = 77

|doi = 10.1093/icesjms/fsz209

|issue = 4

|url = https://doi.org/10.1093/icesjms/fsz209

|url-access = subscription

}}

• {{cite journal

|journal = Proceedings of the National Academy of Sciences

|year = 2015

|title = Equation-free modeling unravels the behavior of complex ecological systems

|pages = 3856–3857

|author = Donald L. DeAngelis, Simeon Yurek

|volume = 112

|doi = 10.1073/pnas.1503154112

|issue = 13

|pmid = 25829536

|pmc = 4386356

|doi-access = free

}}

;Animations

• {{YouTube|fevurdpiRYg|State Space Reconstruction: Time Series and Dynamic Systems}}
• {{YouTube|QQwtrWBwxQg|State Space Reconstruction: Takens' Theorem and Shadow Manifolds}}
• {{YouTube|NrFdIz-D2yM|State Space Reconstruction: Convergent Cross Mapping}}

;Online books or lecture notes

• [https://sugiharalab.github.io/EDM_Documentation/ EDM Introduction]. Introduction with video, examples and references.
• [https://arxiv.org/abs/math.HO/0111177 Geometrical theory of dynamical systems]. Nils Berglund's lecture notes for a course at ETH at the advanced undergraduate level.
• [https://www.arxiv.org/list/math.DS/recent Arxiv preprint server] has daily submissions of (non-refereed) manuscripts in dynamical systems.

;Research groups

• [https://deepeco.ucsd.edu/ Sugihara Lab], Scripps Institution of Oceanography, University of California San Diego.

{{Authority control}}

Category:Nonlinear systems

Category:Data modeling

Category:Predictive analytics

Category:Machine learning

Category:Nonlinear time series analysis