Overview of neural network simulation

The framework of the study is described in Figure 1. The neural network model is trained to minimize the precision-weighted prediction error of the next step value in the target sequence obtained from facial expression videos, based on predictive processing theory. It is important to note that no emotion labels are given to this model. Whether this model is able to recognize emotions is evaluated by observing the “self-organization” of clusters for each emotion in the higher-level neuron, i.e., parametric bias (PB) space, which will be explained later. The experimental procedure for evaluating ASD-like performance consists of a training phase, which is analogous to the developmental learning process, and a test phase, which is analogous to the emotion recognition process from unknown facial expressions.

Neural network model

The main component of the neural network model used in the simulation is the stochastic continuous time recurrent neural network with parametric bias (S-CTRNNPB), which is analogous to the biological brain, based on predictive processing theory (Murata et al., 2013; Murata et al., 2017). In the feed-forward prediction of S-CTRNNPB, the internal state of the ith neuron at time step t for the sth sequence is calculated as follows,

where I_I, I_P, I_L, I_M, and I_V are index sets of the input, parametric bias (PB), lower-level, predicted mean, and variance neurons, respectively; w_ij is the synaptic connection weight from the jth neuron to the ith neuron; $x_{t,j}^{(s)}$M4 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[x_{t,j}^{(s)}\] \end{document} is the jth external input value at time step t of the sth sequence, $l_{t,j}^{(s)}$M5 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[l_{t,j}^{(s)}\] \end{document} is the jth lower-level neuron, $p_{t,j}^{(s)}$M6 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[p_{t,j}^{(s)}\] \end{document} is the jth PB activity; τ_i is the time constant of the ith neuron; and a_i is the activity threshold of the th neuron. As represented in the above equation, S-CTRNNPB is a hierarchical neural network model, and higher-level neuron is called PB. Equation (1) indicates that the internal state of PB neurons does not change with time.

Parameter optimization is performed by minimizing the negative log-likelihood, which assumes a Gaussian distribution for observations, as shown in the following equation,

where $v_{t,i}^{(s)}$M7 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[v_{t,i}^{(s)}\] \end{document} and $y_{t,i}^{(s)}$M8 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[y_{t,i}^{(s)}\] \end{document} are the predicted mean and variance, respectively; ${\widehat{y}}_{t,i}^{(s)}$M9 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\hat y_{t,i}^{(s)}\] \end{document} is the target value (input value at the next time step). Minimizing this negative log-likelihood can be regarded as minimizing the precision-weighted prediction error. Details of feed-forward prediction, activation function, and parameter optimization are described in the Supplementary Methods.

Experimental procedures and ASD-like performance measures

In the training phase, parameter optimization is performed on the network structure, e.g., synaptic weights, and PB activities corresponding to each target sequence. Target sequences consisted of 6 basic emotions × 16 subjects and were prepared by extracting 9 features from facial expression videos (Supplementary Methods). After training, each target sequence was associated with a specific PB activity, and the relationships (similarities and differences) between target sequences were expected to be “self-organized” in the state space of the PB activity (referred to as “PB space” below). In the test phase, the neural network was required to predict unseen test target sequences. In this test phase, parameter optimization is only applied to PB activities such that the prediction error for test target sequences is minimized while the network structure remains fixed.

This PB update process for the test target sequence was considered “emotion recognition”, based on the similarity of the PB activity for the test sequence to the PB activity clusters for training sequences of the same emotion. This similarity of PB activities within the same emotion is quantitatively evaluated by a clustering index, an average silhouette width, called an “emotion recognition index” herein (Supplementary Methods). In addition to the emotion recognition index, we also included the prediction error of the test target sequence (test error) as an ASD-like performance measure. Based on predictive processing theory, the test error reflects impairments in generalization, a cognitive trait of ASD (Haker et al., 2016). This corresponds to the cognitive tendency in ASD to focus on details of sensory information, making it difficult to extract abstract meaning, resulting in overfitting (Van de Cruys et al., 2013). Eight-fold cross validation was used to calculate ASD-like performance.

Experimental conditions in neural network simulation

Based on previous studies (Idei et al., 2020; Idei et al., 2021; Lanillos et al., 2020; Takahashi et al., 2021; Thomas et al., 2016), we examined effects of the following two parameters on the ASD-like performance in the neural network model (Figure 1).

The first parameter is the FC_model, which indicates the proportion of synaptic connections between groups of neurons at different hierarchical levels. FC_model is assumed to correspond to FC as measured by fMRI (hereafter referred to as FC_MRI in comparison to FC_model). As shown in Figure 1, the FC_model between the PB and the lower-level neuron and the FC_model between the lower-level neurons and the input/prediction neurons are referred to as higher-level FC_model and lower-level FC_model, respectively, which were set to one of the following values: 100, 80, 60, 40, or 20%. The second parameter is network excitability homogeneity. The intrinsic homogeneity of network excitability is important for efficient information processing (Hunsberger et al., 2014; Mejias & Longtin, 2012, 2014), and its alterations are thought to be related to ASD, i.e., altered “excitatory-inhibitory balance” (Idei et al., 2020). In the current experiment, as shown in Equation (3), the activity threshold of lower-level neurons, i.e., a_i in Equation (1), is initialized to follow a Gaussian distribution and fixed without being updated by learning. Note that in the training phase, the network structure (weight and PB) is optimized based on the distribution of these activity thresholds.

The K parameter in Equation (3) determines the homogeneity of intrinsic neuronal excitability, and as parameter K decreases, excitability of the network becomes more homogeneous. In the current analysis, the experimental conditions of K = 0.1, 1, and 10 are referred to as the highly homogeneous, modestly homogeneous, and heterogeneous network conditions, respectively.

Examining the association between simulation results and fMRI datasets

Based on the assumption that there is a correspondence between neural network parameters and fMRI parameters, we examined whether the relationship between neural network parameters and ASD-like performance in the neural network simulation could be validated between subject subgroups allocated on the basis of fMRI parameters and ASD diagnosis/symptoms in the fMRI dataset. As an fMRI dataset, we used resting state fMRI datasets for 849 subjects (410 ASD and 439 TD) from the publicly available Autism Brain Imaging Data Exchange (Craddock et al., 2013; Di Martino et al., 2014). See Supplementary Methods, Supplementary Table 1, and Supplementary Table 2 for details.

We determined the fMRI parameters corresponding to neural network parameters by the following procedure. First, referring to previous neural circuit studies, insula (Fusar-Poli et al., 2009; Gobbini & Haxby, 2007; Haxby et al., 2002; Wicker et al., 2003; Wright et al., 2004), middle temporal gyrus (temporo-occipital part) (Beauchamp et al., 2004; Deen et al., 2020; Fusar-Poli et al., 2009; Gobbini & Haxby, 2007; Haxby et al., 2002; Sabatinelli et al., 2011; Wang et al., 2016), and visual cortex (intracalcarine cortex) (Bernstein & Yovel, 2015; Fusar-Poli et al., 2009; Haxby et al., 2002) were selected as key anatomical regions for facial emotion recognition, and the three levels of hierarchical neural network model, i.e. PB, lower-level, and input neurons, in the neural network were assumed to correspond to each brain region, respectively. Based on these assumptions, higher-level FC_model and lower-level FC_model were assumed to correspond to the higher-level FC_MRI, i.e., FC_MRI between insula and middle temporal gyrus, and the lower-level FC_MRI, i.e., FC_MRI between middle temporal gyrus and visual cortex, respectively. In addition, network excitability homogeneity was assumed to correspond to the ReHo of the middle temporal gyrus (Figure 1). It should be noted that there are other choices regarding selection of anatomical regions. For example, other regions related to facial emotion recognition include the prefrontal cortex (7–9), amygdala (7–10), fusiform gyrus (7–9), thalamus (8–10), and parahippocampal gyrus (8, 9). Furthermore, it is sometimes difficult to determine whether a given region serves as a higher-level or lower-level neuron. Application of the simulation results to the fMRI dataset in this study is not intended to be exhaustive and is still at the proof-of-concept stage. The Harvard Oxford atlas was used to identify the above anatomical regions (Makris et al., 2006). The aforementioned FC_MRIs and ReHo were transformed to normal distributions using Fisher r-z and Box-Cox transformations, respectively, and were adjusted for covariates, i.e., age, sex, FIQ, mean framewise displacement, and sites.

Next, each experimental condition of neural network simulation was mapped to a subject subgroup on the basis of fMRI parameters as follows. In fMRI datasets, 849 subjects were divided into 5 × 5 × 3 = 75 subgroups by dividing them into 5, 5, and 3 groups of approximately the same numbers of subjects, according to their higher/lower-level FC_MRI, and ReHo ranks, respectively. Each subgroup was assigned to one of 5 × 5 × 3 = 75 neural network parameter settings with higher/lower-level FC_model of 20, 40, 60, 80, and 100% and network excitability homogeneity of highly homogeneous, modestly homogeneous, and heterogeneous, respectively.

Finally, we examined whether the subject group corresponding to the experimental condition that exhibited ASD-like performance in the neural network simulation actually had an ASD diagnosis/symptoms.

Performance of a typical development (TD) model in neural network simulation

The network with the experimental condition in which network parameters were set at a heterogeneous network excitability and no reduction in FC, i.e. higher/lower-level FC_model = 100%, is called a TD model. The learning curve for the TD model shows that both training and test error substantially decrease (Figure 2A). This indicates that the model not only successfully reproduces the training target sequence, but also generalizes to an unknown test target sequence.

Examples of target, prediction, lower-level neuron and PB activity sequences in the test phase are shown in Figure 2B. The target sequence varies with emotion, and the prediction reproduces the target sequence well. Lower-level neuron activity corresponds to dynamics of the short-term target sequence, whereas PB activity corresponds to a more abstract level of characteristics of the target sequence.

PB activities corresponding to all target sequences, i.e., PB space, are shown in Figure 2C. In Figure 2C, PB activities corresponding to training target sequences seem to form emotion clusters, indicating that emotion clusters are self-organized in the predictive processing framework, even though no emotion labels are given in training. Furthermore, we can see that PB activities corresponding to the test sequence are located near PB clusters of the same emotion as the training sequences. This indicates that the model successfully recognizes facial emotions even for unknown test data based on a predictive processing framework.

Influence of neural network parameters on ASD-like performance and the interaction between parameters

The results of prediction error, one of the ASD-like performance indicators, are shown in Figure 3A (training error) and Figure 3B (test error). These figures show the change in prediction error when varying network excitability homogeneity and the higher/lower-level FC_model. In the highly homogeneous network with the high FC_model condition, the training error was small, but the test error was large, indicating that the model was overfitted to training sequences and failed to acquire generalization capability, i.e., ASD-like performance. On the other hand, in the other network excitability homogeneity conditions (modestly homogeneous and heterogeneous), both training and test errors were small, regardless of the FC_model condition, indicating that the model successfully acquired generalization capability to predict emotional facial expressions, i.e., TD-like performance.

Next, to examine the other ASD-like performance measure, emotion recognition, i.e., the similarity of PB activity for test to training PB activity clusters within the same emotion, we illustrated PB spaces under different experimental conditions (Figure 4A–D and Supplementary Figure 1). Comparing these four PB spaces, in the heterogeneous network with the FC_model = 100% (Figure 4C), PB activities for test were most clearly located within training PB activity clusters of the same emotion, i.e., most successful emotion recognition. In the highly homogeneous/heterogeneous network with FC_model = 20% (Figure 4B and 4D), PB activities for test were not so far from training PB activity clusters, but boundaries between emotional PB activity clusters seem a little unclear, i.e. almost successful emotion recognition. In the highly homogeneous network with FC_model = 100% (Figure 4A), PB activities for test were located farthest away from training PB activity clusters, i.e., most unsuccessful emotional recognition. From these findings, regarding network excitability homogeneity, the homogeneous network condition appears to show more unsuccessful emotion recognition, i.e., ASD-like performance. On the other hand, regarding FC_model, performance in emotion recognition appears to be reversed, depending on network excitability homogeneity. That is, in a heterogeneous network, the high FC_model condition shows more successful emotion recognition, i.e., TD-like performance, than the low FC_model condition, while in a highly homogeneous network, the high FC_model condition shows more unsuccessful emotion recognition, i.e., ASD-like performance than the low FC_model condition. These intuitive findings are more clearly illustrated in Figure 4E by quantitative analyses using a clustering measure of PB activity by emotion called “emotion recognition index”, i.e., average silhouette width, described in Methods. Specifically, regarding network excitability homogeneity, the homogeneous network showed a lower emotion recognition index, i.e., more unsuccessful emotion recognition. Regarding FC_model, in a heterogeneous network, the high FC condition showed a higher emotion recognition index than the low FC condition, while conversely, in a highly homogeneous network, the high FC condition showed lower emotion recognition index than the low FC condition.

From the above analysis, we found that the two ASD-like performance measures, test error and emotion recognition, share common trends. First, ASD-like performance is exacerbated as network excitability becomes more homogeneous. Second, the effect of FC_model on ASD-like performance depends on network excitability homogeneity, i.e., FC_model and network excitability homogeneity interact.

The mechanism by which changes in neural network parameter cause ASD-like performance

In order to clarify the mechanism by which network excitability homogeneity and FC_model interact to cause ASD-like performance, we examined how each parameter affects neural representations obtained in developmental learning.

Regarding network excitability homogeneity, the aforementioned analysis revealed that heterogeneous network conditions tended to acquire generalization capability in that they had lower test error and were more successful in emotion recognition. We then hypothesized that the PB space in a heterogeneous network condition would tolerate subtle differences among target sequences for prediction, while the homogeneous network would be fragile to subtle differences due to overfitting to a particular sequence of the training sequences. To investigate this hypothesis, we evaluated the number of training sequences that could be well predicted, i.e., average mean squared error <0.005, based on a particular value of PB activity, i.e., closed loop analysis. See Supplementary Methods for details. This number of successfully predicted sequences would reflect the tolerability of PB space for application to different sequences. As expected, it is clear that individual PB activities in the heterogeneous network condition (Figure 5B) are able to predict more sequences with small errors than PB activities in the highly homogeneous network condition (Figure 5A), and this is confirmed by quantitative analysis, which calculated the average per network condition (Figure 5C).

As mentioned above, network excitability and FC_model interact, and network excitability affected the tolerability of PB space, which influenced the generalization of top-down prediction. Based on the fact that prediction in predictive processing system is based on both top-down prediction and bottom-up sensory input, we hypothesize that FC_model determines whether the information processing is top-down prediction dependent or sensory input dependent (or hypo-prior (Pellicano & Burr, 2012)). To investigate this hypothesis, prediction errors for the unreliable (random) PBs, i.e., sensory input-driven generation (Figure 5D), and prediction errors for closed-loop generation, i.e., top-down only generation (Figure 5E), were calculated. In the low FC_model condition, the prediction error for sensory input-driven generation was small (Figure 5D), but the prediction error based on top-down only generation was large (Figure 5E), indicating that the low FC_model condition induced sensory input-dependent information processing, while in the high FC_model condition, the opposite was true: top-down prediction-dependent information processing.

The above examination provides the following explanation for the interaction of neural network parameters to exhibit ASD-like performance. Under the homogeneous network condition, PB space is intolerant, and top-down predictions are overfitted to subtle differences among sequences. As such, when top-down prediction is not accurate for test sequences, high FC_model conditions with top-down prediction-dependence exhibit more ASD-like performance than low FC_model conditions. On the contrary, under a heterogeneous network condition, PB space is tolerant, and the network provides accurate top-down predictions for test sequences. In this case, high FC_model conditions with top-down prediction-dependence exhibit more TD-like performance than low FC_model conditions.

In addition, there is a debate as to whether ASD information processing is top-down prediction or sensory input-dependent (hypo-prior) (Idei et al., 2018; Karvelis et al., 2018; Lawson et al., 2014; Pellicano & Burr, 2012; Philippsen & Nagai, 2018, 2020). This study suggests that both top-down prediction-dependent (high FC_model) and sensory input-dependent (low FC_model) information processing may exhibit outwardly ASD-like performance, i.e., impaired emotion recognition, depending on the network excitability homogeneity.

Applicability of neural network simulation results to fMRI data

Neural network simulation showed that parameters interacted to cause ASD-like performance, and provided an explanation of information processing mechanisms. Of these findings, the relationship between parameters and ASD-like performance can be examined for biological reproducibility using the following procedure. First, by assuming a correspondence between the neural network parameter, i.e., network excitability homogeneity and FC_model, and the fMRI parameter, i.e., regional homogeneity and FC_MRI, each experimental condition of the neural network is mapped to a subject subgroup identified by fMRI parameters (see Methods for details). As in the Methods, it should be noted that a significant simplification in the selection of anatomical regions has been made in this mapping. Second, we examined whether subject subgroups corresponding to neural networks that exhibited ASD-like performance actually showed more ASD diagnoses/symptoms.

Following the above procedure, in order to examine the correspondence between ASD-like performance of neural network and ASD diagnosis in subject subgroup, Figure 6A shows the proportion of ASD diagnoses for each subject subgroup by arranging fMRI parameters for each axis in a way that corresponds to Figure 3B (test error) and Figure 4E (emotion recognition index). Despite variations across subgroups, the trend in ASD diagnosis in the fMRI data set (Figure 6A) appears to be similar to ASD-like performance of neural network simulations (Figures 3B and 4G). Specifically, the overall trend is for ASD diagnoses to be more common in subgroups with higher ReHo (P = 0.021 for the main effect of logistic regression analysis). Furthermore, the impact of FC_MRI on ASD diagnosis appears to differ between the high and low ReHo subject groups. That is, in the high ReHo group there seemed to be more ASD patients with high FC_MRIs, while in the low ReHo group there seemed to be more ASD patients with low FC_MRIs (P = 0.025 for interaction effect in logistic regression analysis).

To further investigate the correspondence between the ASD-like performance of the neural network and ASD diagnosis in fMRI datasets, the scatterplot of test error and emotion recognition index created from the results of neural network simulation (Figure 6B) was colored to represent the ASD diagnosis of corresponding subjects based on the fMRI parameter (Figure 6C). In the histogram along the X-axis in Figure 6C, there appears to be a statistically significant trend (P = 0.003) toward more ASD among subjects corresponding to neural networks with larger test errors. Next, in the histogram in the Y-axis direction, the subject subgroups corresponding to neural networks with a large emotion recognition index tend to have more TDs, and this is statistically significant (P = 0.001). In the scatter plot in Figure 6C, clusters appear to be divided into the three groups, A, B, and C, shown in the figure, and the number of ASDs and TDs in each of these groups is shown in Figure 6D. There was a significant difference among the three groups (P = 0.007), with more TDs in the group with a high emotion recognition index and low test error (Group A) and more ASDs in the group with a low emotion recognition index and high test error (Group C).

While the above analysis showed an association between the ASD-like performance measure in neural networks and ASD diagnosis in fMRI datasets, we next hypothesized that emotion recognition performance in neural networks would be associated with impaired social interaction symptoms in ASD patients, because emotion recognition is essential for social interaction. To examine this hypothesis, we performed correlation analyses between ASD-like performance measures and each ASD symptom in the ASD population, evaluated by Autism Diagnostic Interview-Revised (ADI-R (Lord et al., 1994)) (Table 1). As we hypothesized, the emotion recognition index was negatively correlated with impaired social interaction symptoms (r = –0.123; 95% CI = –0.233 to –0.008) with a P-value of 0.034. This result suggests similarities between characteristics of neural network performance and corresponding characteristics of subject ASD symptoms.