On the Aggregation of Survey-Based Economic Uncertainty Indicators Between Different Agents and Across Variables

We analyse the effects of aggregating the level of disagreement in survey-based expectations. With this aim, we construct several indicators based on two metrics of disagreement: the standard deviation of the balance and a geometric measure of discrepancy. We use data from business and consumer surveys in eleven European countries and the Euro Area. We evaluate the dynamic response of economic growth to shocks in agents’ uncertainty gauged by the discrepancy measures in a bivariate vector autoregressive framework. We find that while the effect on economic activity to a shock in aggregate discrepancy is always negative for firms’ disagreement, the effect to consumers’ disagreement is positive in all countries except Italy. To shed some light regarding the effect of aggregating disagreement both across variables and economic agents on forecast accuracy, we also examine the predictive performance of the discrepancy indicators, using them to generate out-of-sample forecasts of economic growth. We do not find evidence that the aggregation of disagreement improves forecast accuracy. These findings are especially relevant when using cross-sectional dispersion of survey-based expectations of firms and households.


Introduction
The analysis of economic uncertainty has gained renewed interest since the Great Recession. Despite the evidence that uncertainty shocks have an effect on real activity (Baker et al. 2016;Bloom 2009;Paloviita and Viren 2014), the elusive nature of uncertainty and the difficulty of measuring it, has meant that until recently its impact on the economy were not further explored. Dibiasi and Sarferaz (2020) emphasise the importance of defining what is understood by economic uncertainty. Based on the different ways in which economic agents form their expectations regarding unknown future events, Knight (1921) differentiated risk from uncertainty. While under risk, agents are able to allocate probabilities over future outcomes, uncertainty would be defined as the state in which agents are no longer able to form expectations about future events. As noted by Rossi et al. (2020), disagreement on the probability distribution of future outcomes would be a special case of Knightian uncertainty, since disagreeing on probability distributions automatically implies that the probability distributions are not correctly specified.
The unobservable nature of economic uncertainty has given rise to different approaches to proxy it. Some authors have opted to gauge economic uncertainty by using the realized volatility in equity markets (Basu and Bundick 2017;Bekaert et al. 2013;Yıldırım-Karaman 2017), while others in oil and natural gas prices (Atalla et al. 2016;Hailemariam and Smyth 2019). Other authors have used the conditional volatility of the unforecastable components of a broad set of economic variables (Chuliá et al. 2017;Jurado et al. 2015). The ex-post nature of the latter approach has generated a strand of research that looks for ways to approximate economic uncertainty ex-ante. Most of this research makes use of quantitative expectations made by professional forecasters (Clements and Galvão 2017;Dovern et al. 2012;Krüger and Nolte 2016;Lahiri and Sheng 2010;Oinonen and Paloviita 2017). Jo and Sekkel (2019) used the forecast errors of consensus survey forecasts of various economic indicators to capture a real-time measure of uncertainty surrounding subjective forecasts. Rossi et al. (2020) proposed an uncertainty index based on density forecasts, which measures the distance, on average across forecasters, between the forecast distribution provided by an individual forecaster and the perfect forecast. As this proxy allows distinguishing between uncertainty and risk, the authors used it to analyse how both concepts relate to each other.
Recently, Altig et al. (2020) proposed measuring uncertainty using the subjective probability distributions of managers about their own firm outcomes at a 1-yearahead horizon. Castelnuovo (2019) and Cascaldi-Garcia et al. (2020) provide an overview of recent developments regarding the measurement of uncertainty. Kozeniauskas et al. (2016) distinguished between three types of uncertainty: micro uncertainty (firm-level shocks), macro uncertainty (aggregate shocks) and higher-order uncertainty (disagreement). Glas (2020) has carried out an in-depth analysis of the relationship between forecaster disagreement and macroeconomic uncertainty. Disagreement indicators based on survey expectations make use of prospective information, and in this sense are especially appropriate to evaluate the anticipatory properties of uncertainty proxies. One of the main sources of survey expectations are economic tendency surveys (ETS). Consequently, in recent years a growing number of studies have used the information coming from ETS to approximate uncertainty (Bachmann et al. 2013(Bachmann et al. , 2018(Bachmann et al. , 2020Binding and Dibiasi 2017;Claveria 2020;Dibiasi and Iselin 2019;Girardi and Reuter 2017;Meinen and Roehe 2017;Mokinski et al. 2015).
In ETS, agents' expectations are elicited by asking about the expected direction of economic variables. Respondents are asked whether they expect a broad set of variables to rise, fall or remain unchanged. Dibiasi and Iselin (2019) have recently proposed using the non-response category in forward-looking questions to directly approximate Knightian uncertainty. Their approach relies on firm-level data and measures the share of firms that do not formalise expectations about their future demand.
The most important ETS in Europe are the business and consumer surveys conducted by the European Commission. Firms are asked about production and other variables concerning developments in their sector, while households are asked about their spending intentions and the general economic situation influencing those decisions (price trends, unemployment expectations, etc.). We use information coming from both surveys to elicit agents' economic expectations in eleven European countries and the Euro Area (EA). By focusing on two independent surveys, the industry survey and the consumer survey, we can simultaneously measure disagreement about a wide range of economic variables for both firms and households, combining the perspective of the supply and the demand side of the economy. We use two alternative measures of expectations' dispersion to compute the level of disagreement in order to construct several indicators of discrepancy among firms' and households' expectations that gauge their perception of uncertainty.
This study contributes to the existing literature by analysing the effects of aggregating the level of disagreement across variables and different types of agents. We provide a comparative view of firms versus households of the dynamic relationship between innovations in their expectations about future economic uncertainty and the evolution of economic growth. We use a bivariate vector autoregressive (VAR) framework to compute the impulse response functions (IRFs) and to generate outof-sample forecasts of economic growth. We compare the forecasting performance of the uncertainty proxies and evaluate whether the aggregation of discrepancies among firms' and households' expectations, as well as across variables, helps to improve forecast accuracy.

Data and Methodology
The empirical analysis focuses on firms' and households' expectations about the future evolution of different economic variables. Of all the variables in the surveys, we focus exclusively on those with prospective information (Table 1). That is, we exclude all survey variables about the past or the present. We use monthly data from the joint harmonized EU industry and consumer surveys (European Commission 2020), and annual rates of change of the Gross Domestic Product (GDP) (Eurostat 2020). We have used linear interpolation for time disaggregation. The study focuses on Austria, Belgium, Finland, France, Germany, Greece, Italy, the Netherlands (NL), Portugal, Spain, the United Kingdom (UK) and the EA. The sample period goes from January 2003 to January 2020. The last 2 years are used as the out-ofsample period to evaluate forecast accuracy.
In the industry survey, manufacturers are asked about their expectations regarding production and other economic variables (Table 1), and they are faced with three options: "up", "unchanged" and "down". The aggregated percentages of the individual replies in each category are respectively denoted as P t , E t , and M t . Consumers, for their part, are asked about how they think the general economic situation or the price trends will change over the next months. Consumers have three additional response categories: two at each end ("a lot better/much higher/sharp increase", and "a lot worse/much lower/sharp decrease"), and a "don't know" option. We opt for grouping all positive responses in P t , all negative ones in M t , and incorporating the "don't know" share in E t .
Survey results are generally presented as balances, which are obtained as P t − M t . The most common indicators of disagreement among survey respondents use the dispersion of balances as a proxy for uncertainty (Bachmann et al. 2013;Girardi and Reuter 2017). Bachmann et al. (2013) proposed an indicator of disagreement based on the standard deviation of the balance: The omission of the information contained in E t led Claveria et al. (2019) to develop a disagreement metric that incorporates the information coming from all the reply options, whose number is denoted as N. Given that the sum of the shares of responses adds to a hundred, the authors compute an N-dimensional vector that aggregates the information from all answering categories ( V ), and project it as a point on a simplex of N − 1 dimensions that encompasses all possible combinations of responses.
For N = 3 , the simplex would take the form of an equilateral triangle ( Fig. 1), where V corresponds to the unique convex combination of three reply options for each period in time. See Claveria (2018) for an extension of the methodology for a larger number of reply options, and Claveria (2019) for an application of the methodology for N = 5.
Insomuch as all vertices are at the same distance to the centre of the simplex ( O ), the ratio of the distance of a point to the barycentre ( VO ) and the distance from the barycentre to the nearest vertex ( OP ) provides the proportion of agreement among respondents. Consequently, the indicator of discrepancy for a given period in time can be formalised as: This metric is bounded between zero and one, and conveys a geometric interpretation. The centre of the simplex corresponds to the point of maximum disagreement, indicating that the answers are equidistributed among all response categories. Conversely, each of the N vertexes corresponds to a point of minimum disagreement, where one category draws all the answers and G reaches the value of zero.
In this study we applied expressions (1) and (2) to measure the level of disagreement among firms' expectations in the industry sector ( DIS F t ) and among households' expectations ( DIS H t ). We used these two proportions to compute the mean level of disagreement of manufacturers and households, which we denote as "average discrepancy": We also construct an indicator of cross-sectional discrepancy between firms' and households' expectations: First, we computed expressions (3) and (4) for the question regarding firms' expectations about future production ( X1 F t ) and consumers' expectations about the general economic situation ( X1 H t ), which we respectively denote as AD Ec t and CSD Ec t . (2) Projection of the combination of the three reply options. Notes. V is the vector of the three aggregated reply options for a given period in time: P corresponds to the % of "increase" replies, M to the % of "fall", and E to the % of "remains constant". O represents the centre of the simplex (barycentre), which corresponds to the point of maximum disagreement 1 3 Then, we computed the average level of disagreement for all survey variables in Table 1, both for the industry survey ( AD F t ), and for the consumer survey ( AD H t ). Finally, we used the two previous indicators as the input to measure "average aggregate discrepancy" ( AAD t ) and "aggregate cross-sectional discrepancy" ( ACSD t ). All these discrepancy indicators are computed twice, using as inputs both expressions (1) and (2) to measure the level of disagreement.
In Figs. 2 and 3 we compared the evolution of the geometric measure of disagreement (G) to that of the standard deviation of the balance (D) in the EA. Figure 2 shows the evolution of both metrics for the question regarding firms' expectations about future production ( X1 F t ), and Fig. 3 for households' expectations about the general economic situation ( X1 H t ). Both series co-evolve, showing a high positive correlation. The main difference between both measures mainly lies in their average level and dispersion: G shows higher variability, and D is higher in most countries (see Table 2). By means of a simulation experiment, Claveria et al. (2019) showed that the omission of neutral responses resulted in an overestimation of the level of disagreement.
In Table 2 we compare the summary statistics for the different disagreement indicators. We found that the average degree of discrepancy among households' expectations ( AD H ) is higher than among firms' expectations ( AD F ) in all countries. This result may be in part attributable to the fact that there are differences in the horizon of the questions between the two surveys. Everything else equal, consumer disagreement should be somehow higher than firms' disagreement due to the effect that the longer time horizon of the questions may have. We also observed that the Netherlands, Portugal and Spain are the countries that show a higher level of cross-sectional disagreement in expectations. Table 3 contains the correlation coefficients between the different indicators of discrepancy for both measures of disagreement and the evolution of GDP (Eurostat 2020). In most cases, we found that average disagreement is negatively correlated with GDP growth both for firms and for households. As it could be expected, when the sign of the correlation of average discrepancy with economic growth differs between firms' and households' expectations, as it is the case in Greece and Portugal, the level of cross-sectional discrepancy (CSD and ACSD) shows a higher linear relationship with GDP growth. The rest of the study will focus on the comparison of average discrepancy indicators of firms and households ( AD F , AD H ) and the average between both (AAD).

Empirical Results
There exists empirical evidence on the bidirectional relationship between uncertainty and macroeconomic variables (Glocker and Hölzl 2019;Gupta et al. 2019;Ozturk and Sheng 2018). By means of a VAR approach, in this section we first analyse the dynamic response of economic growth to shocks in agents' perception of uncertainty gauged by the disagreement measures computed in the previous section. A linear VAR(p) model can be written as:     and AD H average discrepancy of firms' and households' expectations respectively, and AAD and ACSD aggregate average and cross-sectional discrepancy using all variables in Table 1  where y t is a vector of K endogenous variables that are modelled as a function of p lags of those variables; is a vector of intercept terms; p are the coefficient matrices and t is a white noise vector of innovations. In this study, we used the following bivariate VAR model: where Y t denotes the macroeconomic variable of reference (GDP growth), and X t the corresponding disagreement measure ( AD F , AD H , AAD). The number of lags of the dependent variables was selected by means of Schwarz's Bayesian information criterion (BIC). We want to note that some of the results may be conditioned by the setup of the analysis. As pointed out by Carriero et al. (2018), the fact that uncertainty measures are not fully embedded in the econometric models at the estimation stage might cause measurement errors in the regressors and lead to an endogeneity bias. Notwithstanding, Dibiasi and Sarferaz (2020), recently found that the endogeneity biases that may be caused by measurement errors in the regressors do not have a major effect.
Additional potential biases may also arise from the omission of variables due to restricted information sets in country-specific analysis. Some authors have included additional financial variables (Alessandri and Mumtaz 2019; Caldara et al. 2016). Other authors have circumvented this issue by assessing uncertainty shocks in a multi-economy context (Crespo et al. 2017;Hauzenberger et al. 2018). Sekhposyan (2015, 2017) introduced an index that allows computing country-level contributions and helps to analyse the heterogeneity of uncertainty across countries.
In Fig. 4 we compare the IRFs of output growth to innovations in disagreement among firms' expectations ( AD F ) and households' expectations ( AD H ) and to shocks average aggregate discrepancy (AAD).
In Fig. 4 we observe that shocks in average aggregate discrepancy always have a negative effect on economic growth. This result is in line with previous research While Sahinoz and Cosar (2019) recently found that Turkish firms' and consumers' uncertainties co-evolved, in most countries we observe differences between the effects of innovations in average disagreement of firms' expectations and average disagreement of consumers' expectations. Although shocks in aggregate discrepancy among firms always have a negative effect on economic growth, shocks in aggregate discrepancy among households have positive effects in all countries except Italy. The fact that we used the degree of disagreement in agents' expectations, and that there are differences in the subject matter and the horizon of the questions between the two surveys, could be explaining part of the obtained results.
(5) y t = + 1 y t−1 + 2 y t−2 + ⋯ + p y t−p + t (6) Y t X t =  Figure 5 presents the forecast error variance decomposition (FEVD) of GDP growth, which provides information about the relative importance of each innovation in affecting the forecast error variance. We can observe that in most cases, the fraction of the forecast error variance of GDP growth can be mostly attributed to orthogonalised shocks to itself. In Finland, Germany, Italy and Portugal the proportion attributable to innovations to the disagreement indicators are found to be larger than in the rest of the countries. In the literature, there is mixed evidence regarding the information content of disagreement among agents to refine predictions. On the one hand, Junttila and Vataja (2018) and Sorić and Lolić (2017) obtained improvements in forecast accuracy of predictions of economic activity in Croatia, the UK, the US, and the EA, when including uncertainty measures. On the other hand, Poncela and Senra (2017) did not find that uncertainty helped to refine predictions of GDP and inflation in the EA.
With the aim of shedding some light so as to whether the aggregation of the level of uncertainty across variables and across agents improves forecast accuracy, we examined the predictive performance of the different indicators of discrepancy, using them to generate out-of-sample forecasts of economic growth. We designed a 24-period ahead forecasting experiment in order to assess forecast accuracy. We used the last 2 years of the sample to compute the root mean squared forecast error (RMSFE): where e t refers to the forecast error at time t. We also ran the Diebold-Mariano (DM) test of forecast accuracy (Diebold and Mariano 1995) to evaluate whether there are significant differences in forecast accuracy and computed the Harvey-Leybourne-Newbold (HLN) statistic (Harvey et al. 1997), which is a modification for small samples of the DM statistic. Under the null hypothesis that there is no significant difference in precision, the HLN statistic follows a Student-t distribution. A negative sign of the statistic implies that the second model has bigger forecast errors. Table 4 summarises the results of the forecasting exercise to evaluate the role of discrepancy measures in anticipating economic growth when introducing them as covariates.
On the one hand, when comparing AD F versus AD H , we obtained mixed results across countries. Only in France, Netherlands, Portugal, UK and the EA forecast errors are significantly lower with the indicator of average discrepancy among firms' expectations. We want to note that these results may be in part due to the fact that some countries may have experienced more uncertainty shocks on the supply sidecountries in which average discrepancy among firms improves forecasting accuracy-, than others where most of all the uncertainty shocks where experienced by households (demand side).
On the other hand, when comparing AD Ec versus AAD , that is average discrepancy for the question regarding firms' expectations about future production and consumers' expectations about the general economic situation ( AD Ec ) to average aggregate discrepancy ( AAD ), we also found differences across countries. Only in half of the cases the aggregation of disagreement across variables helps improve forecast accuracy.
In this sense, Morikawa (2019) analysed the uncertainty of production forecasts, obtaining heterogeneous forecast errors among individual manufactures and sectors. In a recent study, Claveria (2020) evaluated the dynamic response of different macroeconomic variables to shocks in agents' perception of three dimensions of uncertainty (economic, inflation and employment), and found that the effects of shocks to agents' perception of uncertainty on economic aggregates are shown to be variable-specific and dependent on the type of agent. The obtained results in the forecasting exercise add to this notion, which suggests that the aggregation of the level of discrepancy, both across variables and between different types of agents, may not help improve the predictive capacity of uncertainty proxies, especially when agents' expectations regarding the evolution of the economy tend to diverge, something that is more likely to occur during periods of major uncertainty.

Conclusion
In this study we have analysed the effect on economic growth of shocks in the perception of uncertainty gauged by different indicators of disagreement in survey expectations. We have used qualitative data about the expected direction of change in production and a set of economic variables to compute the level of disagreement between firms' and households' expectations in eleven European countries and the Euro Area, using different discrepancy indicators. First, we have found that the average degree of discrepancy among households is higher than among firms in all countries.
Second, we have assessed the dynamic relationship between innovations in the different indicators of discrepancy and the evolution of economic activity. We have found that, while shocks in aggregate disagreement among firms' expectations have a negative effect on economic growth, the effect on economic activity to a shock in aggregate discrepancy among consumers' expectations is positive in all countries except Italy. Finally, in order to shed some light regarding the effect on forecast accuracy of aggregating disagreement both across variables and economic agents' expectations, we have examined the predictive performance of the discrepancy indicators. We have generated out-of-sample forecasts of economic growth using the indicators as covariates. When testing for differences in accuracy, we have not found evidence that the aggregation of disagreement across firms' and households' survey expectations helps improve forecast accuracy. These findings are of special relevance for researchers when using cross-sectional dispersion of survey-based expectations, since the effects of shocks to agents' perception of uncertainty on economic growth may vary depending on the country and the phase of the cycle.
We want to note some of the limitations of the present study. Above all, it should be highlighted that, the findings in this research could be partly attributable to differences between the two surveys in the subject matter and the horizon of the questions. In this sense, the main aim of this research was to point at the effects of aggregating disagreement in expectations across economic variables and different types of agents. An issue left for further research is the extension of the analysis to other surveys and countries. Other opened lines of research include the extension of the methodological framework, using alternative measures of disagreement, and applying new developments in VAR analysis.
Funding This research received no financial support.

Availability of Data and Material
The datasets used and/or analysed during the current study are: The Joint Harmonised EU Consumer Survey conducted by the European Commission, which can be freely downloaded at: https ://ec.europ a.eu/info/busin ess-econo my-euro/indic ators -stati stics /econo mic-datab ases/busin ess-and-consu mer-surve ys_en. Gross Domestic Product provided by Eurostat at: https :// ec.europ a.eu/euros tat/web/natio nal-accou nts/data/main-table s.

Compliance with Ethical Standards
Conflict of interest The authors declare that they have no competing interests.
Code Availability Not applicable. Figure 6 contains the evolution of the disagreement indicators presented in Sect. 2 computed with the geometric indicator of discrepancy (G).    AD denotes the geometric measure of disagreement-aggregate disagreement for firms ( AD F ), for households ( AD H ), or average aggregate discrepancy ( AAD)