Uncertainty indicators based on expectations of business and consumer surveys

In this study we evaluate the dynamic response of different macroeconomic variables to shocks in agents’ perception of three dimensions of uncertainty (economic, inflation and employment). First, we apply a geometric indicator to compute the proportion of disagreement in business and consumer expectations of eight European countries and the Euro Area. Next, we use a bivariate vector autoregressive framework to estimate the impulse response functions to innovations in disagreement. While we find an adverse reaction in unemployment rates to shocks in discrepancy, results differ markedly between disagreement in business and in consumer surveys with regard to economic growth and inflation: shocks to manufacturing production discrepancy lead to a decrease in economic activity, as opposed to shocks to consumer economic discrepancy; and the opposite in the case of a shock in the perception of price uncertainty. Finally, we perform a forecasting exercise to assess the predictive performance of the disagreement indicators for different time horizons, obtaining more accurate out-of-sample recursive forecasts of economic growth with the indicators of discrepancy of manufacturing firms and, of unemployment with the indicators of consumer discrepancy. When compared to recursive autoregressive predictions used as a benchmark, we find that vector autoregressions with industry discrepancy tend to outperform the benchmark in more cases that models with indicators of consumer discrepancy.


Introduction
The analysis of economic uncertainty has gained renewed interest since the advent of the 2008 financial crisis. While there is a widespread consensus that uncertainty shocks have an effect on real activity (Bachmann and Bayer 2013;Baker et al. 2016;Bloom 2009;Paloviita and Viren 2014;Zarnowitz and Lambros 1987), there are several strategies to measure uncertainty. Since economic uncertainty is not directly observable, some authors have opted to proxy it by using the realized volatility in equity markets (Basu and Bundick 2017;Bekaert et al. 2013;Caggiano et al. 2014;Yıldırım-Karaman 2017) or in oil and natural gas prices (Atalla et al. 2016;Hailemariam and Smyth 2019). Other authors have used econometric unpredictability, understood as the conditional volatility of the unforecastable components of a broad set of economic variables (Chuliá et al. 2017;Jurado et al. 2015;Meinen and Roehe 2017). The ex-post nature of this latter approach, has recently generated a strand of the empirical research that makes use of survey-derived measures of expectations dispersion (Binder 2017;Binding and Dibiasi 2017;Clements and Galvão 2017;Krüger and Nolte 2016).
Disagreement measures based on survey expectations make use of prospective information, as agents are asked about the expected future evolution of a wide range of variables. The ex-ante nature of survey expectations make them especially appropriate to evaluate the anticipatory properties of disagreement-based uncertainty indicators. While most studies rely on quantitative macroeconomic expectations made by professional forecasters (Dovern 2015;Lahiri and Sheng 2010;Mankiw et al. 2004;Oinonen and Paloviita 2017), an alternative source of survey expectations are business and consumer tendency surveys Claveria et al. 2019;Girardi and Reuter 2017;Meinen and Roehe 2017;Mitchell et al. 2007; Mokinski et al. 2015).
The European Commission conducts monthly business and consumer tendency surveys in which respondents are asked whether they expect a set of variables to rise, fall or remain unchanged. Firms are asked about production, selling prices, employment and other variables concerning developments in their sector, and households are asked about their spending intentions and the general economic situation influencing those decisions (price trends, unemployment expectations, etc.). We use the information coming from both surveys to elicit agents' expectations about production and economic activity, prices, and employment in eight European countries: Austria, Belgium, Finland, France, Germany, the Netherlands (NL), Spain, and the United Kingdom (UK).
In this research we use qualitative survey data from two independent tendency surveys conducted by the European Commission, the industry survey and the consumer survey. This dual approach allows us to simultaneously measure disagreement about economic activity, prices and employment in both business and consumer expectations. We use Claveria et al.'s (2019) geometric indicator of discrepancy to compute agents' perception of uncertainty. This study contributes to the existing literature by providing a comparative view of firms versus consumers of the dynamic relationship between the perception of three different dimensions of uncertainty (economic, inflation and employment) and the evolution of the corresponding aggregates.
We apply a bivariate vector autoregressive (VAR) framework to analyse the dynamic response of the evolution of economic activity, prices and unemployment rates to innovations in each type of uncertainty. We use the VAR models to generate out-of-sample recursive forecasts of the three macrovariables in order to assess the forecasting performance of the uncertainty proxies. Finally, we also run Grangercausality tests to evaluate whether the proposed uncertainty measures are useful in forecasting their respective reference series.
The paper is organised as follows. The next section introduces the data and the methodological approach. Empirical results are provided in Sect. 3. Finally, concluding remarks and future lines of research are drawn in Sect. 4.

Data
The empirical analysis focuses on manufacturing firms' and consumers' expectations about the future evolution of economic activity, inflation and unemployment. We use monthly data from the joint harmonised EU industry and consumer surveys conducted by the European Commission. Regarding the quantitative information, we use annual rates of change of the gross domestic product (GDP) and the harmonised index of consumer prices (HICP) provided by Eurostat, and the unemployment rates of the OECD. The sample period goes from May 2005 to December 2017. The last 2 years are used as the out-of-sample period to evaluate forecast accuracy.
In the survey, manufacturers are asked about their expectations regarding production, selling prices and employment for the months ahead, and they are faced with three options: "up", "unchanged" and "down". The aggregated percentages of the individual replies in each category are respectively denoted as P t , E t , and M t . Consumers, for their part, are asked how they think the general economic situation, the cost of living, and the level of unemployment in the country will change over the next twelve months. Consumers have three additional response categories: two at each end ("a lot better/much higher/sharp increase", and "a lot worse/much lower/sharp decrease"), and a "don't know" option. We opt for grouping all positive responses in P, all negative ones in M, and incorporating the "don't know" share in E for each time period.

Measurement of uncertainty
The most common way of presenting survey results is the balance, obtained as P t − M t . The most widespread measures of disagreement among survey respondents use the dispersion of balances as a proxy for uncertainty Girardi and Reuter 2017).  proposed an indicator of disagreement based on the square root of the variance of the balance:

3
The omission of the information contained in the "no change" category led Claveria et al. (2019) to develop a disagreement metric that incorporated the information coming from all the reply options, whose number is denoted as N. Given that the sum of the shares of responses adds to a hundred, the authors compute an N-dimensional vector that aggregates the information from all answering categories and project it as a point on a simplex of N − 1 dimensions that encompasses all possible combinations of responses. For N = 3 , the simplex takes the form of an equilateral triangle (Fig. 1), where the point V corresponds to a unique convex combination of the three reply options for each period in time.
Insomuch as all vertices are at the same distance to the centre of the simplex ( O ), the ratio of the distance of a point to the barycentre ( VO ) and the distance from the barycentre to the nearest vertex ( OP ) provides the proportion of agreement among respondents. Consequently, the indicator of discrepancy for a given period in time can be formalised as: This metric is bounded between zero and one, and conveys a geometric interpretation. The center of the simplex corresponds to the point of maximum disagreement, indicating that the answers are equidistributed among the three response categories. Conversely, each of the N vertexes corresponds to a point of minimum disagreement, where one category draws all the answers and D t reaches the value of zero.
In this study we apply expression (2) to measure discrepancy in manufacturing surveys ( D I t ) and in consumer surveys ( D C t ), and we compare it to the standard Projection of the combination of the three reply options. Notes: V is the vector of the three aggregated reply options for a given period in time: P corresponds to the % of "increase" replies, M to the % of "fall", and E to the % of "remains constant". O represents the centre of the simplex (barycentre), which corresponds to the point of maximum disagreement deviation of the balance (1). We use both expressions to gauge the perception of uncertainty regarding economic activity, inflation and unemployment. Table 1 contains the summary statistics of disagreement in business and consumer surveys for each indicator and each of the variables: production/economic situation, selling prices/cost of living and employment/unemployment. For all variables the average degree of consumer disagreement is higher than manufacturing disagreement. When comparing both measures of disagreement, DISP and D, we observe a lower level of dispersion for the standard deviation of the balance. This notion is further confirmed in Figs. 2, 3 and 4, where we compare the evolution of both measures for all survey questions. We also observe different patterns for each type of disagreement. Table 2 contains the bivariate correlations between the evolution of the quantitative aggregates and their corresponding measures of discrepancy, as well as among the different measures of discrepancy. To facilitate the comparison between both surveys we use GDP growth instead of industrial production for the manufacturing survey. While the evolution of economic activity is negatively correlated with manufacturing production disagreement for all countries, we obtain mixed results for consumers. A similar result is observed for unemployment. On the contrary, inflation is positively correlated  with manufacturing price disagreement, and negatively with consumer price disagreement. We obtain low and mainly negative correlations between disagreement in business surveys and in consumer surveys. We want to note that, to some extent, the discrepancies between firms and consumers can be partly attributable to differences in the questions in both surveys: while consumer survey questions refer to objective variables, business surveys questions refer to firm-specific factors.

Empirical results
There exists empirical evidence on the bidirectional relationship between uncertainty and macroeconomic variables (Alessandri and Mumtaz 2019; Glocker and Hölzl 2019; Gupta et al. 2019;Mumtaz and Musso 2019). By means of a VAR approach, in this section we first examine the dynamic relationship of the discrepancy measures computed in the previous section to gauge the perception of uncertainty and the corresponding macromagnitudes. As we are estimating independent vector autoregressions per country and no spillover effects are considered, we introduce an index i = 1, … , N to denote the N countries analysed in the study. We use the following bivariate model per country: where D • , it refers to the proposed disagreement measure for businesses (B) and consumers (C) respectively and, z it refers to the macroeconomic variable of reference: output growth, inflation and unemployment for the i-th country at time t (t = 1, … , T) . The number of lags, p, is selected by means of Schwarz's Bayesian information criterion (BIC). We use heteroscedasticity-consistent standard errors for the estimation. Thus, in the resulting two-variable VAR models each of the uncertainty measures (three for the consumer survey and three for the business survey) is related to its macroeconomic reference series.
In Figs. 5, 6, and 7 we compare the estimated impulse response functions (IRFs) of output growth, inflation and unemployment rates to innovations in consumers' and manufacturers' perception of uncertainty as captured by the discrepancy measures.
While Sahinoz and Cosar (2019) have recently found that Turkish firms' and consumers' uncertainties coevolve, our analysis of the dynamic effects of shocks in the perception of three different types of uncertainty on their respective aggregates shows an asymmetric response between innovations in the disagreement in business surveys and in consumer surveys. A one standard deviation shock to manufacturing production and selling prices discrepancy leads to a fall in output growth and an increase in inflation. This result is in line with previous research (Alexopoulos and Cohen 2015;Cerda et al. 2018;Charles et al. 2018;Istiak and Serletis 2018;Meinen and Roehe 2017).
On the contrary, a surprising result is that in most countries a one standard deviation shock to consumers' perception of uncertainty about economic activity and price trends respectively leads to an increase in output growth and a decrease in inflation. This finding is partially in line with the results obtained by Morikawa (2019), who analysed the uncertainty of production forecasts and found heterogeneous forecast errors among individual manufactures and sectors. In this sense, Henzel and Rengel (2017) showed that different dimensions of uncertainty have diverse effects on aggregate fluctuations of the economy.
In the case of unemployment, shocks in unemployment disagreement, which tend to be of smaller magnitude, lead to a decrease in unemployment rates in most countries. Caggiano et al. (2017) and Netšunajev and Glass (2017) found evidence that unanticipated increases in uncertainty negatively affected the evolution of unemployment in the United States (US) and the EA.
We want to note that some of these results may be conditioned by the setup of the analysis. As recently pointed out by Carriero et al. (2018), the fact that uncertainty measures are not fully embedded in the econometric models at the estimation stage might cause measurement errors in the regressors and lead to an endogeneity bias. Additional potential biases may also arise from the omission of variables due to restricted information sets in country-specific analysis. Some authors have

3
Empirica (2021) Fig. 7 IRFs of GDP, HICP and unemployment to shocks in disagreement. Notes: Shaded area represents the 90% bootstrap confidence interval. 24-month forecast horizon. UK refers to the United Kingdom and EA to the Euro Area circumvented this issue by assessing uncertainty shocks in a multi-economy context (Crespo et al. 2017;Hauzenberger et al. 2018;Mumtaz and Theodoridis 2017). Bloom (2014) noted that economic uncertainty varies heavily across countries. Ozturk and Sheng (2018) found that common uncertainty shocks have larger and more persistent effects on economic activity than variable-specific and country-specific innovations.
In this sense, Sekhposyan (2015, 2017) introduced an index that allows to compute country-level contributions and helps to analyse the heterogeneity of uncertainty across countries. Other authors have included additional financial variables (Alessandri and Mumtaz 2019;Caldara et al. 2016;Gilchrist et al. 2014).
In a recent study, Glocker and Hölzl (2019) used a direct measure of uncertainty based on a business survey for the Austrian economy in which firms are asked directly about their degree of certainty. The authors evaluated the extent to which uncertainty is useful for forecasting and found that although uncertainty shocks are important in shaping macroeconomic fluctuations indirect measures tended to underestimate the effects on GDP and other macroeconomic aggregates.
In the literature, there is mixed evidence that the information coming from the degree of disagreement among agents helps to refine predictions. While Poncela and Senra (2017) did not find that uncertainty helped to refine predictions of GDP and inflation in the EA, Junttila and Vataja (2018) obtained improvements in forecast accuracy of predictions of economic activity in the UK, the EA and the US when including uncertainty measures. Similarly, Sorić and Lolić (2017) computed several uncertainty proxies for the Croatian economy and found that a vast part of the analysed indicators were significant predictors of economic activity. In the case of employment, Sakutukwa and Yang (2018) found that macroeconomic uncertainty contained useful information to forecast employment, especially in the construction and manufacturing industries. These results are in line with those obtained by Claveria (2019) for unemployment rates. To evaluate whether the proposed uncertainty measures are useful in forecasting their respective reference series we run Granger causality tests. We find evidence that disagreement in business surveys Grangercauses macroeconomic aggregates in most cases, especially for economic growth and inflation, while the opposite happens for disagreement in consumer surveys. Results are included in Table 7 of the "Appendix".
With the aim of further assessing the predictive power of the indicators of disagreement used to gauge the perception of uncertainty, we use the VAR models to generate out-of-sample recursive forecasts for different forecast horizons (h). All models are estimated from 2005:05 to 2015:12 and forecasts for 1, 2, 3, 6 and 12 months ahead are computed. The specification of the models is based on information up to that date and then re-estimated each period. Forecast errors are computed in a recursive way. We use the last 2 years of the sample as the out-of-sample period. To summarise this information we compute the root mean squared forecast error (RMSFE), where e t refers to the forecast error at time t: Empirica (2021) 48:483-505 In Tables 3 and 4 we present the average RMSFE values for the out-of-sample  period, while Tables 5 and 6 contain the relative RMSFE, computed as the ratio between the RMSFE of the VAR models with the disagreement measures and the RMSFE of the benchmark model. As a benchmark model we use an autoregressive process of order one and compute out-of-sample recursive forecasts for the different forecast horizons.
When compared with the benchmark model, we find that vector autoregressions with industry discrepancy tend to outperform recursive autoregressive predictions in more cases that models with indicators of consumer discrepancy. When comparing the predictive performance of disagreement in business and consumer surveys for different time horizons, we observe similar results across agents, especially at short forecast horizons. However, as the predictive horizon increases, so do the differences in the obtained forecast accuracy. This result is in line with Patton and Timmermann (2010), who found that the disagreement between forecasters is highest at long horizons where private information is of limited value. In general, for all time horizons we obtain more accurate predictions of economic growth with manufacturing production discrepancy, and of unemployment with consumer unemployment discrepancy, save for Germany, where the lowest RMSFE values are always obtained when using measures of disagreement in business surveys. With the exception of Spain, the most accurate predictions are usually obtained for unemployment rates.
We want to note that these results can be partly attributable to the measurement error arising from the exogenous measurement of uncertainty, to the omitted variable bias and to differences between the two surveys in the formulation of the questions. Neither the subject matter nor the horizon of the questions are identical in each case. While consumer survey questions refer to objective variables, business surveys questions refer to firm-specific factors.

Conclusion
This study analyses the effect on macro aggregates of shocks in the perception of three dimensions of uncertainty: economic, inflation and employment uncertainty. We use qualitative data about the expected direction of change in economic activity, prices, and employment to compare the level of disagreement between business and consumer surveys in eight European countries and the Euro Area. Agents' perception of uncertainty is gauged by a geometric indicator of disagreement in survey expectations. First, we assess the performance of the disagreement indicator by comparing it with the standard deviation of the balance. We observe that the patterns of evolution of disagreement differ between the three variables analysed, although for economic activity, as well as for prices and employment, we find that the average degree of consumer disagreement is greater than that of manufacturers.
The dynamic relationship between innovations in the different dimensions of perceived uncertainty (economic, inflation and unemployment) and their corresponding aggregates is assessed by estimating the impulse response functions in a bivariate vector autoregressive framework. With respect to unemployment rates, in most countries we observe an adverse reaction to shocks in the perception of unemployment uncertainty. However, the results differ markedly between disagreement in business and in consumer surveys for economic activity and inflation. In the case of output growth, we find that shocks to manufacturing production discrepancy lead to a decrease in economic activity, as opposed to shocks to consumer economic discrepancy. In the case of inflation, we obtain the opposite results: the effect on inflation of a shock in the perception of price uncertainty among consumers is found to be negative, and positive in terms of disagreement in sales prices. This finding is of special relevance for researchers when using cross-sectional dispersion of survey-based expectations, since the effects of   shocks to agents' perception of uncertainty on economic aggregates are shown to be variable-specific and dependent on the type of agent.
With the aim of assessing the predictive power of the indicators of disagreement we perform a pseudo out-of-sample forecasting exercise, generating recursive forecasts of the three macroeconomic variables for different forecast horizons using the proxies of perceived uncertainty. When comparing the predictive performance of disagreement between business and consumer surveys, we obtain similar results at short horizons, but we observe that differences in predictive accuracy increase for longer time horizons. In general, for all time horizons we obtain more accurate predictions of economic growth with manufacturing production discrepancy and, of unemployment with consumer unemployment discrepancy, save for Germany, where the most accurate forecasts are always obtained when using measures of disagreement in business surveys. When compared to recursive autoregressive predictions used as a benchmark, we find that vector autoregressions with industry discrepancy tend to outperform the benchmark in more cases that models with indicators of consumer discrepancy.
Finally, we run Granger causality tests to evaluate whether including past values of uncertainty measures improves predictions of their respective reference series based only on their own past values. We find evidence that disagreement in business surveys Granger-causes macroeconomic aggregates in most cases, especially for economic growth and inflation, while the opposite happens for disagreement in consumer surveys.
We want to note some of the limitations of the present study. It should be highlighted that the findings of this research may be conditioned by several biases derived from the exogenous measurement of uncertainty and the omission of variables, as well as from the differences between the two surveys, since the questions of the consumer survey refer to objective variables, while the questions of the business survey refer to specific factors of the firm. The main aim of the research is to point at potential differences in the perception of uncertainty across agents. In this sense, an issue left for further research is the extension of the analysis to other variables included in the surveys such as order-book levels, stocks of finished products, exports or major purchases and savings. Other lines of future research include the extension of the methodological framework: using other measures of disagreement, applying new developments in VAR analysis, as well as generating density forecasts with models including alternative proxies of uncertainty. Empirica (2021) 48:483-505