Local e-government Benchlearning : Impact analysis and applicability to smart cities benchmarking

We claim that local e-services benchmarking studies summarized in indexes do little to enhance city managers’ and academics’ understanding of actual e-government performance, or to improve the e-services offered by cities. We undertook a different benchmarking approach, focused on learning best practices among cities, in late 2008 and early 2009. A benchlearning methodology (BLM) was developed, and a pilot study with 15 European cities was carried out. In this paper, we present the actual impact of the benchmarking study with respect to improvements in services, as the effectiveness of e-government benchmarking has rarely been evaluated. We discuss and analyse the results of a survey carried out in the same 15 cities four years after the pilot study. This paper presents evidence that BLM helped cities to identify good practices that they could learn from, and that some e-services were subsequently improved. The survey reveals that some changes are needed in the benchmarking methodology. The main one is the updating of the BLM bottom-up e-services catalogue, which is deeply discussed within the changing context of Smart Cities, especially the enlargement of the ecosystem of e-services to include citizens, the third sector, entrepreneurs, companies and other actors. A second one is the measurement of the adoption of e-services by citizens, also rarely assessed.


Introduction
Both practitioners and academics are interested in benchmarking e-government services, because eservices and websites are the most visible parts of e-government and have the greatest impact [33].Some well-accepted e-government stage models [7,9] facilitate the performance of e-services maturity assessments [23,28,33,42,44], which are a widespread benchmarking strategy.However, a literature review shows that e-government benchmarking mostly addresses national level, and its granularity is not useful for cities [5].
Cities are gaining importance in the global economy and play a key role in service provision, but benchmarking studies of cities' e-government are scarce.Some reasons for the lack of local benchmarking methodologies and rigorous cities surveys are the diverse socioeconomic environments and contexts

Related work
Benchmarking means evaluation through comparison.In this paper we are concerned with benchmarking local e-government in a way that is useful for city managers and researchers alike.An in-depth review of the most important e-government benchmarking studies shows that very little attention has been paid to local public administration.Not a lot has changed since this was noted by [42].However, quite a few comparisons have been performed at state level, particularly [11,36,48,49,64,65].Recently [63] introduced multicriteria decision support incorporating better stakeholders' own viewpoints and preferences.The few studies that address local e-government do not deal with the complexity resulting from the richness of local e-services [31], which is a key element of local e-government and a focus of our study.Some relevant exceptions are Kaylor [37], Shackleton et al. [60], Flak et al. [20] and Löfstedt [46].

A U T H O R C O P Y
It might be argued that the diversity of cities could hinder any useful comparisons [16], but research shows that cities around the world actually share a lot of services.A field study [53] on how seven major European cities provide an advanced level of transactional e-services revealed a panel of seventeen common services.More recently, [22] showed strong evidence of commonalities in city e-government services at a much wider international scale.Thus, there should be room for cities e-government benchmarking, based on a set of similar services that are delivered, rather than a comparison of similar organizations [32].Consequently, local e-government transnational benchmarking looks feasible.
Although we can only fully appreciate the actual impact of e-services on citizens by addressing provision and adoption [49], most e-government benchmarking studies focus only on the delivery of eservices by administrations, and pay no attention to their real use by citizens.Scholars and international organizations have repeatedly requested studies of this kind [15,28,33,34,41,51].However, little progress has been achieved to date.The most recent benchmarking studies, carried out by CapGemini et al. [10], the OECD [50] and the United Nations [62,65,66], indicated that service adoption is key to assess egovernment performance, but did not measure this factor.
Most city benchmarking studies result in rankings according to indexes.Moonen and Clark [47] identified up to 150 city indexes, benchmarking studies and comparative rankings, based on macroeconomic indicators or subjective surveys, with samples ranging between 6 and 2,000 cities.Recently, similar rankings of cities according to their smartness, intelligence or sustainability have appeared.The most rigorous approaches come from academia or economic research institutes.They are based on aggregate indexes, resulting from weighted multiple indicators.An example is the European Smart Cities Index [25], which was designed by 3 universities. 1It considers 6 characteristics by which cities could be "smart": economy, mobility, environment, people, living and governance, leading to 74 indicators to be measured.This index was first used on around 70 European mid-sized cities in 2007.Other rankings are less rigorous, as they do not provide information about the selection of the sample, they use a small number of indicators, or they lack transparency in the data gathering method and index calculation, as already discussed by [24].
Rankings based on aggregated indexes have often been criticized by academia due to the difficulty of designing a well-grounded method for them [3].More recently, Rorissa et al. [56] highlighted that the computation of current e-government indexes has significant limitations, as it does not differentiate e-government development levels or national dimensions and their development.These limitations are even greater for smart cities indexes, as there is no widely accepted definition of a smart city [18].Giffinger and Gudrun [24] stated that rankings are quantitative approaches that concentrate on the aspects that are measurable, instead of the important issues.A further limitation results from the adaptation of indexes to what is available in databases (e.g. the Global Power City Index).According to [63], rankings should be based on transparent computational procedures to maximize their acceptability by both governments and the scientific community, leading to frameworks and indexes that achieve wide consensus.However, Kourtit et al. [39] claim that no list of indicators will ever be complete or fitfor-purpose.In the end, rankings are tools for city marketing, rather than being useful for improving e-government.
Batlle et al. [4] introduced BLM to address the lack of useful benchmarking methods for local egovernment.Here, we discuss the main (novel) characteristics of BLM, to ground the design of the survey questions and the survey analysis. 1The Centre of Regional Science (Vienna University of Technology), the Department of Geography (University of Ljubljana) and the Research Institute for Housing, Urban and Mobility Studies (Delft University of Technology).

A U T H O R C O P Y
The new method takes into account the multiplicity and variety of e-services in cities, and proposes a bottom-up approach (as suggested later by the UN [64]) to build a hierarchical catalogue of services, also aligned with the much more recent [32].Through participatory data collection on the services provided in each city, up to 81 common and less common services were identified.These services were classified into nine thematic categories (Channelling, Citizens' Engagement, Education, Employment and Business, Environment, Life-cycle, Social Care, Transport and Urban Planning).The citizens' engagement category includes the online services that open up channels to citizens for online participation in public affairs [65].
To measure the maturity of e-services, the well-known Gartner Model [7] was adopted for the BLM, and extended with a fifth level of maturity to cope with richer citizens' interaction and involvement in eservices provision through web 2.0.BLM focuses on citizens' interaction, and its model is quite similar to Lee's [44], which is the revisited version of Kayne and Lee [43], and can be considered a simplified version of the Manchester e-Government Maturity Model [29] in which citizen-led service provision is at higher maturity stages.
BLM assesses both the provision and adoption of e-services.While most research focuses on understanding the drivers of citizens' use of e-government and users' satisfaction [38], BLM addressed the lack of comprehensive data on actual e-government usage, and introduces a simplified measurement framework.Expected versus actual adoption is estimated on the basis of self-assessment surveys after training (which is a widely used technique that is accepted in e-government measurements, as argued in [6]).
Finally, instead of presenting a ranking of cities based on an aggregated index, BLM uses a novel city chart display approach, to try to offer decision-makers materials that are better suited to informing decisions on e-government strategies.As a result of the data analysis and reporting, a display of each city's profile is generated according to the provision and adoption of the thematic categories of services (with respect to average values), and the objectively measured best practices in different categories.
The methodology and the results of the first pilot benchmarking study involving 15 diverse European cities are discussed in depth in [6].
A further step towards the validation of a methodology, or even a repeated benchmarking exercise, is the analysis of its real impact on cities e-government.However, this aspect has received very little attention in the e-government benchmarking literature.Janssen's [33] critique of two of the most important national-level benchmarking studies, CapGemini Ernst & Young and Accenture e-Government Benchmarking, provides insights into how to increase quality and usefulness, for example by including the demand side or take-up as a criterion, taking into account the multichannel approach, adopting a citizen-centric rather than an organization-centric logic, adapting the maturity framework to include proactive services, and including services at regional and local level.BLM addresses most of these issues.Heeks [28] examines the real use of e-government benchmarking data, and concludes that there is a lack of evidence of the use and actual impact of such comparisons.Bannister [3] discusses the difficulties of carrying out rigorous and useful e-government comparative evaluations, and [3] and [33] highlight the unhealthy effects of repeating some benchmarking studies, such as [9], because organizations will often adapt their behaviour to what is being measured and design policy around scoring better instead of around citizens' needs.Indeed, Schellong [59] analysed CapGemini Ernst & Young, United Nations and Brown University benchmarking studies and concluded that they may lead policy makers to allocate resources to improve country rank, rather than investing in infrastructure, e-participation or other areas that are important to citizens.
According to [56,63], the impact of e-government benchmarking falls into three categories: (1) measuring retrospective achievement, which helps policy makers to compare how their country or agency

A U T H O R C O P Y
ranks in terms of e-government; (2) charting prospective direction/priorities that policy makers can use to make strategic decisions and identify appropriate courses of action; and (3) holding government decision makers accountable for investments in e-government.
The survey analysed in this paper tries to decide whether BLM as a benchmarking study actually helped city managers to improve e-services, basically aligned with category (2).As indicated earlier, the survey took place four years after the benchmarking study, and was administered in the same cities that took part in the BLM.The aim of the survey was also to determine the effectiveness of the new features of the BLM, and to put these features in the framework of the changing landscape of local e-government in the progress towards smart cities.
Thus, the results discussed below provide empirical evidence that helps us to understand the advantages and limitations of the methodology, and to design more useful local e-government benchmarking studies.

Techniques of the BLM impact survey
The aim of the BLM and the pilot study was to design a benchmarking method that could help to improve city e-services.Consequently, it was logical to then assess whether the actual pilot benchmarking study had led to any improvements, and whether the methodology itself needed to be revamped.We undertook this evaluation and analysis in 2013-2014, through a survey of the 15 cities2 that had taken part in the original BLM.Ten of the cities responded, i.e. approximately 2/3.In addition, a city responded to the survey that had not participated in the pilot study, but had benefited from the final benchmarking report.
An online survey was administered to the 15 cities that took part in the benchmarking.It was also sent to four other cities that had initially been interested in participating in the 2008 benchmarking but did not take part for various reasons.High staff mobility in city administrations meant that the relevant people could have changed jobs in the period between the pilot study and the survey.We identified those in charge when the survey was administered, and sent them a link to the survey, along with an explanatory e-mail.
In the survey, the first question identified the respondent and was followed by a block of 3 questions on the impact of the benchmarking study on e-services improvement.The next block was about the new context of cities, and suggestions for improvements.Two final questions were about participation in a new benchmarking study.Most of the ten questions had several parts, and respondents were guided through multiple choice options, giving a total of 54 'simple' questions.Free text input was allowed only for clarifications or suggestions.The aim of simple questions is to obtain a lot of detail, to compensate for the subjective nature of the survey as much as possible.This seems to have worked in view of the results of consistency checks (on answers and raters).
The statistics (Table 1) show considerable internal consistency of the answers.The Kuder-Richardson Formula 20 test was passed by all dichotomous questions, with a KR20 reliability index of 0.56.Questions based on Likert scales were analysed according to Cronbach's Alpha reliability test.Q4 and Q8 had a high Alpha value of over 0.85; the Alpha value for Q9 was 0.65, which is moderate, but valid.

Results
From the 15 + 4 cities contacted, 10 of those which had participated in the benchmarking and 1 which had been interested but finally did not replied, i.e. approximately 2/3 of answers.

Learning from the benchmarking pilot study
The answers to question (Q2) about each e-service category, What did you or your organization learn from the results of the Local e-Government Bench-learning survey?(Answer per category: Yes / No / I don't know) are summarized in Table 2.
All respondents who participated in the benchmarking study agreed that it had been useful to identify similar cities and good practices to learn from.Most benchmarking reports end with a summary index, which hides the actual good practices; BLM intends to shed light on them, and the answers seem to confirm that this stress on the learning dimension was successful.More specifically, 78% of respondents declared that the survey had helped them to understand the position of their cities compared to other similar cites.A total of 71.43% respondents declared that they had learned the extent of the catalogue of services that cities were offering, and new services that could be offered to citizens; 64.29% learned new functionalities or richer interaction levels and categories in which their city was under-performing.
In another aspect, 85.71% declared that they understood the importance of measuring citizen's eservices adoption.This is a positive perception of e-services comparisons that require an analysis of both the provision and adoption of services to see the whole picture.

Usefulness of display tools and impact on e-services
BLM display and reporting are aimed at understanding parameters related to the services, by presenting each city's results compared to the average.
The answers to question (Q3) Please indicate which aspects of your city e-government have benefited from the city maps offered by the Local e-Government Bench-learning report (Answers per category: Yes / No / I don't know) are summarized in Table 3.The BLM display and reporting proved useful, since 92.86% of respondents declared that city maps helped them to understand the position of their city.Some improvements are needed, as only 64.29% found city maps useful to understand the maturity and coherence of the city's e-service offer.The city map was useful to identify strengths and weaknesses in e-services or to better plan improvements for 64.29% of respondents in both cases.

A U T H O R C O P Y
One free-text comment hints at a shortcoming that has been "patched" by the ICT supplier: "The report was shared with individuals and has inspired insights, but has not been used strategically for improvements because there is no e-government department -only an outsourced ICT supplier that deals with individual departments".This is an extreme case in which there is no department or position to deal with e-government strategic decisions.It shows the divide between e-government practitioners and ICT suppliers.In such a situation, the benchlearning seemed to help to realize the importance of defining such a position.

Impact on specific e-services
The answers to question (Q4) Please indicate in which areas or service categories the good practice list resulting from the Local e-Government Bench-learning has been useful for your city to identify initiatives to learn from (Answers: Very useful / Quite useful / Useful / Slightly useful / Not useful at all) are summarized in Table 4.
When we asked how useful the good practices list for each category had been to identify initiatives to learn from (Q4), the answers revealed quite a positive outcome.In each category, at least 57.14% of respondents found the good practices list either very useful, quite useful or useful, with Citizens' Engagement and Transport and Mobility in first place with 85.71%, while Channelling and Employment and Business both received 57.14%.
The number of services included in the list of good practices might vary in different categories.Therefore, we analysed the correlation between the number of good practices per category and the perceived overall usefulness (measured as 1*very_useful + 0.75*quite_useful + 0.5*useful + 0.25*slightly_useful + 0*not_useful).The correlation was low and negative (correlation index = −0.399),so we discarded it.This result reinforces the reliability of the survey, as the answers seem to be reflective rather than obvious from the information presented to the respondents.The salient practices list resulted from their position (= rho*100 + adoption level, where rho is distance to average maturity).A correlation analysis of position and perceived usefulness per category resulted in a low positive coefficient of 0.341, which increased to 0.735 when the two extreme values were discarded.Therefore, there seems to be a slight direct relation between the perceived usefulness of the good practices list and the objective measure of saliency.This weakness reinforces the idea that summary indexes are not highly relevant for services improvement.

A U T H O R C O P Y
The answers to question (Q5) Please indicate in which areas or service categories your city has started, or plans to start, any action to improve the e-services offer as a consequence of the Local e-Government Bench-learning results (Answers: Yes / No / I don't know) are summarized in Table 5.
Q5 results show the cities in each category that had started, or planned to start, actions to improve their e-services offer as a result of the benchmarking study.Citizens' Engagement was the most popular area (71.43%), followed by Environment (57.14%), while in Life Cycle and Urban Planning the percentage was only 28.57%.There was a weak positive correlation (0.511) between this percentage and the usefulness of the good practices list.

Measuring techniques (self-evaluation)
Table 6 summarizes the answers to question (Q6) Is self-evaluation a good method to assess the service provision, maturity of e-services and citizens' adoption?(Answers: Yes / No).If the answer is (No), please suggest how to improve the evaluation method.
The results indicate that self-evaluation of service provision, as practised in the BLM pilot study, was considered a good assessment method (85.71%), and that the expanded maturity framework was adequate (71.43%).Nevertheless, the method for measuring citizens' adoption was only accepted by 57.14% of participants, while 46.86% thought that this aspect was not well assessed.
A suggestion for improvement was: "Need national benchmarks to judge if adoption rate is significant.It would be good to compare it against an excellent example of fully fledged service provision from inside the municipality; however, I believe we should ask citizens or service users, to get a balanced picture".This suggestion reveals the lack of standards and references to measure adoption; and the proposal is to provide examples of different adoption levels to support self-evaluation.The national dimension seems more relevant for a detailed benchmarking study within a country: references might be drawn from different countries, as there seems to be commonality of services, despite wide differences among cities.The final remark is of key importance: as adoption refers to uptake by users, experts' adoption measurements should be complemented by objective measures and/or the users' views.

A U T H O R C O P Y
The benchmarking study took place in 2008-2009 when it was not common to measure the use of e-services, which is the first step towards measuring adoption.Indeed, in 2009, the OECD [49] found that only 14 out of 22 OECD countries were measuring uptake.However, this situation has changed enormously, according to Q7.
Table 7 summarizes the answers to question (Q7) Which metrics does your organization use to measure citizens' use of e-services?(Answers: Yes / No / I don't know).
Cities have progressed considerably in the objective measurement of citizens' use of e-services.Data such as the number of web pages viewed (85.71%) and the number of web visits or sessions (100.00%) are now widely used, and have been consolidated as basic indicators of e-services use.More importantly, the number of forms submitted (85.71%) and the number of transactions (85.71%) have been widely adopted and are better adoption indicators that open the way to establishing a new adoption measurement framework.

Overall benchlearning method
Table 8 summarizes the answers to question (Q8) Please indicate which areas of the Local e-Government Bench-learning methodology need to be improved (Answers: It's OK / Needs some adjustments / Needs to be redesigned).
More than 70% of respondents thought that the catalogue of services, the service categories, and the matrix of service coverage needed to be adjusted or even redesigned (in the case of the service catalogue) to be more useful.One comment ". . .'needs some adjustment' assuming that e-services have moved on considerably since we did the first study and more one-stop shop services are now on offer, adoption of mobile devices is changing how citizens interact with us, etc." highlights the changes in e-services discussed at the beginning of the paper.Specifically, it indicates that there have been organizational changes in service provision (the introduction of one-stop shops) and that new channels have been introduced for citizens to interact with government (a high rate of mobile devices).
The evaluation method needs some adjustments according to 50% of respondents.The e-government maturity model was considered "OK" by 50%, while the framework for measuring adoption must be redesigned, according to 57.14% of respondents, which is in line with the results indicated above.The list of good practices needs some adjustments (57.14%), which is again in line with the changes in eservices.Global maps and city maps were considered to be suitable by 50% or more of the respondents.

Benchmarking frequency and specific interest in benchlearning
Table 9 summarizes the answers to question (Q9): How often does your city participate in egovernment benchmarking surveys?(Answers: Never / Sometimes / Once a year / I don't know).
The answers to this question reveal that cities do participate in e-government benchmarking at regional and state levels, at the European level (the most common) or worldwide (the least common), but that the frequency is less than once a year.
One comment in the free-text area stated "The council does compete nationally for 'beacon' status and other awards.These can include e-government services, but they are specific to a service area.To my knowledge, there is no national cross cutting e-government comparison that looks at the variety of services that the benchlearning study did."This shows that the variety of services that the BLM takes into account is one of its strengths, and adds real value compared with other initiatives.
The answers to question (Q10) Would your city be interested in participating in a new edition of the Local e-Government Bench-learning survey at European level?show that the vast majority of respondents (78.57%) would participate again.One city would not, and two did not know.This interest in participation seems to confirm the positive perception of BLM.

Discussion
Below we discuss the results in greater depth, within a stronger research context.

Does BLM support learning by cities?
We hypothesized that benchmarking studies summarized in indexes with an associated city ranking are probably not very useful, while BLM that provide greater detail, including a comparison of e-services within a city itself, a comparison with other cities through specific averages of e-services or e-services categories, and the identification of advanced and highly adopted services, would be more useful.
The wide range of Q2 responses in Table 2 seems to support the success of this approach.Respondents identified a range of areas in which they had better understood both their e-services offering and what could be improved, using examples from other cities.A ranking does not help a city manager who wants to improve e-services, but BLM reveals what a city's e-service offering is like, what its strengths are, where it fails, how heterogeneous it is, etc.To learn from others, it is important to identify similar cities in terms of the offering of e-services rather than dimension, population or organization [32], and identify the services that are successful in those cities.BLM represents a step forward in these aspects,

A U T H O R C O P Y
as the visual tools that support this understanding through a non-competitive presentation (Q3) received a positive evaluation.
Q4 inquired quite precisely about the usefulness of the best practices list, which comprises the most advanced and commonly adopted practices.The answers were diverse: the good practices in the list were found to be useful across all categories, but to different degrees.The correlation between the usefulness of the list and the number of cases included in it was low (which shows the high reliability of the answers), as was the correlation with the higher ranked practices (which indicates that rich information is necessary to be useful).
Therefore, BLM achieved its purpose of helping cities to learn from one another on specific points, and to improve the organization [1].Detailed pictures of the services enabled city managers to locate performance gaps, and to prioritize opportunities and areas for enhancement [59].
Nevertheless, the BLM pilot study and the result of this survey cannot claim to be representative.One aspect is that cities involved in the BLM pilot are medium-to-large, and small cities and villages were not benchmarked.The authors intended to complement the pilot with an extensive benchmarking covering small cities and villages in a specific region (Catalonia), and with a larger one at the European level.This would have increased statistical significance, and allowed a more refined segmented analysis according to different city characteristics.The difficulties faced by European public administrations in the period after the pilot are the main causes that this gap has not been filled yet.

Does understanding translate into actual e-services improvement?
One of the aims of BLM is for better understanding to be translated into e-services improvement.Q5 showed a weak positive correlation between the best practices list and the categories of e-services that were reported as improved.Thus, the best practices identified through BLM led to e-services improvements.The weakness of the correlation was to be expected, as cities are diverse, and have different priorities.This means that benchmarking studies should provide rich pictures, to support diverse learning and strategic planning.

Methodological aspects
Aspects of BLM were rated differently.First, it was considered that the bottom-up catalogue of services should be redesigned, due to the manifold changes in e-services since the pilot study.A specific subsection discusses this in depth in the smart cities context.The role of the bottom-up catalogue is described below.Rather than using an existing taxonomy, categories were defined in order to group existing services into thematic areas that were identified as key for city e-government.The resulting categories were instrumental to provide aggregated information for strategic planning in a legible way.This contingent catalogue now needs to be revised and adapted, to take into account new trends in e-services and public policies.
Although the e-government maturity model for assessing the services offering was well-rated, it should be updated to better cope with the new types of e-services on offer.According to [29], egovernment develops over time, and further developments emerge that need to be incorporated into any revised maturity model.A high number of e-government maturity models exist [19].However, the model that is selected should address the richness and variety of e-services, and be easy to apply in the self-evaluation process used for measurement.The Manchester e-Government Maturity Model [29] once split in its two dimensions, front-office and the back-office, seems a good recent alternative that will add a new dimension to the e-services maturity analysis.

A U T H O R C O P Y
The rich display and reporting, and the good practices list, were also well-rated and need fewer significant adjustments.
The self-evaluation technique is key to affordable benchmarking studies, and has been used in other e-government research [32,35,58].The survey answers indicated that it provides fair accounts, but less so when measuring service adoption.This aspect is discussed in more detail in the subsection below.

Measuring adoption and introducing citizens' views
The self-evaluation technique used to assess provision relies on a widespread, accepted (maturity) framework to significantly reduce subjectivity.An accepted framework is essential to assess adoption based on self-evaluation [64].Such a framework is lacking, but we still need to measure citizens' use of services to understand where e-government stands.Therefore, as a first step to reducing the existing design-actually gap [27], BLM adopted a simplified approach that could be self-assessed: expected versus actual adoption.
The e-services adoption concept is more specific than use: it is the rate of service provided through e-means vs traditional channels.In 2004 [34], the following factors were identified as key: the number of individuals and businesses that used e-services, the percentage of citizens that visited websites to search for information, the number of businesses that made payments online, and the percentage of internet traffic related to e-service delivery.Two years later [14], different web metrics were proposed: the number of hits or user contact sessions, the number of document downloads, the amount of time users spend on a site, the number of transactions completed, and web analytics (click streams, repeat use and cross-usage).Some of these indicators are not supported by current web metrics.More importantly, they seem more aligned with absolute service use than with relative adoption, which should be in terms of rates, rather than absolute numbers.Nevertheless, measuring use of e-services is a first step towards assessing adoption.
When the BLM benchmarking study was carried out (2008)(2009), no metrics on use were shared by the participating cities. McKinsey remarked in 2009 that governments had few web analytics experts, 3and in 2012 the U.S. government launched the Digital Analytics Program to help agencies meet the requirement 4 of implementing web analytics in 3 months.European governments seem to be following this pattern, as most survey respondents reported that they use web metrics.Now, Table 7 reveals that some standard quantitative metrics related to adoption have been widely introduced by the cities surveyed.
However, many barriers still hinder the assessment of adoption.A first one is that services with different levels of sophistication need different adoption measurements [61].Higher maturity means different modalities and user interaction: while the use of an information service (level 1) can be measured through the number of web pages served, a transactional service (level 3) needs to count the number of completed transactions.Further research is needed to identify the right adoption indicators to be used at each stage of maturity.Smart e-services, discussed in the following subsection, will require similar indicators.For instance, the actual use of apps -not just the number of downloads -needs to be measured in the context of a tool-shift or process-shift in the way citizens do things.
A second barrier is that targets are not the same for each service.Different services involve different populations with different social characteristics and frequency of use.Therefore, rates are difficult to compute.
Although research has led to a greater understanding of the drivers of citizens' use of egovernment [38], there is still a lack of data on real use [66].
The measurement of service adoption means taking into account citizens' views, and not just the perspectives of the administrations (which are more clearly concerned with service provision): this is very important for a benchmarking study to be useful to improve e-services.In fact, to improve e-services, it is essential to increase our understanding of citizens' motivations and their satisfaction beyond quantitative metrics, and to determine whether e-services actually play a significant role in the improvement of quality of life.In other words, it is vital to learn directly from citizens.How this can be performed would probably require a lengthy discussion on Human-Computer Interaction techniques, a field moving towards understanding user experience (see for instance [26]), and using "in the wild" strategies [55].However, this is beyond the scope of the BLM, which is a benchmarking method for the useful comparative evaluation of cities' e-services.

A new catalogue of smart e-services
With the dawn of the century, smartness appeared as an ideal for cities, in which new management based on advanced infrastructures would maximize citizens' quality of life, economic prosperity, energy sustainability and respect for nature [8].Most proponents of the concept depict it in a different way, as a solution to current challenges: waste management, scarcity of resources, air pollution, human health concerns, traffic congestion, and inadequate, deteriorating and ageing infrastructures, including social ones.Cities label as smart the most innovative ways to manage these problems [13].
The concept relies mainly on the appearance of new, diverse, cheaper sensors that provide substantial quantities of urban environment data, which could be useful to run a city.Most definitions focus on the role of ICT infrastructures, but some stress human capital and education, social and relational capital, and environmental interest [18].Nevertheless, the smart city concept remains fuzzy [12], and its meaning has been changing, so there is no commonly agreed definition as yet [18].This lack of consensus is obviously a clear obstacle to identifying a real smart city [30].Our main focus [45] defines, perhaps too widely, a smart city service as any innovative service using ICT in an urban habitat.
The availability of large amounts of data generated by new sensors or by users (human sensors) does not lead per se to new or improved e-services, although it establishes the basis for a better government.It seems convenient to stick to basics to get a clearer picture.For instance, Rogers claims that "Communication technologies are transforming our economies, our ways of learning, our methods of work, our capacity to alter the environment and even our daily chores and pleasures; they are unmistakably reshaping our lives" [54, p. 147].Now, citizens' minds and sophisticated technologies replace raw materials and brawn: "The networking of creativity is now driving the new 'creative' economy.Exchanges between art and technology -the exchange of ideas rather than of commodities-are becoming the lifeblood of the new economy and of our future prosperity" [54, p. 162].Let us reflect on how these general ideas are shown through specific changes in local e-services.
First, smart city dwellers and workers use smart services for their daily activities regardless of time or place: citizens use them to meet their daily needs, companies and people carrying out economic activities in the city use them to improve their businesses, and city managers and public service operators use them to ensure high quality public service provision and city maintenance.Thus, services cover more activities, and are used ubiquitously.Second, e-service provision itself has been evolving in two aspects that need to be analysed: city administrations now provide e-services in different ways; and eservices of public interest have emerged that are provided by new players, the citizens, the civic sector

A U T H O R C O P Y
and companies.E-services analysis needs a new basis in this landscape [17].The traditional analysis, centred on e-government offerings, should be extended to cover the new formulas of public provision based on public-private partnerships [21], and offerings by voluntary citizens and the third sectorwhich are especially relevant with regards to improved e-governance.Research should move from the study of public organizations to a more holistic analysis of the whole digital city creative ecosystem, as anticipated by [2].E-services have been strongly influenced by the app explosion.Mobile applications, or apps, run on mobile devices such as smart phones and tablets [57].Apps are small, cheap, purpose-specific, functionally restricted and have simple user interaction.The apps phenomenon is due to several factors, including new and more powerful devices, high bandwidth wireless technologies, and the creativity of developers [62].A study [17] states that Web 2.0, Open Data and Open Infrastructures are also boosting the creation of new apps.Although apps provide services that meet very specific needs, often for small groups of citizens, their impact on quality of life is perceived positively.Citizens adopt them because they seem easier and more efficient: they require fewer resources in terms of time and/or money and yield better results.Users are adopting them to meet their current basic needs (housing, food, employment, safety and health, among others), their secondary needs (mobility and access to information) or their personal needs (culture, leisure, hedonism, ambition, friendship, anonymity, social recognition, etc.) [40].
Some apps created by citizens and private companies are becoming real services of public interest, used by citizens.Some are strong competitors of government e-services, while others go beyond those provided by the public administration.This blurs the borders of e-government, and changes the dominant position of governments as providers of e-services.In fact, city governments risk becoming marginal providers, only of legally established public e-services.
This situation has encouraged city administrations to provide traditional e-services as apps too, and to increase efforts to introduce more personalized delivery [52].In addition, city councils intend to position themselves as prescribers of third party urban e-services, as illustrated by the initiatives Apps for Amsterdam, 5 Helsinki App Store6 or Barcelona's Apps4BCN. 7Cities are also actively engaging citizens and companies in the creation of apps of public interest through contests.If we examine our initial research stages on the e-services provided by Barcelona, the main novelty seems to be the offering of a variety of mobile e-services delivered through smart phones, which have been specifically designed for the new channel.They support the devices' design logic to create a better user experience, and are not just a port for existing e-services.In other words, the services themselves have been redefined or deeply transformed.
Recently [45], reviewed and analysed existing taxonomies for classifying e-services in smart cities, and proposed a new typological framework that has four dimensions: technology mode, service purpose, service authority, and delivery mode, resulting in 17 classification categories.This classification is citizen-centric, and borrows the "purpose of consumption" concept from marketing.As it contains characteristics that we discussed earlier in the new e-services, it could be an interesting starting point for the development of a new bottom-up catalogue.A new catalogue would also need empirical data on smart city services that are currently provided in a variety of cities, but there is no such study, to the best of our knowledge.This new catalogue could be the basis of a useful local e-services benchmarking study.

Table 3
Aspects in which cities have benefited from BLM benchmarking

Table 5
Areas in which cities have started or planned actions to improve e-services

Table 9
How often cities participate in e-government benchmarking