The core enabling technologies of big data analytics and context-aware computing for smart sustainable cities: a review and synthesis

Smart sustainable cities

The concept of smart sustainable cities has emerged as a result of three important global trends at play across the world, namely the diffusion of sustainability, the spread of urbanization, and the rise of ICT [5]. As echoed by Höjer and Wangel [17], the interlinked development of sustainability, urbanization, and ICT has recently converged under what is labelled ‘smart sustainable cities.’ Accordingly, smart sustainable cities is a new techno–urban phenomenon that materialized around the mid–2010s (e.g. [5, 6, 15–17, 72]). The idea revolves around leveraging the advance and prevalence of ICT in the transition towards the needed sustainable development in an increasingly urbanized world. Therefore, the development of smart sustainable cities is gaining increasing attention worldwide from research institutes, universities, governments, policymakers, and ICT companies as a promising response to the imminent challenges of sustainability and urbanization.

 The term ‘smart sustainable city’, although not always explicitly discussed, is used to denote a city that is supported by a pervasive presence and massive use of advanced ICT, which, in connection with various urban systems and domains and how these intricately interrelate and are coordinated, enables the city to control available resources safely, sustainably, and efficiently to improve economic and societal outcomes. Here ICT can be directed towards, and effectively used for, collecting, processing, analyzing, and synthesizing data on every urban domain and system in terms of forms, structures, infrastructures, networks, facilities, services, and citizens. The resulting knowledge can then be employed to develop urban intelligence functions and build urban simulation models for strategic decision-making associated with sustainability. Further, the combination of smart cities and sustainable cities, of which many definitions are available, has been less explored as well as conceptually difficult to delineate due to the multiplicity and diversity of the existing definitions (see [5] for an overview). ITU [18] provides a comprehensive definition based on analyzing around 120 definitions, ‘a smart sustainable city is an innovative city that uses…ICTs and other means to improve quality of life, efficiency of urban operation and services, and competitiveness, while ensuring that it meets the needs of present and future generations with respect to economic, social and environmental aspects’. Another definition put forth by Höjer and Wangel ([17], p. 10), which is deductively crafted and based on the concept of sustainable development, states that ‘a smart sustainable city is a city that meets the needs of its present inhabitants without compromising the ability for other people or future generations to meet their needs, and thus, does not exceed local or planetary environmental limitations, and where this is supported by ICT’. This entails unlocking and exploiting the potential of ICT of the new wave of computing as an enabling, integrative, and constitutive technology for achieving the environmental, social, and economic goals of sustainability due to the underlying transformational, substantive, and disruptive effects [5, 6].

Context-aware computing

Context awareness has been defined in multiple ways depending on the application domain in terms of the number and nature of the subsets of the context of a given entity (e.g. traffic system, energy system, healthcare system, education system, information system, human user, etc.) that can be integrated in the design and development of a given computational artifact. Originated in pervasive computing the term ‘context awareness’ is used to describe technology that ‘is able to sense, recognize, and react to contextual variables, that is, to determine the actual context of its use and adapt its functionality (and behavior) accordingly or respond appropriately to features of that context.’ ([19], p. 76). Another definition of context proposed by Dey [20] states: ‘context is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves.’ Context-aware applications and systems in the urban domain entail the acquisition of contextual urban data using sensors of many types to perceive situations of urban life, the abstraction of contextual urban data by matching sensory readings to specific urban context concepts, and application behavior through firing actions based on the outcome of reasoning against contextual urban information, i.e. the inferred context, to draw on Schmidt [21].

In recent years, the concept of context awareness has been expanded beyond the ambit of HCI applications to include urban applications, such as energy systems, transport systems, communication systems, traffic systems, power grid systems, healthcare systems, education systems, security systems, and so on (e.g. [1, 6, 9–11, 21–24]). Here context denotes, drawing on Chen and Kotz [25], the environmental conditions within the urban landscape that either determine applications’ behavior or in which application events occur and are interesting to different classes of users, including citizens, urban administrators, urban operators, urban authorities, and urban departments.

Big data analytics: characteristics and techniques

The term ‘big data’ is used to describe the growth, proliferation, heterogeneity, complexity, availability, temporality, changeability, and utilization of data across many application domains, which renders the processing of these data exceed the computational and analytical capabilities of standard software applications and conventional database infrastructure. In short, the term essentially denotes datasets that are too large for traditional data processing systems. Traditional analytic systems are not suitable for handling big data (e.g. Katal et al. [26]; [27]). This implies that big data entails the use of tools (classification, clustering, regression algorithms, etc.), techniques (data mining, machine learning, statistical analysis, etc.), and technologies (Hadoop, HBase, MongoDB, etc.) that work beyond the limits of the data analytics approaches that are used to extract useful knowledge from large masses of data for timely and accurate decision-making and enhanced insights. As a common thread running through most of the definitions of big data, the related information assets are of high-volume, high-variety, and high-velocity and thus require cost-effective, innovative forms of data processing, analysis, and management for enhanced decision-making and insight. While there is no canonical or definitive definition of big data in the context of smart sustainable cities, the term can be used to describe a colossal amount of urban data, typically to the extent that their manipulation, analysis, management, and communication present significant computational, analytical, logistical, and coordinative challenges. It is near on impossible to humanly make sense of or decipher big urban data based on existing computing practices. Important to note is that such data are invariably tagged with spatial and temporal labels, largely streamed from various forms of sensors, and mostly generated automatically and routinely. Regardless of the lack of agreement about the definition of big data, there seems to be consensus that big data will lead to, in light of the projected advancements and innovations, immense possibilities and fascinating opportunities in the coming years. Moreover, big data solutions requires novel technologies to proficiently process large volumes of data emanating from multiple sources, in unprecedented quantities, and in quick time.

Big data are often characterized by a number of Vs. The main of which—identified as the most agreed upon Vs—are volume, variety, and velocity (e.g. [28, 29]). Additional Vs include veracity, validity, value, and volatility (e.g. [27]). The emphasis here is on the main characteristics of big data, namely the huge amount of data, the velocity at which the data can be analyzed, and the wide variety of data types.

The term ‘big data analytics’ refers commonly to any vast amount of data that has the potential to be collected, stored, retrieved, integrated, selected, preprocessed, transformed, analyzed, and interpreted for discovering new or extracting useful knowledge, which can subsequently be evaluated and visualized in an understandable format prior to its deployment for decision-making purposes (e.g. a change to or enhancement of operations, strategies, practices, and services). Other computational mechanisms involved in big data analytics include search, sharing, transfer, querying, updating, modeling, and simulation. In the context of smart sustainable cities, big data analytics denotes a collection of sophisticated and dedicated software applications and database systems run by machines with very high processing power, which can turn a large amount of urban data into useful knowledge for well-informed decision-making and enhanced insights pertaining to various urban domains, such as transport, mobility, traffic, environment, energy, land use, planning, and design.

The common types of big data analytics include predictive, diagnostic, descriptive, and prescriptive analytics. These are applied to extract different types of knowledge or insights from large datasets, which can be used for different purposes depending on the application domain. Urban analytics involves the application of various techniques based on data science fundamental concepts—i.e. data-analytic thinking and the principles of extracting useful knowledge (hidden patterns and meaningful correlations) from data, including machine learning, data mining, statistical analysis, regression analysis (explanatory modeling versus predictive modeling), database querying, data warehousing, or a combination of these. The use of these techniques depends on the urban domain as well as the nature of the urban problem to be tackled or solved.

Pervasive sensing for urban sustainability

Sensor technologies in context-aware computing

Sensor types and sensing areas in context-aware applications

As with big data analytics, context-aware computing involves a wide variety of sensors. A sensor can be described as a device that detects or measures a physical property or some type of input from the physical environment, and then indicates or reacts to it in a particular way (e.g. [19]). The output is a signal in the form of human-readable display at the sensor location or a recorded data that can be transmitted over a network for further processing. Commonly, sensors can be classified according to the type of energy they detect as signals, and include, but are not limited to, the following types:

• Location sensors (e.g. GPS, active badges).

• Optical/vision sensors (e.g. photo-diode, color sensor, IR and UV sensor).

• Light sensors (e.g. photocells, photodiodes).

• Image sensor (e.g. stereo-type camera, infrared).

• Sound sensors (e.g. microphones).

• Temperature sensors (e.g. thermometers).

• Heat sensors (e.g. bolometer).

• Electrical sensors (e.g. galvanometer).

• Pressure sensors (e.g. barometer, pressure gauges).

• Motion sensors (e.g. radar gun, speedometer, mercury switches, tachometer).

• Orientation sensors (e.g. gyroscope).

• Physical movement sensors (e.g. accelerometers).

• Biosensors (e.g. pulse, galvanic skin response measure).

• Vital sign processing devices (heart rate, temperature).

• Wearable sensors (e.g. accelerometers, gyroscopes, magnetometers).

• Identification and traceability sensors (e.g. RFID, NFC).

While there are different ways of sensing that could be utilized for detecting various features of context, in the realm of smart sustainable cities not all the above are of use in relation to context-aware applications in terms of optimization, control, management, operation, and service delivery associated with sustainability dimensions. How many and what types of sensors can be used in relation to a given context-aware application is determined by the way in which context is operationalized (defined so that it can be technically measured and thus conceptualized) in terms of the number of the entities of context that are to be incorporated in the system based on the application domain, and also whether and how these entities can be combined to generate a high-level abstraction of context (e.g. the physical, situational, behavioral, and social dimension of context). Too often, in relation to both citizens and urban systems, various kinds of sensors are used to detect context.

Acquisition of sensor data about citizens and urban systems (energy, traffic, transport, mobility, etc.) and their behavior and functioning is an important factor in addition to the knowledge domain for analysis of such data by data processing units. In relation to context-aware applications pertaining to citizens, data can be generated from multiple sources, including software equivalents in relation to citizens’ devices, such as smartphones, computers, laptops, and other everyday objects. In other words, data are collected and captured from a variety of digital sensors as well as online interactive applications. Observed information about the citizen and urban system’ states or situations in conjunction with the dynamic models for the citizen and system’ relevant processes serve as input for the process of computational understanding. This entails the analysis and estimation of what is going on in the surrounding environment in the context of smart sustainable cities. Accordingly, for a context-aware application or system to be able to infer high-level context abstraction based on the interpretation of and reasoning on context information, it is first necessary to acquire low-level data from physical sensors (and other sources). Researchers from different application domains within the field of context-aware computing have investigated context recognition for the past 2 decade or so by developing a diversity of sensing devices (in addition to methods and techniques for signal and data processing, pattern recognition, modeling, and reasoning tasks). Thus, numerous types of sensors are currently being used to detect various attributes of context.

Multi-sensor data fusion and its application in context-aware applications and systems

In context-aware computing, underlying the multi-sensor fusion methodology is the idea that an abstraction of context as an amalgam of different, interrelated contextual elements can be generated or inferred on the basis of information detected from multiple, heterogeneous data sources, which provide different, yet related, sensor information. Thus, sensors should be integrated to yield optimal context recognition results, i.e. provide robust estimation of context. A given dimension of the context, a higher level of the context, can be deduced by using a number of external or internal contexts as an atomic level of the context. Figure 1 illustrates multisensor fusion for context awareness.

The use of multi-sensor fusion approach in context-aware applications and systems allows gaining access simultaneously to varied information necessary for accurate estimation or inference of context. Multi-sensor fusion systems have the potential to enhance the information gain while keeping the overall bandwidth low [19]. Figure 1 illustrates a multi-sensor fusion approach.

Wireless communication network technologies and smart network infrastructures

In the context of smart sustainable cities, wireless solutions are set to proliferate in ways that are hard to imagine, as ICT continues to be fast embedded and interwoven into the very fabric of current smart and sustainable cities in terms of their systems and processes in an increasingly computerized urban society. This is a future world of pervasive computing infrastructures and communication networks. Countless sensors will use various wirelessly ad-hoc and mobile networks to provide cities with all kinds of data necessary for a wide variety of applications and services. In particular, the widespread diffusion of wireless network technologies will, as a by-product of their normal operations, enable to sense, collect, and coordinate massive repositories of spatiotemporal data pertaining to urban systems, which represent city-wide proxies for all kinds of activities and operating and organizing processes.

Also, smart networks are necessary for big data applications in terms of connecting the components and entities of smart sustainable cities, including diverse citizens’ everyday objects (computers, smart phones, cars, house devices, etc.) and city infrastructures and facilities as well as urban departments, authorities, and enterprises. Such networks are intended to provide efficient means for transferring the collected data from heterogeneous and distributed sources to data warehouses where big data are to be stored, coalesced, organized, and integrated for processing and analysis in connection with intelligent decision support systems. This involves transferring responses back to the different citizens’ devices and urban entities’ systems for the purpose of improving different aspects of sustainability.

In relation to ICT of the new wave of computing, networking is a core enabling technology, in addition to cheap, low-power sensing and computing devices. In this context, the role of networking lies in tying hardware and software systems all together for the functioning of ubiquitous applications and services in urban areas, to draw on Bibri [19]. Accordingly, many heterogeneous components and devices across dispersed infrastructures and disparate networks need to interconnect as part of vast architectures enabling big data analytics, context-aware computing, intelligence functions, and service provisioning on a hard-to-imagine scale [19]. To put it differently, wireless network technologies are prerequisite for coordinating data as well as linking up many diverse distributed sensing devices and computing components and enabling them to interact in the midst of a variety of hardware and software systems necessary for realizing smart urban environments for advancing sustainability. Wireless technologies, especially satellite-enabled GPS, Wi-Fi, and mobile phone networks, enable to sense, collect, and coordinate massive environmental and socio-economic data representing enormous proxies for the operations, functions, and services of smart sustainable cities and thus powerful physical-environmental and socio-behavioral microscopes (e.g. [6]). This may facilitate, by means of big data analytics (data mining and database integration capabilities) which offer the prospect for adding value in terms of massive data analysis and integration, discovering the hidden patterns, correlations, and models that characterize, on the one hand, human mobility and movement as part of daily trajectories and activities of citizens and, on the other hand, physical structures and spatial organizations, which can be instrumental in strategic decision-making associated with urban sustainability planning (see [6]). In all, while pervasive sensing and computing infrastructures allow for monitoring, understanding, and analyzing urban life in terms of infrastructure, built form, administration, and ecosystem and human services, pervasive networking infrastructures allow for collecting and coordinating extensive data in terms of how these data are stored, made accessible, and utilized.

In the context of smart sustainable cities, advanced digital networks are crucial to urban operational functioning and planning due to the interrelationships between urban components and domains that are too many to catalogue (transport, mobility, communication, building, energy, environment, water, waste, land use, healthcare, etc.). These are planned to be further heavily networked while the activities relating to these domains to be linked up. The key domains ‘which currently are being heavily networked involve: transport systems of all modes in terms of operation, coordination, timetabling, utilities networks which are being enabled using smart metering, local weather, pollution levels and waste disposal, land and planning applications, building technologies in terms of energy and materials, health information systems in terms of access to facilities by patients the list is endless. The point is that we urgently need a map of this terrain so that we can connect up these diverse activities’ ([2], p. 493). Especially, the evolving techno-urban contexts are opening spaces for smart sustainable initiatives in domain networking at current times of tension as alternative trajectories are actively being sought due to the challenge of sustainability, which entails creating innovative solutions that further facilitate collaboration among urban domains and hence integrate urban systems.

In parallel, the aim of emerging technological platforms such as UbiComp, AmI, the IoT, and SenComp is to orchestrate and coordinate the various computational entities in the informational landscape of smart sustainable cities and merging it with their physical landscape into an open system that helps diverse urban entities cope with and plan their activities in relation to improving sustainability. Besides, the growing depth, scale, and complexity of urban networks in terms of both domains and technological infrastructures call for developing and coordinating such networks and enhancing their digital capabilities in ways that increase and sustain the contribution of smart sustainable cities to the goals of sustainable development. Advanced wireless technologies are extremely placed to initiate this development and coordination. Moreover, with their ever-growing volume, variety, velocity, and timeliness, data on the state of urban networks as built artifacts as well as on that of their use as part of urban activities and processes provide enormous potential to improve urban operational functioning and planning (see, e.g. [1, 2]) in terms of sustainability, efficiency, and the quality of life by exploiting the analytical power of big data for deep insights and enhanced decision-making. To effectively use these data when implementing big data applications in smart sustainable cities requires fostering these data by advanced wireless technologies, especially in relation to real-time applications. The rationale is that such applications entail that the data from distributed sources should be aggregated and fused prior to being transferred in real-time to cloud computing infrastructures or data processing platforms for stream processing and decision-making. Important to note is that the aggregation and fusion should be carried out in ways that enable data to remain reliable, accurate, and correct for more effective results and thus beneficial knowledge in terms of decision-making processes. This is in turn of critical importance for maintaining the quality and performance of real-time big data applications in terms of decision-making processes [76].

Data processing platforms for big data analytics

There is a variety of available data processing platforms for big data analytics, which provide the stream processing required by real-time big data applications in relation to various urban domains. Therefore, data processing platforms are a key component of the ICT infrastructure of smart sustainable cities of the future with respect to big data applications. Among the leading platforms for big data storage, processing, and management include Hadoop MapReduce, IBM Infosphere Streams, Stratosphere, Spark, and NoSQL-database system management (e.g. [1, 28, 53, 60, 62, 63]). These platforms work well on cluster systems to meet the requirements of big data applications for smart sustainable cities; entail scalable, evolvable, optimizable, and reliable software and hardware components; and provide high performance computational and analytical capabilities (namely selection, preprocessing, transformation, mining, evaluation, interpretation, and visualization), in addition to storage, coordination, and management of large datasets across distributed environments. As ecosystems, they perform big data data analytics related to a wide variety of large-scale applications intended for different uses associated with the process of sustainable urban development, such as management, control, optimization, assessment, and improvement, thereby spanning a variety of urban domains and subdomains. In all, they are prerequisite for data-centric applications for smart sustainable cities of the future. The focus on Hadoop MapReduce is justified by the suitability of its functionalities as to handling urban data as well as to its advantages associated with load balancing, cost effectiveness, flexibility, and processing power compared to other data processing platforms. Hadoop MapReduce has become the primary big data storage and processing system given its simplicity, scalability, and fine-grain fault tolerance [59]. For example, it is capable of handling all data types collected from multiple sources to derive actionable insights. However, it does pose issues regarding processing efficiency, rigid data flow, and low-level abstraction. NoSQL (e.g. Mongo DB and Cassandra) is also fast becoming a choice for storing and sorting structured and unstructured data and cluttering them with greater efficiency and scalability.

Cloud computing for big data analytics: characteristic features and benefits

Big data analytics can also be performed in the cloud. This involves both big data platform as a service (PaaS) and infrastructure as a service (IaaS) (e.g. [77]). Having attracted attention and gained popularity worldwide, cloud computing is becoming increasingly a key part of the ICT infrastructure of both smart cities and sustainable cities (e.g. [1, 5, 7, 53, 66, 67, 71]) as an extension of distributed and grid computing due to the prevalence of sensor technologies, storage facilities, pervasive computing infrastructures, and wireless communication networks. Especially, most of these technologies have become technically mature and financially affordable by cloud providers. By commoditizing services, low cost open source software, and geographic distribution, cloud computing is becoming increasingly an attractive option [78].

Big data analytics is associated with cloud computing (e.g. [1, 77]; [79], an Internet-based computing model that is increasingly seen as the most suitable solution for highly resource intensive and collaborative applications as an on-demand network access to a shared pool of computing resources (memory capacity, energy, computational power, network bandwidth, interactivity, etc.) [1, 7, 80]. This entails that computer-processing resources, which reside in the cloud, are virtualized and dynamic, which implies that only display devices for information and services need to be physically present in relation to urban domains where diverse stakeholders (administrators, planners, landscape architects, sustainability strategists, authorities, citizens, etc.) can make use of software applications and services to improve sustainability. Such stakeholders can access cloud-based software applications through a web browser and a lean client (a computer program that depends on its server to fulfill its computational roles) or mobile devices while software tools and urban data of all kinds are stored on servers at a remote location. Indeed, cloud computing model is based on hosted services in the sense of application service provisioning running client server software locally. In this respect, smart sustainable city applications pertaining to transport, traffic, mobility, energy, public health, civil security, education, and so on reside ‘in the cloud’ and can be accessible per demand. Moreover, the software development platform can be offered in a public, private, or hybrid network, where the cloud provider manages the platform that runs the applications and relieves the cloud clients from the burden of securing dedicated platforms, which would otherwise be very demanding and costly in terms of resources and time. The cloud clients can accordingly benefit from tested, scalable, reliable, and maintainable platforms offered by the cloud provider. Another advantage involves service process optimization through advanced functionalities of software development platforms, namely flexibility, interoperability, reusability, scalability, and cooperation. There is also a great opportunity to slash or minimize energy consumption associated with the operation of ICT infrastructure, especially when it comes to large-scale deployments like in the case of smart sustainable cities as to different departments and service agencies. Beloglazov et al. [81] develop policies and algorithms that aim at increasing energy efficiency in cloud computing. Energy consumption is way too lower than if all urban entities have their own software development platforms. These are indeed shared by multiple users as well as dynamically reallocated per demand. This approach maximizes the use of computational power and reduces energy usage and thus mitigate GHG emissions associated otherwise with powering a variety of functions as well as data centers dispersed throughout the departments and service agencies of smart sustainable cities. Whether public or private, the cloud provider includes the cloud environment’s servers, storage, networking, and data center operations. This implies that the cloud provider has the actual energy-consuming computational resources; users or clients can simply log on to the network without installing anything, thereby curbing energy usage and making the best of the available computational power. Energy efficiency in cloud computing can result from energy-aware scheduling and server consolidation [82]. Mastelic et al. [83] provides a survey on energy efficiency in cloud computing. Also, cloud computing is seen as a form of green computing, especially if it is based on renewable energy like solar panels. It has other intuitive benefits because it relies on sharing of resources and maximizing the effectiveness of the shared resources, thereby reducing the costs otherwise incurred by ICT operations as to human, technical, and organizational resources. In cloud computing, supercomputers in large data centers as a distributed system of many servers are used to deliver services in a scalable manner as well as to enable the storage and processing of vast quantities of data. Cloud computing offers great opportunities for streamlining data processing [84]. In all, cloud computing constitutes an efficient and elegant solution in terms of facilitating the huge demand for computing resources associated with big data analytics for decision-making processes in relation to the operational functioning and planning of cities in terms of sustainability. Through the use of cloud computing, smart sustainable cities can accordingly have higher possibilities to perform more effectively and efficiently thanks to the advanced technological features underlying the functioning of cloud computing model.

In addition, cloud computing performs service-oriented computing. In this regard, it can rapidly process large and complex data produced from urban activities and simultaneously serve citizens in relation to healthcare, education, housing, utility, and so on, providing a kind of integrated and specialized center for information services to both the general public and urban departments across various urban domains. In light of this, with reference to smart sustainable cities, cloud computing has the ability to run smart applications on many connected computers and smartphones at the same time for different purposes associated with increasing sustainability performance.

In sum, among the key advantages provided by cloud computing technology include cost reduction, location and device independence, virtualization (sharing of servers and storage devices), multi-tenancy (sharing of costs across a large pool of cloud provider’s clients), scalability, performance, reliability, and maintenance. Therefore, opting for cloud computing to perform big data analytics in the realm of smart sustainable cities remains thus far the most suitable option for the operation of infrastructures, applications, and services whose functioning is contingent upon how urban domains interrelate and collaborate, how efficient they are, and to what extent they are scalable as to achieving and maintaining the required level of sustainability.

Middleware infrastructure for context-aware computing: characteristics and functions

Middleware infrastructure is associated with pervasive computing environments and distributed applications. These encompass UbiComp, AmI, and SenComp environments and applications. Middleware infrastructure (e.g. [44, 47, 48]) plays a key role in the functionalities of complex distributed applications, including context-aware applications. Thus, context-aware computing, which is associated with UbiComp, AmI, and SenComp, requires middleware infrastructure to operate. This infrastructure can also run on cloud computing [platform as a service (PaaS) and infrastructure as a service (IaaS)]—i.e. cloud middleware.

Middleware infrastructure represents the logic glue in a distributed computing system, as it connects and coordinates many components constituting distributed applications. This occurs, more specifically, ‘in the midst of a variety of heterogeneous hardware systems and software applications needed for realizing smart environments and their proper functioning. To put it differently, in order for the massively embedded, distributed, networked devices and systems, which are invisibly integrated into the environment, to coordinate require middleware components, architectures, and services. Middleware allows multiple processes running on various sensors, devices, computers, and networks to link up and interact to support (and maintain the operation of context-aware applications needed by citizens and urban entities to cope with and perform their) activities wherever and whenever needed.’ ([19], p. 50). Indeed, it is the ability of multiple, heterogenous hardware and software systems to cooperate, interconnect, and communicate seamlessly across disparate networks that create smart environments rather than just their ubiquitous presence and massive use. In the context of smart sustainable cities, such systems in their various forms (e.g. sensors, smartphones, computers, databases, data warehouses, application integration methods, application servers, web servers, context management systems, and messaging systems) are highly distributed, interoperable, and dynamic, involving a myriad of embedded devices and information processing units ‘whose numbers are set to increase by orders of magnitude and which are to be exploited in their full range to transparently provide services on a hard-to-imagine scale, regardless of time and place’ [19]. This in turn allows for the functioning of context-aware applications across the diverse domains of smart sustainable cities.

As regards to some of its characteristic features compared to cloud computing, middleware-based architectures entail reusable software infrastructure that resides between the application programs (in this case context-aware applications) and the underlying hardware and operating systems. That is to say, middleware sits between the kernel and applications. Incidentally, the functionality of network protocol stacks (TCP/IP) was previously provided separately by middleware, but nowadays is integrated in every operating system. Moreover, middleware simplifies and supports the development of complex distributed applications, using such tools as web servers, application servers, messaging systems, and content management systems. These applications collaborate with, or leverage services from, other disparate applications that are systematically tied using methods of application integration. In addition to handling the distribution and heterogeneity of computing resources associated with the logic of context-aware applications in this context, middleware is intended to bridge the gap between the applications and the underlying lower-level hardware and software infrastructure to ensure and boost coordination, cooperation, interconnection, dynamicity (e.g. sensors join and leave AmI infrastructure in a dynamic fashion), and interoperability of the different components of distributed applications (e.g. [19, 45, 85]). These functionalities are in fact necessary for supporting scalable systems as well as highly heterogeneous and distributed components, such as agents and services. In relation to this, middleware support and deploy data-centric distributed systems (e.g. network-monitoring systems, sensor networks, and dynamic web) whose ubiquity creates large application networks spreading over large geographical areas [19]. Especially, AmI and UbiComp infrastructures are highly dynamic and involve high degree of heterogeneity (e.g. [86]. As to interoperability, for instance, context-aware applications run on different operating systems, thereby the role of middleware in enabling interoperability between applications by supplying services for exchanging data in a standard way. Indeed, in the realm of context-aware computing, which entails distributed processing in the sense of multiple applications being connected to create larger applications over a network, middleware provides services beyond or more than those available from the operating system of these applications to enable the various elements of the underlying distributed system to communicate and manage data, thereby serving as a kind of a software glue. Therefore, distributed processing is empowered by middleware for transferring signals from various sources and for realizing information fusion from multiple perceptive components [44].

Middleware and cloud computing infrastructures differ in their technical details as to how they provide application services and which kind of services they are concerned with, as well as in the characteristic features of their operation and complexity. Yet, they denote computing models where machines in large data centers across distributed environments can be used to deliver a variety of services and meet the needs of different urban constituents in terms of the use of big data and context-aware applications for improving sustainability. Hence, both are prerequisite for in the operation of smart sustainable cities. This is anchored in the underlying assumption that big data and context-aware applications are an integral part of ICT of the new wave of computing, and smart sustainable cities typically rely on the fulfillment of its underlying visions.

Big data management

Given the volume, variety, and velocity characterizing big data, effective and suitable big data management tools are extremely important to ensure a useful utilization of big data in terms of analytics and the related results and inferences. Accordingly, as smart sustainable cities involve the generation of large, varied, and time-based data pertaining to such urban domains as transport, traffic, mobility, energy, environment, land use, healthcare, education, and so on, huge data management capabilities are necessary to allow to make sense of these data. Especially the field of urban sustainability necessitates these domains to be interrelated and coordinated to collaborate and inform one another. In this respect, the urban data are generated on a regular basis in the form of massive repositories, i.e. huge amounts of data on environmental and socio-economic aspects of urban areas, which provide a powerful microscope of, and a real-time view of what is happening in, the city as to sustainability performance across several spatial scales and over multiple temporal scales. A successful utilization of these valuable data in smart sustainable cities requires advanced big data management tools and methods. This entails the development and implementation of scalable and powerful architectures, best practices, and dedicated computational processes for properly managing data lifecycle throughout various phases of data use, particularly in terms of addressing the issue of variety and velocity, i.e. recognizing their different formats and sources as well as organizing, cataloguing, classifying, and controlling all classes and structures of data. In addition, for smart sustainable city applications, big data management should provide tools for scalable handling of massive data to serve real-time applications and support offline applications (see [1]). For the interested reader, there are several studies that have addressed the topic of big data management (e.g. [84–89]) in terms of concepts, approaches, techniques, and challenges.

Advanced big data analytics techniques and algorithms

In smart sustainable cities, big data analytics should involve highly sophisticated and dedicated techniques and algorithms (data mining, machine learning, statistics, database query, etc.) that can perform complex computational processing of data for timely and accurate decision-making purposes. Traditional techniques and algorithms are inadequate for handling big data associated with smart sustainable city applications due to their high-volume, high-variety, and high-velocity. Urban big data necessitate high speed processing power and high performance to obtain useful results necessary to enhance decision-making pertaining the urban operational functioning and planning of smart sustainable cities. Therefore, existing techniques and algorithms need to be improved in ways that can handle the extreme volume of data, the wide variety of data types, and the time constraints on data processing. In particular, data mining algorithms and techniques are by far unfit for handling big data because they are designed to deal with limited and well-defined datasets (e.g. [90]). In the context of smart sustainable cities, such techniques and algorithms need to be exploited, enhanced, and extended in order to yield the desired outcomes in terms of extracting the useful knowledge (patterns and correlations) necessary for improving sustainability performance (see, e.g. [2, 5, 6]. Alternative or novel solutions in this regard are required to be designed with more scalability and flexibility to handle dynamic and real-time aspects of big data applications for smart sustainable cities, among other things. Moreover, they are to operate as an integral part of cloud computing (PaaS) and thus collaborate across diverse networks for aggregating, fusing, processing, analyzing, and visualizing data collected from countless sensing devices from multiple sources, stored in massive repositories, and coordinated through smart networks. In other words, they need to work effectively across disparate networks, dispersed infrastructures, distributed geographical locations, and heterogeneous computing environments, as well as to be capable of operating in highly scalable and dynamic settings, to reiterate. New approaches to storing, managing, coordinating, and analyzing big data, in particular in relation to smart sustainable city applications should rely on advanced artificial intelligence programs and machine learning techniques. This is in contrast to loading big data into traditional relational databases for analysis, a process that relies on data schema and is time consuming and computationally expensive. For a detailed account of big data analytics techniques and algorithms from a general perspective, the interested reader might want to read Provost and Fawcett [57]. For a relevant account of data mining techniques and algorithms, the interested reader can be directed to Barbi [55]. Also, Chen et al. [58] provide a thorough survey on data mining techniques and algorithms.

Privacy mechanisms and security measures

It is highly important to ensure that all technological components associated with big data and context-aware applications for smart sustainable cities are supported by security measures and privacy mechanisms. It is essential to control big data [91] and context data [92]. Massive repositories of urban data are at stake, and failure to protect these data will pose risks and threats to the functioning of smart sustainable cities as well as to the safety and well-being of their citizens on several scales. Therefore, security measures and privacy mechanisms should be at the core of urban policy and governance practice associated with the design, development, deployment, and implementation of big data and context-aware applications within smart sustainable cities. Any attempt of an unauthorized access, malicious attack, or abuse of information on citizens, infrastructures, networks, and facilities can compromise the integrity of such applications and related services. Smart sustainable cities generate colossal amounts of data on virtually every urban process, which are to be stored, processed, and shared. Urban environments ‘are now being continually forged and re-forged in (sensorial), informational, and communicative processes. It is a world where…cities think of us, where the environment reflexively monitors our behavior’ ([93], p. 1), including whether and the extent to which we behave in a sustainable way through the activities we perform in cities.

However, it is commonly held views that the more cities think of and know about us and technologies monitor urban environments and collect information, ‘the larger becomes the privacy threats, and the larger…the networks, the higher the security risks’ ([92], p. 218). When sensing, computing, and networks become ubiquitous, ‘when everything is embedded with intelligence and connected to everything else via the internet and other networks, the threats and vulnerabilities will become even greater than they are nowadays’ ([92], p. 218). There is a need for technological safeguards as a response to the risks posed by the emerging urban trends of big data analytics and context-aware computing. Clear guidelines, recommendations, and requirements must be identified and put in place in relation to big data and context-aware applications for smart sustainable cities. Among the privacy mechanisms proposed thus far for addressing the issue of privacy include ‘anonymity, pseudonymity, unlinkability, and unobservability’, yet they need to be ‘fully developed, evaluated, and instantiated in their operating environment to test their performance—how well they work’ [92]. Big data and context-aware applications for smart sustainable cities require the development of more robust, if not unconventional, privacy-protecting safeguards by considering the most likely ways through which the information from different urban domains can be leaked and breached. As regards to the security, the scientific challenges ‘include methods supporting the evaluation of risk exposure…, security design principles to enable control of the risk exposure, methods for…security analysis, security of big (and context) data…, secure cloud of physical and smart things, cyber physical systems security, lightweight security solutions, authentication and access control…, identification and biometrics…, cyber-attacks detection and prevention, and so on’ ([92], p. 223–4). While information security risks are of diverse nature, including ‘modification, destruction, theft, or lack of availability of computer assets such as hardware, software, data, and services’ ([94], p. 442), integrity and confidentiality—i.e. protection of information from modification and unauthorized use—should be more of focus as categories of security threats in the ICT of the new wave of computing networks than in the traditional networks [92]. This is due to the fact that there are ‘possible conflict of interests between communicating entities; network convergence; large number of ad-hoc communications; small size and autonomous mode of operation of devices; and resource constraints of mobile devices’ ([95], p. 50). Of critical importance, nevertheless, is to develop a new security paradigm which supports advanced features of context-aware technology, as conventional password entry schemes using traditional input devices have proven to be vulnerable to attacks. To address these issues, new research endeavors are focusing on such new techniques as authenticating with minds; pointing and selection using gaze and keyboard; and gaze-based user authentication [92]. In relation to this, Wright et al. [95] suggest some research directions, including ‘improving access control methods by multimodal fusion, context-aware authentication and unobtrusive biometric modalities’, and ‘increasing security by detection of unusual patterns.’

Standards and open standards

It is important to follow standards when it comes to data integration to make sense of data proliferation as well as to ensure data quality. Standard rules are also needed for evaluating the accuracy and correctness of data and for dealing with such issues as uncertainty and incompleteness of data, especially in relation to real-time big data (and context-aware) applications which require the data to be described using advanced models of the very urban systems that that data are associated with in case of missing and inconsistent data. It is of equal importance to set and comply with standard rules with respect to new applications for advancing urban sustainability to achieve seamless integration between the available urban systems (in terms of infrastructural, physical, operational, and functional forms) and the introduced big data (and context-aware) applications across diverse urban domains. In this regard, the way forward is to carry out a thorough investigation of the different urban entities and actors as well as the infrastructure, built form, administration, and ecosystem and human services as to their operation as urban systems to strategically assess the benefits of new solutions and the readiness of urban stakeholders to join any smart movement associated with improving urban sustainability. In light of such investigation, new practices, regulations, and standard models of design and rules can be developed for big data and context-aware applications for smart sustainable cities.

Concerning other areas related to big data and context-aware applications for smart sustainable cities, it can be advantageous to pursue open standards for designing and implementing solutions with respect to various urban domains, as such applications involve large-scale and heterogeneous data systems. The rationale behind open standards in this respect is to provide some flexibility for scaling up, upgrading, improving, and maintaining applications for smart sustainable cities, as new challenges are most likely to emerge and thus operative solutions may become inadequate to handle potential complexities and difficulties as to translating sustainability into the built, infrastructural, operational, and functional forms of such cities.

The process of data mining

For a detailed description and discussion of these steps, including an account of data mining techniques, algorithms, and tasks based on supervised and unsupervised methods, the interested reader can be directed to [55]. Also, Chen et al. [58] provide a systematic review of data mining in technique view, knowledge view, and application view, supported with the latest application cases related mostly to business intelligence. A well-understood process of urban analytics places a structure on the problems pertaining to urban sustainability, allowing reasonable consistency, repeatability, and objectiveness. The codification of the data mining process espoused here is adapted from CRISP-DM [50, 51], as illustrated in Fig. 2. The deep technical details of the sub-processes of the data mining process and how they relate to urban domains and subdomains in the context of sustainability dimensions is beyond the scope of this paper.

Applicable to the domains of smart sustainable cities, the process of data mining emphasizes the idea of iteration. This implies that solving a particular urban sustainability problem may require going through the process more than once.

The objective of the process of data mining is to discover new knowledge in large masses of urban data pertaining to different sustainability dimensions to improve the environmental and socio-economic performances of smart sustainable cities. Accordingly, such process is concerned with solving problems related to the spatial, physical, infrastructural, operational, and functional forms of such cities in the context of sustainability. For an optimal outcome in the case of spatial data mining, for example, it is invaluable to integrate different methods (spatial analysis, spatial statistics, fuzzy logic, probability theory, cluster analysis, etc.), especially these methods are not mutually exclusive [65]. As to the spatial data, they include such features as spatial, massive temporal, massive multidimensional, and complex [96].

The process of context recognition

These steps can be applicable to the recognition of different dimensions of context, e.g. situational, spatiotemporal, behavioral, and environmental. Researchers from different application domains in the field of context-aware computing have investigated context recognition for the past 2 decades by developing and enhancing a variety of approaches, techniques, and algorithms in relation to a wide variety of context-aware applications [19].

Basic issues of context-aware applications

Context awareness technology for urban sustainability

The focus in this paper is on the physical, situational, spatiotemporal, and socio-economic features of the citizen context as to the interactive applications pertaining to healthcare, education, learning, security, accessibility, utility, and so forth, as well as on the physical, operational, environmental, and spatiotemporal aspects of the urban system context as to the operating and managing processing relating to energy, transport, traffic, mobility, power grid, and so on. Thus, the capabilities of context awareness technology directed for enhanced decision-making and service delivery processes are of high relevance to smart sustainable cities in terms of urban analytics associated with sustainability. The adaptive and responsive features of context-aware applications and systems constitute forms of urban intelligence in its wider sense. One of the cornerstones of ICT of the new wave of computing (UbiComp, AmI, and SenComp) is the intelligent behavior of applications and systems in response to the different contexts of citizens and urban systems (situations, events, locations, settings, behaviors, activities, etc.).

Context-aware computing is becoming increasingly a key component of the infrastructure of smart sustainable cities (e.g. [5, 6, 11, 97] and future smart cities (e.g. [98]). Having access to context information in smart sustainable city applications plays a key role in supporting decision-making processes pertaining to sustainability (e.g. [1, 5, 6, 11]). As one of the prerequisites technologies for enabling ICT of the new wave of computing and for realizing the ICT visions of pervasive computing, context awareness aims to ‘support human action, interaction, and communication in various ways wherever and whenever needed’ ([19], p. 1) by enabling sensorily and computationally augmented urban environments to provide the most efficient services to citizens and intelligent support to urban actors within a variety of settings (e.g. [5, 6]). Entailing a set of advanced computational functionalities, context-aware applications are able to, by relying on context knowledge, control over processes and automate operative tools as well as provide services and support decision-making needs, thereby exhibiting intelligent behavior. In short, context awareness technology is associated with control, automation, optimization, and management, as well as with the adaptation of services. The system behavior’s adaptation is based either on pre-programmed heuristics or real-time reasoning capabilities. The purpose of machine learning and reasoning is to monitor the behaviors of urban systems and citizens and the changes in their environment using sensors of many types to generate inferences (high-abstractions of contexts) based on reasoning mechanisms, and then use physical actors (actuators) or application actions to react and pre-act accordingly in ways that are more constructive in terms of enhancing urban operations, functions, and services in line with the goals of sustainable development. The widespread adoption of diverse sensors within cities provides interactions through opportunistic and people-centric sensing [99, 100]. In this regard, context-aware applications can monitor what is happening in urban environments, analyze, interpret, and respond to them in a variety of ways—be it in relation to smart energy, smart street lights, smart traffic, smart transport, smart mobility, smart education, smart healthcare, or smart safety—across several spatial and temporal scales (e.g. [4]). It is becoming increasingly evident that smart urban environments based on context-aware technologies—especially within smarter cities as future forms of smart cities, namely ambient cities, sentient cities, ubiquitous cities, Internet-of-everything, and real-time cities (e.g. [2, 5, 9, 10, 98–105]; Kyriazis et al. 106)—which can support sustainable urban living in various ways through intelligent service provision and decision support, will be commonplace in the near future.

Context awareness and its feasibility in urban intelligence

Being of prominence and significance to ICT of the new wave of computing in the context of smart sustainable cities [5, 6, 55], context awareness is grounded in the idea that it becomes possible to, through the use of artificial intelligence, detect, monitor, analyze, and model situations of urban life in ways that enable a wide variety of applications and systems to adaptively and proactively take more knowledgeable actions. This relates to the prevailing idea in AmI, UbiComp, and SenComp that the environment can sense and intelligently react to contextual features of urban systems and citizens in the realm of smart sustainable cities of the future [5, 6]. This constitutes also a core characteristic aspect of what is labelled ‘smarter cities’ [5, 6]. However, in contrast to the context-aware applications directed to humans (cognitive, emotional, social, and conversational features of context), where the feasibility issues are essentially linked with the inherent complexity surrounding the modeling of situations of human life [19], the context-aware applications directed for urban systems are not expected to involve such feasibility issues since it is possible to model situations of urban systems as intelligent operating and organizing entities. However, these models should be sophisticated enough and well suited to include the evolution of urban phenomena under major changes. In other words, new models should focus on addressing the problems of sustainability and urbanization. Investigating approaches to context modeling (representation and reasoning techniques) constitutes a large part of a growing body of research on the use of context awareness as a technique for developing context-aware applications and systems that can adapt to and act autonomously on behalf of citizens and urban systems. ICT of the new wave of computing will enhance the quality and speed of urban models in terms of both computational capabilities and structures, as well as the scope of the evolving issues that new modeling approaches can address [2]. Moreover, with the huge potential of artificial intelligence programs and machine learning and data mining techniques being under development for smart sustainable cities of the future, novel methods will emerge to address the issues related to most of the reasoning processes suggested for urban scenarios as to the complexity of generating inferences based on relatively limited and imperfect sensor data. Explicitly, new development are expected to provide ways of filling in missing data and addressing data uncertainty and vagueness using dynamic models of the very systems that these data pertains to.

Sensor observations and dynamic urban models

Context-aware applications and systems involve sensors to monitor or observe different contextual aspects of intelligent entities in urban environments, analyze and interpret the collected contextual data, generate inferences by reasoning against context models, and then react (and pre-act) accordingly. These processes pertain to both urban systems and information systems used by citizens. Regarding urban systems, multilevel integrated modeling is central to the endeavor of merging real-time data with traditional data across urban domains as sectional sources in a way that link real-time issue to long term strategic planning in terms of sustainability, i.e. energy systems, transport systems, and traffic systems. However, the idea of smart sustainable cities of the future is that, based on context-aware computing, related applications and systems can be made more perceptive and responsive by becoming aware of their surroundings (monitoring or observing the urban environment in its various forms, including physical, environmental, operational, behavioral, spatiotemporal, and socio-economic) and reacting to this awareness that can be attained by means of multiple, diverse sensors embedded throughout the urban environment that help build and maintain diverse models that represent the state of the dynamically changing and evolving urban world. Both the observed information about and the existing dynamic models for this world serve as input for the process of computational understanding, which involves the analysis and estimation of what is happening in the urban environment. The dynamic models can be viewed as interpretations of the urban environment, and thus are to be continuously improved to inform intelligent decision-making through machine reasoning about the meaning of all kinds of relevant situations taking place in that environment. Hence, they stand between the urban environment to be sensed and analyzed based on context data and the abstract notion of application actions. Dynamic models represent situations as problem and solution statements or sets of propositions expressing relationships among urban constructs which form the vocabulary of, and used to describe problems and to specify their solutions within urban sustainability domain, to draw on Bibri [19]. They therefore serve as an important input together with the collected sensor data from multiple sources to context-aware applications and systems. Important to note is that new dynamic models should be developed based on what constitutes smart sustainable cities in terms of their systems, processes, and forms. They are key in context information processing associated with real-time (and sometimes offline) applications in relation to such cities. Crucially, such models must be comprehensive, consistent, flexible, robust, and have a high degree of fidelity with real world urban phenomena.

A common use of the sensing and computing devices in smart sustainable cities of the future is to build and maintain an urban world model (see Fig. 3), which allows various context-aware applications and systems to be constructed and operate intelligently to catalyze and boost the process of sustainable development using such strategies as optimization, control, and management concerning urban systems as well as service efficiency and enhancement with regard to citizens.

Figure 3 illustrates an adaptive context awareness process where the smart sustainable city information and urban systems incrementally create the urban world they observe using sensors (and datasets). In this process, the users of such systems (citizens and urban operators/administrators) can be taken into the loop to train and detect contexts on the spot; the systems learn new situations by example, with their users as the teachers. This flexible learning scheme, which is essential to the evolution of urban models, is known as incremental learning, that is, old classes can be retrained should they have changed, and new classes can be trained with no need to retrain the ones that were already trained (see [19]). In this way, urban contexts can be learned and recognized automatically, that is, sensor observations can be associated to human-defined context labels using machine learning and reasoning techniques.

Urban context recognition techniques and algorithms

Conceptual context models and a framework for integrating the key ingredients

As high-level abstractions of contexts can be semantically abstracted from contextual cues extracted from low-level context data obtained from physical sensors, human knowledge and interpretation of the urban world must be formally conceptualized and modeled according to certain formalisms. In smart sustainable cities, conceptual context models are concerned with what constitutes various types of contexts and their conceptual structures depending on the application domain. While the semantics of what constitutes ‘context’ has been widely discussed in the literature and defining what constitutes context information has been studied extensively in relation to humans (e.g. [19]), little, if no, attention has been given to what constitutes context information in relation to urban systems (energy, traffic, transport, etc.). This is a fertile research area in the realm of smart sustainable cities. Generally, context information in the urban domain refers to the representation of the situation of an entity (energy system in a building, a traffic system in a district, a transport system in an urban area characterized by mixed-land use and density, etc.) in some computer system run by, for example, an urban department, where a set of contextual features are of interest to a provider of operational functioning services for assessing the timeliness and context-dependent aspects of the system behavior. Works that can identify qualitative features of urban system context information remain scant. And those related to citizens are numerous and diverse (see [19] for a detailed survey). However, most of the latter class of works does not provide formal representations of the proposed models.

However, one of the challenges in smart sustainable cities is to provide frameworks that cover the class of context-aware applications that exhibit computational understanding and intelligent behavior in relation to sustainability performance. Here computational understanding entails performing the analysis and interpretation of context data, including reasoning about such data, and estimating what is happening in the urban environment (high-level abstractions of context pertaining to various urban systems), a process for which input are observed information about the situations of urban systems over time (i.e. urban monitoring) and dynamic models for the spatial, spatiotemporal, physical, infrastructural, environmental, and socio-economic processes of smart sustainable cities. Note that different types of models are needed. Intelligent behavior entails context-aware applications coming up with and firing situation-dependent actions that provide support for different aspects of sustainability. With the above in mind, a basic framework can be suggested, which combines different models and methods, as illustrated in Fig. 4.

Figure 4 shows a basic framework which combines urban state and history models, environment state and history models, dynamic process models (about urban functioning), dynamic environment process models, ontologies and knowledge from city-related disciplines (including urban sustainability), and analysis methods on the basis of such models, such as spatial analysis, spatiotemporal analysis, environmental analysis, behavioral analysis, and so on. As a template for the class of context-aware applications showing computational understanding and intelligent behavior, the framework can encompass slots where the content of applications specific to various urban systems together with the generic operative methods can be filled in order to obtain an executable design for a working application. Accordingly, context-aware applications can show advanced computational understanding of the urban environment and react from this understanding in a knowledgeable manner through intelligent decision-making for improving different aspects of sustainability through optimization, control, management, and planning of urban systems.

A basic multilayered architecture of context information processing

Researchers from different application domains have investigated context awareness for the past 2 decades or so by developing a diversity of architectures (see [19] for an overview). Context awareness involves a wide range of architectures that basically aim to provide the appropriate infrastructures for context-aware applications pertaining to diverse application domains. Context-aware applications are based on a multilayered architecture, as shown in Fig. 5. Here the focus is on urban systems.