Persistent earth observation from space with automated interpretation of data and information, including artificial intelligence, cloud solutions and real time on-board processing by sensors

Topic description

Specific Challenge:

The role of Earth observation (EO) and geo information within defence, security and intelligence operations is increasing both from a strategic and tactical point of view. High resolution EO data are undergoing an explosive growth, increasing complexity, like the diversity and higher dimensionality characteristic of the data. EO data are regarded as “big data” and their management requires techniques to solve data-intensive problems. Artificial Intelligence (AI) is key to cope with this challenge.

The huge amount of available data sources requires new methodologies to assist the intelligence community streamline their processing and exploitation. The Activity Based Intelligence (ABI) complements the geo spatial activities by rapidly integrating data from multiple sources to discover relevant patterns, determine and identify changes, and characterize those patterns to drive collection and create decision advantage.

The challenge is to define and test a prototype of a system that implements the concept of ABI which is able, in particular, to:

• Extract entities (e.g. events, changes, features) from different data sources in an automated way;
• Derive insights from the aggregation, correlation and monitoring of such entities in terms of patterns and anomalies.

The challenge is then to provide the ABI system as a new means for ISR (Intelligence, Surveillance and Reconnaissance) to fulfil the command and control needs. This would be achieved by providing tools to analyse in an interactive way the ABI information linked to contextual intelligence information.

The already introduced ABI represents a new analysis methodology focused on the “time dimension” and aimed at discovering relevant patterns, detecting and characterising changes, analysing a target’s behaviour and highlighting anomalies. Detected patterns feed a never-ending observation cycle by driving new collections and generating decision advantage.

Consequently, AI has evolved dramatically over the last years especially with the emergence of deep learning techniques. These techniques rely on the traditional neural network approach and, by exploiting hardware evolution (graphics processing unit availability and cloud), are able to reach unmatched accuracies in the traditional machine learning tasks, such as classification, detection, segmentation, etc. Deep learning is applicable to all the different data analysis needs of ABI, relevant to both EO data (Very High Resolution satellites (VHR), Synthetic Aperture Radar (SAR) satellites, constellation of micro-satellites, etc.) and un-conventional data sources (social network data, news feeds data, etc.).

The scope of this new system is to support the automated information extraction by applying AI techniques whenever possible and in particular for the extraction of entities such as:

• Events, extracted from news feeds and social networks using natural language processing and image analysis with deep learning techniques;
• Changes and features from satellite imagery “time series” using deep learning techniques such as CNN (Convolutional Neural Network).

After the extraction of information from heterogeneous sources, ABI includes the ability to aggregate entities and fuse them using spatio-temporal analysis (e.g. density maps of events over time) in order to create insights on different phenomena.

Targeted activities:

The proposals shall cover the development of a prototype that implements the concepts of ABI and that includes in particular:

• Analysis of automated methods for entity extraction from EO and non-EO data;
• Design of the prototype system including the Preliminary Design Review (PDR) and the Critical Design Review (CDR);
• Development and demonstration of the prototype.

Main high-level requirements:

The proposals shall fulfil the following general requirements:

• State-of-the-art for automated multi-SAR/optical and multi-platform data interpretation focused on time series analysis for ABI;
• Analysis of existing satellite missions to be exploited and fused for ABI tasks and definition of mission requirements to fill gaps in data availability;
• Support to the geospatial intelligence (GEOINT) analyst in the definition of the most suitable satellite and non-satellite data planning and data collection strategy to carry out ABI tasks. As an example, regarding the different sensor types, it is important to underline that:
  • SAR missions offer a primary source of information thanks to their reliability (no clouds or atmospheric effects), geo-location accuracy and capability to detect changes;
  • VHR optical missions and, above all, new missions based on constellations, offer the capability to classify and recognize targets with unprecedented revisit time;
  • MWIR (Medium Wave Infrared), TIR (Thermal Infrared) future missions are able to detect parameters such as heat as an indicator of activity over specific targets.
• Exploitation of innovative AI tools and methodologies (machine learning/deep learning) to support the extraction of relevant information and behaviour from large quantities of unstructured data;
• Extraction of motion information as a powerful activity indicator and for pattern-of-life analysis by exploiting:
  • The capabilities of SAR sensors to highlight moving targets and therefore provide information about “instantaneous activity” over a target area;
  • The capabilities of new missions to provide high definition videos, taken from space, lasting tens of seconds.
• Implementation of functionalities for entity extraction applicable to EO data based on deep learning techniques in order to detect changes and recognize entities;
• Implementation of functionalities for entity extraction applicable to non-EO data to overcome the limitation of data sources like social network and news feeds (e.g. data bias, geo-referencing, multi-language);
• Integration of multiple sources of information to derive insights, including remote sensing (satellite and airborne), local sensors and open source information such as social network data, properly organized, interpreted and analysed;
• Specification and implementation of unconventional data analysis methods on existing data. An example is the use of SAR images to detect and characterise signal from ground-based radars or other electromagnetic devices (interference analysis);
• Use of large-scale (both space and time wise) data filtering and analysis in order to develop an understanding by correlating heterogeneous data sets;
• Provision of specific workflow for ABI tasks for maritime and land application in order to support the GEOINT analysts work;
• Development of the system in a modular and expandable way to incorporate new data streams, new analysis methodologies and operational procedures.

• Enable the application of ABI for large scale, highly automated analysis of multi-source EO (including high revisit and/or high resolution constellations) and non-EO data;
• Use innovative and scalable methodologies to discover location based behaviours and activities relevant for intelligence purposes, taking advantage of the recent technological developments in EO, AI and ICT (Information and Communication Technologies) infrastructures for big data management;
• Enhance the European industry’s capabilities and competiveness to provide operational ABI services exploiting state-of-the-art EO and non-EO data, platforms and methodologies;
• Foster the debate inside the user community about the requirements and challenges of ABI approach for GEOINT applications including, but not limited to, data sources availability and accessibility, data management tools, analysis methodologies, information confidentiality;
• Establish a proof-of-concept or a prototype, which can act as reference for the independent user assessment, in light of product extensions and service improvements and for the further technological developments;
• Provide tools and best practices for the operational application of ABI principles within the intelligence user community;
• Reduce operator workload for data preparation and information extraction in order to concentrate effort on GEOINT contextual analysis;
• Improve reaction time for decision making by leveraging continuous data streaming analysis;
• Implement information superiority.