CEOS Working Group on Information Systems & Services Technology Exploration Subgroup Webinars
Webinar 8: CEOS Jupyter Notebooks Day – Supporting Global Best Practice and Training
October 21, 2022
Location: Virtual Zoom meeting
The CEOS Working Group on Capacity Building and Data Democracy and the Working Group on Information Systems and Services ran a joint webinar on Jupyter Notebooks for Capacity Development. The aim of this webinar was to introduce space agencies and environmental organisations worldwide to Jupyter Notebooks and take a tour of emerging services from CEOS Agencies and their applications. We illustrated how Jupyter Notebooks can be used to support capacity development and the exploitation of Earth Observation data by a broad range of users.
We surveyed the registrants of the event and the responded with 2 key requirements
- The development of a Jupyter Notebooks Best Practice
- Jupyter Notebooks based training
This event was working meeting to facilitate solutions to those needs
Agenda (Times in UTC)
14:00 – 15:30 Round Table Part 1
15:30 – 15:45 Break
15:45 – 17:00 Round Table Part 2
17:00 -18:00 Break
18:00 – 19:00 Best Practice Review
19:00 – 20:00 Joint training discussion
Contact: esther.conway@stfc.ac.uk
Webinar 7: Jupyter Notebooks for Capacity Development Webinar
Introduction
The CEOS Capacity Development and Data democracy Working Group and the Working group on Information Systems and Services will be running a joint webinar on Jupyter notebooks for Capacity Development. The aim of this webinar is to introduce space agencies and environmental organisations worldwide to Jupyter Notebooks and take a tour of emerging services from CEOS agencies and their applications. We intend to illustrate how they can be used to support capacity development and the exploitation of Earth Observation data by a broad range of users. There will be two sessions via zoom to allow for global attendance.
To view the recorded webinar, please click here.
Agenda
What are Jupyter Notebooks, and why they have the potential to support capacity development?
We will provide an overview of what a Jupyter Notebook is and the benefits of using one. We will demonstrate using a simple example of plotting sea surface temperature data, navigating an archive and adjusting the colour scale. We continue by describing the ways it supported by open-source resources and by different types of platform/environment. We then discuss how collaborative research and activities such as international Hackathons can also be supported.
Jupyter Hub and Notebooks on Data Analysis Platforms:
We take a look at two examples from the UK’s JASMIN Jupyter notebook service, which can access over 20 Petabytes of data on the CEDA archive. We take a look at the Sentinel 5p global archive of data and demonstrate how by using a very basic notebook, we can explore questions such as how did pollution levels change in large cities during the Pandemic. We then continue by looking at the smaller scale specialist example of regional NCEO biomass maps. We can demonstrate how in addition to obtaining domain-specific information from data, we can also train users on some technical aspects such as libraries, modules and shapefiles.
Open Data Cube and Google Earth Engine – A Jupyter Notebook Sandbox Demonstration
The Open Data Cube (ODC) Google Sandbox is a free and open programming interface that connects users to Google Earth Engine datasets. The open-source tool allows users to run Python application algorithms using Google’s Colab notebook environment. This demonstration will look at two examples Landsat applications focused on scene-based cloud statistics and historic water extent. Basic operation of the tool will support unlimited users for small-scale analyses and training but can also be scaled in size and scope with Google Cloud resources to support enhanced user needs.
ESA PGDS data cube and Times series Data
The ESA PDGS Data Cube is a pixel-based access service that enables human and machine-to-machine interfaces for Heritage Missions (HM), Third-Party Missions (TPM) and Earth Explorer (EE) datasets. The pixel-based access service provides the users with advanced retrieval capabilities such as time series extraction, data subsetting, mosaicking, band combinations and indexes generation (e.g. NDVI, anomalies, …) directly from the EO-SIP packages with no need of data duplication or data preparation.
In addition to the Explorer web-based graphic user interface, the ESA PDGS Data Cube service also provides the Jupyter processing environment to allow users to import, write and execute code that runs close to the data. This demonstration will showcase how to retrieve Soil Moisture time-series using the Jupyter environment in order to generate thematic maps (monthly anomalies map) over an area of interest. The benefit of using the pixel-based service with respect to traditional access services in term of resources usage will be also highlighted.
Earth Analytics and Interoperability Lab – Big Data Processing
The CEOS Earth Analytics Interoperability Lab (EAIL) is a platform for CEOS projects to test interoperability in a live EO ecosystem. EAIL is hosted on Amazon Web Services and includes facilities for Jupyter notebooks, scalable compute infrastructure for integrated analysis and data pipelines that can connect to new and existing CEOS data discovery and access services. This demonstration will show how we use Jupyter notebooks with the python Dask library to efficiently compute and perform large-scale analyses (10s GB) with interactive plotting and scalable compute resources in EAIL.
Capacity Development Panel and Discussion
Kenton Ross from NASA’s Capacity Building and Applied Sciences Program will lead discussions with a panel of international experts. We will explore the needs of EO data users worldwide and how this can be supported by CEOS agencies.
Speakers: Kenton Ross (NASA), Yousuke Ikehata (JAXA), Esther Conway (NCEO/UKSA), Brian Killough (SEO/NASA), Giuseppe Troina (ESA), Matt Paget (CSIRO)
Webinar 6: Future Data Access & Analysis Architecture Initiative Webinar (Joint WGISS/WGCapD Activity)
Date: September 4, 2019
Description: The CEOS Future Data Access and Analysis Architecture (FDA) initiative has resulted in a range of new approaches for Earth observation (EO) data access and analysis being deployed globally by CEOS Agencies. Many of the pilot activities are now practical implementations with new standards for Analysis Ready Data (ARD), advanced user-programmable Exploitation Platforms, and vast amounts of data readily available on cloud platforms with open source analytics tools like the CEOS Open Data Cube (ODC). Many of these components allow user algorithms to be brought to the data, rather than having to download and run locally, thus opening new opportunities for researchers, industry, and government use. The FDA ecosystem is still evolving and challenging to navigate, and the CEOS Working Groups are collaborating closely to make it simpler to not only discover EO data, but also to exploit it using the various elements of FDA.
Webinar 5: Agile Development and the Scaled Agile Framework
Date: July 31, 2018
Description: Agile practices have proven themselves time and again for small startups and web applications, but at times those practices seem at odds with the discipline and planning required to build large enterprise systems. In this webinar, we will explore how the Scaled Agile Framework can allow large development teams to meet their users’ needs in a collaborative, agile manner.
Ms. Dana Shum, Raytheon NASA EED2 Software Engineering Manager, presented.
Webinar 4: The Open Geospatial Consortium (OGC) Coverage Standards Suite: Introduction & Overview
Date: January 19, 2018
Description: Typically, “Big Data” in the Earth science domain arise from spatio-temporal sensor, image, simulation, and statistics data. The OGC coverage concept represents a unifying model for such data, based on which the OGC Web Coverage Service (WCS), as OGC’s “Big Data” standard, provides versatile functionality. Large and growing tool support as well as the massive data offerings existing – such as the Petabyte services of the EarthServer initiative – highlight relevance of these standards. This has prompted further standardization bodies such as ISO and INSPIRE to adopt these standards, too.
In this webinar, Dr Peter Baumann (Managing Director of Rasdaman) will present the coverage data and service definitions and their use for building portals, with special emphasis on spatio-temporal datacubes. Ample room will be available for Q&A.
Date: August 22, 2017 (Wednesday August 23rd in Asia and Australia)
Agenda: Burgeoning Role of Python for Earth Observation Data Analysis
- Syed Rizvi (Open Data Cube) discusses the use of recently developed Open Data Cube Jupyter notebooks for a water classification application that allowed developers to parse their code into blocks which can be run independently of each other, with variables stored in the background, saving vast amounts of time.
- Rich Signell (USGS) discusses how Jupyter Notebooks have been used by the US IOOS (Integrated Ocean Observing System) to do manual skill assessment of over 20 forecast models for over 1000 sensors by building standards-based Python code that uses catalog search, standard web data service access, custom skill metric calculation and interactive display of results.
- Alber Sanchez (INPE) discusses how Python is used in a web service that reduces the gap between data and analysis for EO scientists. He will present how to retrieve time series of vegetation indexes of the Amazon forest, and how to filter these data.
Presentations:
- Open Data Cube Jupyter Notebooks
- Automating Model Skill Assessment with Python
- Python Used in a Web Service
Webinar 2: Data Cubes for Large Scale Data Analytics
Date: Monday, June 19, 2017.
Description: Recent work by WGISS members has been fleshing out the concept of Data Cubes to enable analysis of large Earth Observation data sets. Please join us as Rob Woodcock of CSIRO (Australia) and Brian Killough of the CEOS System Engineering Office provide an introduction to Data Cubes. Rob will set the stage for Data Cubes with user needs, key features and basic high-level architecture, followed by Brian to talk about some more of the inner workings of Data Cubes.
Webinar 1: Relevancy Ranking of Data Search Results
Date: Tuesday, March 14, 2017
As CEOS Earth Observations grow in both quantity and variety, users face increasing challenges in finding the most appropriate data for their use. Ranking data search results by their expected relevance for the user is potentially one of the most powerful ways to help users quickly zero in on the best data for their purpose. This webinar will look at some of the techniques being used within Earth Observation data search tools, followed by a discussion of possible future directions in relevancy ranking.
Agenda:
- Introduction to the topic – 10 minutes – Chris Lynnes
- Search Relevancy in EOSDIS – 10 minutes – James Norton
- Search Relevancy in GEO Data Access Broker – 10 minutes – Stefano Nativi
- Looking forward in Search Relevancy – 10 minutes – Chris Lynnes & Stefano Nativi
- Questions/Discussion