Building a 40+ Year Global Dataset Through CEOS Agency Collaboration

Satellite data has a unique ability to provide long-term consistent observations, which is particularly important for climate-related applications. One key focus area of the CEOS Working Group on Information Systems and Services (WGISS) is the preservation and stewardship of data, to ensure EO data is available for many years to come. 

In the dawn of space-based EO, satellite data was not preserved as well as it is today, leading WGISS to work on heritage data recovery. The effort aims to recover past datasets and make them available for current and future use, supporting climate-related research. 

WGISS is currently addressing data from the Advanced Very High Resolution Radiometer (AVHRR) which, in its various versions, has been flown on 18 different missions, most prominently the NOAA Polar Operational Environmental Satellites (POES) and EUMETSAT MetOp series. It was first flown on TIROS-N in 1978, and followed by NOAA-6 in 1979. The third version of AVHRR is flown today on NOAA-18, NOAA-19, MetOp-B and MetOp-C. 

Two different datasets are collected by AVHRR: Global Area Coverage (GAC, at 4 km spatial resolution) and Local Area Coverage (LAC, at 1 km spatial resolution). A consistent and complete global dataset for GAC is provided by NOAA from 1978 onwards, and a complete dataset for LAC from 2008 is provided by EUMETSAT. However, before 2008, LAC data was downlinked only as high-resolution picture transmission (HRTP) data to various data centers around the globe, and hence no complete global record exists. The LAC data, at a finer 1 km resolution, has a high value for the retrieval of Essential Climate Variables (ECVs). Hence, the climate community requested that WGISS undertake a global effort to retrieve and make available a global LAC dataset.

With the European Space Agency (ESA) leading the effort out of the ESRIN Heritage Data Centre, the first step was to build a complete European LAC dataset from 1981 onwards. Data was merged from ESA, the University of Bern and the Dundee receiving station, and is now available through ESA’s dissemination systems. The second step was to further investigate where additional archives may be located all over the world and to start addressing the recovery and consolidation of a global dataset in cooperation with CEOS Agencies through connections via WGISS.

The European Dataset

The ESA Heritage Space Programme worked together with the University of Bern and Dundee Ground Station to consolidate their holdings of AVHRR data from 16 different platforms. Over 260,000 products were harmonised in terms of L1B format and content, consolidated, and reprocessed to Level-1C. The data is now openly available via ESA’s dissemination systems, and safely archived for future use.

A tool was developed to select AVHRR products available at ESA over specific Earth points and time range, to understand gaps in the dataset and trigger data retrieval activities. Black cells indicate no available data, while the red-green heat map indicates the number of products available for each day of the year. 

ESA is now working to reprocess the European AVHRR LAC products series to generate a L1C Fundamental Data Record (FDR), which will include both thermal infrared (TIR) and visible-near infrared (NVIR) channels. The dataset will be cross-mission calibrated, and will feature geocoding and orthorectification, as well as error propagation and accurate uncertainty estimates in the calibration module. The project is also looking to extend the European dataset with 2020-2025 European data, as well as other selected regions (Greenland, Argentina, Kenya, South Africa, Brazil and more). The project is expected to be completed in mid-2026.

The Global Dataset

The following is the current status on various interactions between CEOS Agencies in the frame of WGISS to find, access, and consolidate AVHRR archives:

• Europe (ESA): The European dataset from 1981 onwards has been processed to Level-1B and Level-1C and is now accessible on a free and open basis through ESA dissemination systems.
• Germany (DLR): The DLR catalogue contains data from missions NOAA-14 to NOAA-19, including Land Surface Temperature (LST), Sea Surface Temperature (SST), and Vegetation Index (NDVI). However, it is not clear if the raw data is available at DLR, and there is high overlap with the dataset already possessed by ESA.
• Poland (IGIK): The Polish Institute of Geodesy and Cartography department of Remote Sensing has held NOAA satellite data since 1996 (NOAA-11 to NOAA-19). The full 350 GB record has been retrieved and saved in the ESA archive. ESA will now look into processing the data into Level-1B and -1C, before opening access to users. 
• North America (NOAA & USGS): NOAA hosts the largest archive of AVHRR GAC and LAC data covering North America and other regions where data were acquired and downloaded to US stations. The USGS EROS Archive contains some AVHRR archives (more details are to come regarding the size and coverage of the archive). 
• Canada (NRCan/CCMEO): Level-1C AVHRR data covering 1981-2013 is available at CCMEO, and open source data is available through the WMO CGMS data distribution policy. More details about ingestion of their archive into the global dataset is to come.
• Argentina (GiDyC-Servicio Meteorológico Nacional): The archive contains more than 21000 products (1 TB) from satellites NOAA-14 to NOAA-18, and Metop-A/-B (1995-2015). The data transfer to ESA is ongoing, after which the data will be processed to Level-1B and -1C, and access will be opened to users through ESA systems. 
• Brazil (INPE): The INPE archive contains 287,325 products from 1998-2024. The data transfer to ESA has been completed, and the data will now be processed to Level-1B and Level-1C before access is opened to users through ESA systems. 
• 

  Device used to roll/unroll the Exabyte tapes from the University of Hawaii archive

  Pacific Ocean (University of Hawaii): An AVHRR dataset was acquired at the Hawaii University between 1990 and 2000. The data (on Exabyte tapes) was shipped to ESA and a transcription chain is being assembled. A special device was manufactured to roll/unroll the tapes at very slow speed and clean the tape surface from moisture before transcription. Once transcribed and recovered, data will be processed to Level-1B and Level-1C before access is opened to users through ESA systems.
• South Africa (SANSA): 1 km NOAA AVHRR data was received at Hartebeesthoek, South Africa and maintained by SANSA from 1985-2009. The 7684 products (464 GB) have been transferred to ESA, and will be processed to Level-1B and Level-1C before access is opened to users through ESA systems. 
• Central Africa (Kenya, ASI/UniRoma): Data acquired at the Malindi station in Kenya from 2001 to 2009 was stored on 44 DLT tapes (8452 products, 614 GB). The DLTs have successfully been transcribed by ESA. Data will be processed to Level-1B and Level-1C before access is opened to users through ESA systems. 
• China (CMA/NSMC): Data after 2003 is available at NSMC, and ESA is investigating how to download these products via the machine-to-machine interface. More details on the archive are to come – AVHRR 1 km data covering the Himalaya-Hindukush region for the period 1981 – 1992 would be of exceptional value.
• India (ISRO): ISRO is processing its AVHRR data from 1994 into Level-1A. ESA has provided the pyGAC software to ISRO to process data into Level-1B and Level-1C products, which will then be disseminated by ISRO.
• Mongolia (Remote Sensing Department, Information and Research Institute of Meteorology, Hydrology and Environment): The archive contains 5,016 scenes from 1993-1999. ESA has been provided access to the data and data retrieving has begun.
• Australia (CSIRO): Six ground stations in Australia receive AVHRR LAC data. CSIRO provides a stitched archive, merging the local datasets to create a single dataset with national coverage. Data is available from 1981. CSIRO will reprocess all their data into higher levels and provide access to end users.
• 1Km worldwide campaign: a worldwide campaign was run between 1992 and 1998 in cooperation between USGS and ESA for the acquisition of 1Km LAC data worldwide through a series of Ground Stations coordinated by the two agencies. ESA has retrieved all the acquired data and is processing them into Level-1B and Level-1C. Data will then be made accessible to all users via ESA systems.  

Over 15 organisations have been involved in this global effort – showcasing the true power of CEOS. However, the effort is not complete – WGISS suspects there are many more meteorological and other organisations around the world who may hold AVHRR archives. Anyone in possession of archived AVHRR data is invited to contact the project lead, Mirko Albani (mirko.albani@esa.int).
ESA will shortly perform a worldwide AVHRR LAC data gap analysis, extending the one done for Europe. This will highlight areas of the world where data is still missing, and trigger further recovery efforts. 

WGISS thanks all organisations for their support of this initiative – the complete dataset will be of tremendous value and support various research efforts for years to come. 

WGISS would also like to ask the community if there are other datasets which would benefit from recovery in a similar way. Please contact us if you have any suggestions.