Universität zu Köln

Leuridan, Mathilde ORCID: 0009-0009-5923-4964 (2026). Efficient Feature Extraction of Petabyte-Scale Datacubes for Weather and Climate. PhD thesis, Universität zu Köln.

PDF phd_thesis_final_publication_ready.pdf - Accepted Version Download (11MB)

Abstract

Data volumes across scientific domains are increasing rapidly, driven by advances in simulation, observation and data-driven modelling. In meteorology in particular, kilometre-scale numerical weather prediction and high-frequency machine learning models produce vast and continuously growing datasets. While this enables new scientific insights, it also creates fundamental challenges for efficiently accessing and delivering data. A key limitation lies in current data access approaches, which typically retrieve large data subsets from storage systems before applying application-specific transformations, such as clipping. This model is increasingly inefficient due to the mismatch between storage layouts, which are optimised for block-based, partly sequential access on disk or tape, and user-driven access patterns, which are often sparse and require fine-grained random access. At the same time, a growing diversity of users and applications demands more flexible and tailored data extraction capabilities. These challenges motivate the need for fundamentally new extraction paradigms that minimise unnecessary data movement while supporting user-specific queries. This thesis introduces a new feature extraction framework, which shifts data selection logic from the post-processing stage to the data access layer. Instead of retrieving large data blocks and subsequently filtering them, the proposed method computes the exact byte ranges required to satisfy a query in advance, thereby significantly reducing I/O. At its core, the framework introduces a generalised feature extraction mechanism, which first uses a geometric selection algorithm to identify relevant data subsets and then directly retrieves only the necessary bytes from backend storage, enabling highly efficient data access. The proposed approach is implemented in a prototype system and evaluated on operational meteorological datasets. Benchmarks show significant performance improvements compared to existing extraction systems, including reductions in both data transfer volumes, as well as I/O operations, by up to a few orders of magnitude. These results directly translate into lower computational load on backend systems and faster delivery of application-ready data. Beyond regular gridded data, the framework is extended to support unstructured grids without requiring assumptions on spatial layout. Building on this, the approach is further generalised to arbitrary datacubes with a new abstraction that captures complex multi-dimensional data spaces, such as those typically encountered in modern Earth system science. Together, these contributions form a unified and flexible feature extraction mechanism applicable across a wide range of scientific datasets. The presented methods enable efficient, scalable and user-centric data access, particularly in operational environments, such as numerical weather prediction, and digital twin systems, such as the Destination Earth initiative. More broadly, the concepts developed in this thesis are applicable to other data-intensive scientific domains, where minimising data movement and enabling precise on-demand access are critical for future workflows.

Item Type:	Thesis (PhD thesis)
Creators:	Creators Email ORCID ORCID Put Code Leuridan, Mathilde mathildel@hotmail.de https://orcid.org/0009-0009-5923-4964 UNSPECIFIED
URN:	urn:nbn:de:hbz:38-810986
Date:	5 August 2026
Language:	English
Faculty:	Faculty of Mathematics and Natural Sciences
Divisions:	Faculty of Mathematics and Natural Sciences > Department of Mathematics and Computer Science > Institute of Computer Science
Subjects:	Data processing Computer science Earth sciences
Uncontrolled Keywords:	Keywords Language Data Management UNSPECIFIED Feature Extraction UNSPECIFIED
Date of oral exam:	4 August 2026
Referee:	Name Academic Title Schultz, Martin Professor
Refereed:	Yes
URI:	http://kups.ub.uni-koeln.de/id/eprint/81098