Abstract:

The results presented are from the NOAA EPP CSC NERTO graduate internship project conducted under the mentorship of Steve Ansari, National Centers for Environmental Information. This NERTO experience aligns with the NOAA Cooperative Science Center in CESSRST-II, supporting the Center’s goal of conducting NOAA mission-aligned collaborative research to understand and predict changes in weather and atmosphere, and land and water, by improving how NOAA precipitation observations are accessed and shared. This project, titled Cloud-Optimized Access to NOAA Observational Data addressed the research question: Can storing NOAA data as partitioned Parquet files in cloud object storage, and reading only the needed portions through HTTP range requests, reduce data transfer and response times enough to support interactive, web-based visualization? The work provides value to the scientific community and program stakeholders by advancing NOAA’s mission to share its knowledge and information with others. It makes cloud-hosted NOAA datasets faster and easier to use: a point query reads only about 1% of a gridded file, station and radar precipitation time series return in under two seconds, and daily snapshots of more than 25,000 stations load in about three seconds. The pipeline also offers a reusable template for other NOAA datasets. Through this internship, the student deepened their understanding of NOAA mission areas and gained enhanced skills in cloud computing, data engineering and pipeline development, Python programming, geospatial and time-series data analysis, and scientific communication.