[XPublish]: Performance and scaling improvements

[jonmjoyce]

Project Description

The XPublish community has discussed ideas for measuring and improving performance of serving data, such as caching, integrating with dask for distributed processing of large datasets, and more. This work will improve the ability for XPublish to serve very large datasets.

Expected Outcomes

• XPublish design discussions
• Direction forward for how to improve XPublish performance
• Test cases for measuring XPublish performance
• Create a feature roadmap

Skills required

Python with async paradigms
Basic understanding of dask or other distributed computing libraries

Expertise

Expert

Topic Lead(s)

Jonathan Joyce jonmjoyce, Matt Iannucci mpiannucci

Relevant links

xpublish-community/xpublish#208

[MathewBiddle]

Thank you for taking the time to propose this topic! From the Code Sprint topic survey, this has garnered a lot of interest.

Following the contributing guidelines on selecting a code sprint topic I have assigned this topic to @jonmjoyce . Unless indicated otherwise, the assignee will be responsible for identifying a plan for the code sprint topic, establishing a team, and taking the lead on executing said plan. The first action for the lead is to:

• Create a Sprint Topic website - https://github.com/ioos/ioos-code-sprint/blob/main/CONTRIBUTING.md#creating-sprint-topic-webpages