Speaker: Chris Kurtz (Arizona State University)

Title: Scheduled Scientific Data Releases Using .backup Volumes

Abstract:

The Mars Space Flight Facility at Arizona State University is required by contract with NASA's Jet Propulsion Laboratory to provide releases of data from our instruments every six months to both the general public and scientists by means of NASA's Planetary Data System. Releases were originally performed by rsyncing the entire data repository (because old data may have been reprocessed or updated) from an expensive redundant storage (NetApp) to a single local webserver and multiple off-site locations using a process that would take days to complete. Now this mission-critical data is stored across eight AFS File Servers and accessed by the   science team members using standard AFS volumes. The public websites (now easily load balanced because the data is no longer a local copy) read only from the specific .backup volumes of the data, which are released every six months to match the internal data. By using the .backup snapshots, we are able to have immediate data releases that do not duplicate terabytes of data. Additionally, because of the granularity of individual AFS volumes, specific .backup volumes may be released more frequently, as reprocessing occurs.

Slides: PDF PPT