NAN Data Transport System

Navigation of NDTS Documentation

• NAN Data Transport System
  • NDTS Overview
  • NDTS Installation
    • Installation Overview
    • Installing and Configuring the Gateway Software
    • Installing and Configuring the Daemon Software
  • TopSpin 3.x Requirement
    • Enable TopSpin accounting
      • How NDTS monitors accounting file
  • Managing, Monitoring, and Logging
    • Managing, Monitoring, and Logging the Daemon
  • NDTS Usage Guides
    • NDTS Graphical User Interface (NDTS-GUI)
    • Understanding Experiment Harvesting
  • NDTS Technical Details
    • NDTS Technical Implementation

Overview

The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data.

See the NDTS Technical Implementation page for detailed information on how NDTS functions and some of the design decisions.

NDTS Components

The NDTS system consists of local and central components working together to collect, transfer, store, and index NMR datasets

Component	Location	Role
Daemon	Spectrometer Workstation	Detects completed experiments, associates metadata, and sends data to the Gateway. Sends heartbeat information to the Gateway Pulls user information from the Gateway
NDTS GUI	Spectrometer Workstation	Shows data harvesting statuses Allows the operator to change the NAN user and enter additional metadata
Gateway	Within NMR facility network	Receives data from all Daemons and relays it to the Receiver. Pulls user information from the NAN Receiver. The Gateway should be a dedicated computer, but has minimal requirements with regard to CPU, memory, and storage. The exception to that is if the Gateway is setup to archive all datasets as a backup it needs the storage capacity to accommodate the data or even better mounted to an external storage device.
Receiver	UCHC Data Center	Accepts experiment data and metadata from Gateways Packages user information destined for the spectrometer workstations
Parser	UCHC Data Center	Parses datasets to extract additional metadata, writes database entries, and stores dataset files to primary and disaster recovery storage appliances
PostgreSQL Database	UCHC Data Center	Stores datasets and their structured metadata
Primary Storage	UCHC Data Center	Stores copies of all collected experimental data
Disaster Recovery Storage	Geo-dispersed	Maintains redundant backups of all experimental data on a WORM S3 bucket
Elasticsearch Database	UCHC Data Center	Indexes statistics about harvested datasets and heartbeat information for visualization from the virtual NAN Operation Center (vNOC)

Data Flow Summary

Experimental Data (Spectrometer → NAN Archive)

Datasets that are automatically or manually harvested are processed and sent to the NAN Archive as follows:

1. A user completes an acquisition on a spectrometer.
2. The Daemon detects the completed experiment and sends it to the Gateway.
3. The Gateway transmits the data to the Receiver at UCHC.
4. The Receiver accepts the data and hands it off to the Parser.
5. The Parser extracts metadata and stores it in the PostgreSQL and Elasticsearch databases.
6. The experiment data is stored in primary storage and backed up to disaster recovery storage.
7. The data becomes visible in the NAN Portal (e.g., Data Browser, vNOC) within seconds.

Failures at any stage result in data being spooled locally and retried automatically.

User and Probe Metadata (NAN Archive → Spectrometer)

Metadata includes facility users, their projects, studies, samples, NDTS default settings, mappings of local workstation users to NAN users, and the currently installed probe. These data are packaged and staged for retrieval by the Gateway.

1. A service on the Receiver packages and stages metadata for the spectrometers every 10 minutes.
2. The Gateway compares the checksum of the staged metadata against its current version. If different, it retrieves the updated package; otherwise, retrieval is skipped. This check occurs every 10 minutes.
3. The spectrometer performs the same check every 10 minutes and retrieves a new package only if changes are detected.

The Daemon uses the updated metadata to set NDTS defaults for NAN users and present information in the NDTS GUI.

Heartbeats (Spectrometer → NAN Archive)

At 10-minute intervals, the spectrometer sends heartbeat reports containing NDTS and instrument status.

1. The Daemon packages the heartbeat information and sends it to the Gateway.
2. The Gateway transmits the data to the Receiver at UCHC.
3. The Receiver processes the heartbeat data and stores it in the Elasticsearch database.
4. Heartbeat information is summarized and visualized in the virtual Network Operations Center (vNOC).

Facility Manager NDTS Responsibilities

Facility Managers are expected to:

• Purchase the Gateway computer and install a modern Linux (preferably Ubuntu / Xubuntu / Mint or other Debian based OS)
• Keep the Gateway OS up-to-date with security patches
• Install and configure Gateway and Daemon software
• Update Gateway and Daemon software within a reasonable time-frame of new releases being made available
• Manage facility users through the Facility Dashboard
• Reassign “unselected” or misattributed data through the Dataset Browser
• Monitor the health of NDTS for their facility, including heartbeats, through the virtual NAN Operating Center (vNOC)

Security

• Out-of-date operating systems on spectrometer workstations may lack modern encryption. To mitigate this risk, NDTS employs a dedicated Gateway computer between the workstations and the NDTS Receiver. The Gateway runs a current Linux distribution, and users are expected to apply security updates promptly.
• Because the Gateway resides on the same internal network as the workstations, dataset transfers from a workstation to the Gateway occur over an unencrypted channel; this local scope generally makes encryption unnecessary.
• All outbound communication originates from the Gateway; NAN datacenter services never initiate connections to facility Gateways. Transfers from the Gateway to the Receiver are fully encrypted, and mutual TLS certificates ensure the Gateway is connected to the correct Receiver. Checksums protect every transmission, and any failed transfer, either (workstation-to-Gateway or Gateway-to-Receiver) is queued locally for automatic retry.
• Upon arrival at the Receiver, each dataset is replicated across two independent storage systems. After ingestion, the data is stored redundantly in two additional locations, each offering high durability.