NAN Data Transport System

Navigation of NDTS Documentation

• NAN Data Transport System
  • NDTS Overview
  • NDTS Installation
    • Installation Overview
    • Installing and Configuring the Gateway Software
    • Installing and Configuring the Daemon Software
  • TopSpin 3.x Requirement
    • Enable TopSpin accounting
      • How NDTS monitors accounting file
  • Managing, Monitoring, and Logging
    • Managing, Monitoring, and Logging the Daemon
  • NDTS Usage Guides
    • NDTS Graphical User Interface (NDTS-GUI)
    • Understanding Experiment Harvesting
  • NDTS Technical Details
    • NDTS Technical Implementation

Overview

The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data.

NDTS Components

The NDTS system consists of local and central components working together to collect, transfer, store, and index NMR datasets

Component	Location	Role
Daemon	Spectrometer Workstation	Detects completed experiments, associates metadata, and sends data to the Gateway. Sends heartbeat information to the Gateway Pulls user information from the Gateway
NDTS GUI	Spectrometer Workstation	Shows data harvesting statuses Allows the operator to change the NAN user and enter additional metadata
Gateway	Within NMR facility network	Receives data from all Daemons and relays it to the Receiver. Pulls user information from the NAN Receiver. The Gateway should be a dedicated computer, but has minimal requirements with regard to CPU, memory, and storage. The exception to that is if the Gateway is setup to archive all datasets as a backup it needs the storage capacity to accommodate the data or even better mounted to an external storage device.
Receiver	UCHC Data Center	Accepts experiment data and metadata from Gateways Packages user information destined for the spectrometer workstations
Parser	UCHC Data Center	Parses datasets to extract additional metadata, writes database entries, and stores dataset files to primary and disaster recovery storage appliances
PostgreSQL Database	UCHC Data Center	Stores datasets and their structured metadata
Primary Storage	UCHC Data Center	Stores copies of all collected experimental data
Disaster Recovery Storage	Geo-dispersed	Maintains redundant backups of all experimental data on a WORM S3 bucket
Elasticsearch Database	UCHC Data Center	Indexes statistics about harvested datasets and heartbeat information for visualization from the virtual NAN Operation Center (vNOC)

Data Flow Summary

Experimental data (Spectrometer --> NAN Archive)

1. A user completes an acquisition on a spectrometer.
2. The Daemon detects the completed experiment and sends it to the Gateway.
3. The Gateway transmits the data to the Receiver at UCHC.
4. The Receiver accepts the data and hands it off to the Parser.
5. The Parser extracts metadata and stores it in the PostgreSQL and Elasticsearch databases.
6. The experiment data is stored in primary storage and backed up to disaster recovery storage.
7. The data becomes visible in the NAN Portal (e.g., Data Browser, vNOC) within seconds.

Failures at any stage result in data being spooled locally and retried automatically.

User and probe metadata (NAN Archive --> Spectrometer)

Metadata includes facility users, their projects, studies, samples, NDTS default settings, mappings of local workstation users to NAN users, and the currently installed probe. These data are packaged and staged for retrieval by the gateway.

1. The metadata destined for the spectrometers is packaged and staged by a service on the Receiver every 10 minutes.
2. The Gateway examines the checksum of the metadata package compared to current version on the Gateway. If different a new metadata package is retrieved - if not, the retrieval is skipped. This is done every 10 minutes
3. The spectrometer repeats the process like the Gateway and will pull a new metadata package every 10 minutes as long as there are changes

The daemon uses the new metadata to set NDTS defaults for NAN users and presenting the information to the NDTS GUI.

Facility Manager NDTS Responsibilities

Facility Managers are expected to:

• Purchase the Gateway computer and install a modern Linux (preferably Ubuntu / Xubuntu / Mint or other Debian based OS)
• Keep the Gateway OS up-to-date with security patches
• Install and configure Gateway and Daemon software
• Update Gateway and Daemon software within a reasonable time-frame of new releases being made available
• Manage facility users through the Facility Dashboard
• Reassign “unselected” or misattributed data through the Dataset Browser
• Monitor the health of NDTS for their facility, including heartbeats, through the virtual NAN Operating Center (vNOC)

Security

• Out-of-date operating systems on spectrometer workstations may lack modern encryption. To mitigate this risk, NDTS employs a dedicated Gateway computer between the workstations and the NDTS Receiver. The Gateway runs a current Linux distribution, and users are expected to apply security updates promptly.
• Because the Gateway resides on the same internal network as the workstations, dataset transfers from a workstation to the Gateway occur over an unencrypted channel; this local scope generally makes encryption unnecessary.
• All outbound communication originates from the Gateway; NAN datacenter services never initiate connections to facility Gateways. Transfers from the Gateway to the Receiver are fully encrypted, and mutual TLS certificates ensure the Gateway is connected to the correct Receiver. Checksums protect every transmission, and any failed transfer, either (workstation-to-Gateway or Gateway-to-Receiver) is queued locally for automatic retry.
• Upon arrival at the Receiver, each dataset is replicated across two independent storage systems. After ingestion, the data is stored redundantly in two additional locations, each offering high durability.