NAN Data Transport System (NDTS) Overview

Navigation of NDTS Documentation

• NAN Data Transport System
  • NDTS Overview
  • NDTS Installation
    • Installation Overview
    • Installing and Configuring the Gateway Software
    • Installing and Configuring the Daemon Software
  • TopSpin 3.x Requirement
    • Enable TopSpin accounting
      • How NDTS monitors accounting file
  • Managing, Monitoring, and Logging
    • Managing, Monitoring, and Logging the Daemon
  • NDTS Usage Guides
    • NDTS Graphical User Interface (NDTS-GUI)
    • Understanding Experiment Harvesting

Overview

The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data.

NDTS Components

The NDTS system consists of local and central components working together to collect, transfer, store, and index NMR datasets

Component	Location	Role
Daemon	Spectrometer Workstation	Detects completed experiments, associates metadata, and sends data to the Gateway. Sends heartbeat information to the Gateway Pulls user information from the Gateway
NDTS GUI	Spectrometer Workstation	Shows data harvesting statuses Allows the operator to change the NAN user and enter additional metadata
Gateway	Within NMR facility network	Receives data from all Daemons and relays it to the Receiver. Pulls user information from the NAN Receiver. The Gateway should be a dedicated computer, but has minimal requirements with regard to CPU, memory, and storage. The exception to that is if the Gateway is setup to archive all datasets as a backup it needs the storage capacity to accommodate the data or even better mounted to an external storage device.
Receiver	UCHC Data Center	Accepts experiment data and metadata from Gateways Packages user information destined for the spectrometer workstations
Parser	UCHC Data Center	Parses datasets to extract additional metadata, writes database entries, and stores dataset files to primary and disaster recovery storage appliances
PostgreSQL Database	UCHC Data Center	Stores datasets and their structured metadata
Primary Storage	UCHC Data Center	Stores copies of all collected experimental data
Disaster Recovery Storage	Geo-dispersed	Maintains redundant backups of all experimental data on a WORM S3 bucket
Elasticsearch Database	UCHC Data Center	Indexes statistics about harvested datasets and heartbeat information for visualization from the virtual NAN Operation Center (vNOC)

Data Flow Summary

1. A user completes an acquisition on a spectrometer.
2. The Daemon detects the completed experiment and sends it to the Gateway.
3. The Gateway transmits the data to the Receiver at UCHC.
4. The Receiver accepts the data and hands it off to the Parser.
5. The Parser extracts metadata and stores it in the PostgreSQL and Elasticsearch databases.
6. The experiment data is stored in primary storage and backed up to disaster recovery storage.
7. The data becomes visible in the NAN Portal (e.g., Data Browser, vNOC) within seconds.

Failures at any stage result in data being spooled locally and retried automatically.

Facility Manager Responsibilities

Facility Managers are expected to:

• Purchase the Gateway computer and install a modern Linux (preferably Ubuntu / Xubuntu / Mint or other Debian based OS)
• Install and configure Gateway and Daemon software
• Manage facility users through the Facility Dashboard
• Reassign “unselected” or misattributed data through the Dataset Browser
• Monitor the health of NDTS for their facility, including heartbeats, through the virtual NAN Operating Center (vNOC)

Security

It is not uncommon for the operating systems on the workstation spectrometers to be out of date and possibly not utilizing the latest encryption technology. Because of this NDTS utilizes a Gateway computer as an intermediary between the workstation computers and the NDTS Receiver. The Gateway is a dedicated computer running an up-to-date version of Linux and we encourage users to apply timely security updates on the Gateway.

Transmission of datasets from the spectrometer workstations to the Gateway are not encrypted, but as the Gateway is co-located within the same network as the spectrometer workstation computers this is generally not an issue.

Communication between the Gateway and the Receiver is always initiated by the Gateway and thus is outbound traffic. NAN services in the datacenter never attempt to communicate with facility Gateway computers. Transfers from the Gateway to the Receiver are fully encrypted and the Gateway and certificates are used to ensure that the Gateway is communicating with the proper Receiver. Checksums are utilized for all transmissions to ensure there was no issue with the transfer. Any failed transmissions from the workstation computers to the Gateway, or the Gateway to the Receiver are spooled locally to be retransmitted later.

Once datasets arrive at the Receiver their files are replicated across two storage systems and once ingested the data is again stored in two locations, both with high levels of data durability.