NDTS Technical Implementation

Overview

The NAN Data Transport System (NDTS) was engineered for robust deployment on NMR spectrometer workstations. Its core design objectives were to:

• avoid disrupting data acquisition
• provide automatic harvesting with minimal user intervention
• operate across a broad range of platforms
• guarantee secure and reliable data transfer.

Supported Environments

NDTS operates across a wide array of spectrometer setups, including Bruker TopSpin (v3.x and v4.x), VnmrJ, and OpenVnmrJ. These systems run on diverse operating systems such as RedHat, CentOS, Ubuntu, AlmaLinux, and Windows 10, covering nearly three decades of variability in hardware, libraries, GUI toolkits, and security policies.

NDTS Daemon

The NDTS daemon is implemented in C++98, chosen for its broad compatibility and minimal reliance on external libraries. To simplify deployment, the binary is statically linked, avoiding dynamic dependencies. Legacy Python environments and Java were excluded to reduce compatibility issues and runtime overhead. Additional work was required to accommodate Windows systems, which lack POSIX compliance.

The daemon performs several key functions:

1. Detects the launch of TopSpin or VnmrJ, identifying the user and confirming control of the spectrometer.
2. Monitors acquisition activity:
   • VnmrJ: Listens for InfoProc network events signaling state transitions (e.g., idle to acquiring).
   • TopSpin 3.x: Uses inotify (or ReadDirectoryChangesW on Windows) to watch accounting file changes; detects experiment completion.
   • TopSpin 4.x: Reads a shared memory (shmem) file in the user-specific TopSpin directory to capture both acquisition start and end.
3. Decides whether to harvest an experiment using the following criteria:
   • The user is authorized for harvesting.
   • GUI toggle for harvesting is enabled.
   • The directory contains a valid fid or ser file.
   • Experiment job number is below a configurable threshold (used to exclude test runs or maintenance).
   • Total acquisition time exceeds 1 second.
   • The pulse program is not one of: topshim, rga, wobb, or gs (TopSpin only).
4. Transfers data (including pulse program and probe details) to the NDTS Gateway. If transfer fails, data is cached locally and retried. All actions are logged with full auditing.
5. Every 10 minutes, the daemon:
   • Retries any failed data uploads.
   • Sends a heartbeat signal.
   • Requests updated metadata (users, projects, studies, and samples).
   • Detects if the GUI has closed and resets NDTS defaults.

In manual harvesting mode, most automated checks are bypassed. Previously harvested datasets are skipped unless the Force option is explicitly enabled.

NDTS GUI

The NDTS graphical interface is built using Tcl/Tk, ensuring compatibility with both legacy and modern versions (pre/post Tcl 8.4). It connects to the daemon over a local TCP/IP port. Upon exit, the daemon resets any user-specific settings. While multiple users may launch the GUI, active settings apply only to the current spectrometer user.

NDTS Gateway

Each facility includes a secure NDTS Gateway, located on the same subnet as its spectrometer systems. Written in Python 3, the gateway accepts data from authorized IP addresses using a restricted socket protocol that allows no command execution and is hardened against buffer overflows.

Gateway responsibilities:

• Forwards outbound-only requests to the NAN receiver, including:
  • Heartbeats (fire-and-forget)
  • Metadata requests (checksum-based)
  • Experiment uploads (acknowledged with success/failure)
• Retries failed uploads but not failed heartbeat or metadata fetches.
• Uses HTTPS (port 443) with certificate authentication and 10-minute access tokens. Expired tokens are replaced automatically.
• Transfers data using rsync over SSH (port 22), ensuring encryption and integrity via checksums.
• Only outbound ports (443 and 22) are needed; no inbound access is required. Access may be limited to a fixed receiver IP.

NAN Receiver

The NAN Receiver ingests data from facility gateways into a temporary staging area. If any payload is malformed, the gateway is notified and will retry submission until successful. The receiver is a Flask-based, Python 3 service distributed across multiple servers to support horizontal scaling.

A dedicated parser service:

• Processes newly received datasets immediately.
• Periodically re-evaluates previously failed or missed experiments (every 10 minutes).
• Supports bulk re-processing when metadata schemas change.

Data Lifecycle

NDTS ensures durability and high availability through a multi-tiered storage strategy:

• Staging Storage: Real-time replicated using Qumulo (https://qumulo.com); purged after successful archiving.
• Primary Archive: Hosted on Dell EMC Isilon (https://infohub.delltechnologies.com/...).
• Disaster Recovery: WORM-enabled S3 bucket on HP Scality (https://www.scality.com), geo-replicated across four locations.
• Metadata: Stored in PostgreSQL, with live replication to a separate data center. Daily backups enable full restore if needed.

This design provides a secure, scalable, and fault-tolerant system for managing spectrometer data across the NAN federation.