NDTS Technical Implementation - Revision history

Mmaciejewski at 16:50, 17 July 2025

2025-07-17T16:50:33Z

← Older revision		Revision as of 16:50, 17 July 2025
Line 1:		Line 1:
			{{NDTS_Navbox}}
	== Overview ==		== Overview ==

Mmaciejewski at 16:43, 17 July 2025

2025-07-17T16:43:14Z

← Older revision		Revision as of 16:43, 17 July 2025
Line 1:		Line 1:
	== Overview ==		== Overview ==

	The NAN Data Transport System (NDTS) was engineered for robust deployment on NMR spectrometer workstations. Its core design objectives were to: ~~(1)~~ avoid disrupting data acquisition~~; (2)~~ provide automatic harvesting with minimal user intervention~~; (3)~~ operate across a broad range of platforms~~; and (4)~~ guarantee secure and reliable data transfer.		The NAN Data Transport System (NDTS) was engineered for robust deployment on NMR spectrometer workstations. Its core design objectives were to:

			* avoid disrupting data acquisition
			* provide automatic harvesting with minimal user intervention
			* operate across a broad range of platforms
			* guarantee secure and reliable data transfer.

	== Supported Environments ==		== Supported Environments ==

Mmaciejewski at 16:26, 17 July 2025

2025-07-17T16:26:44Z

Mmaciejewski: Created page with "== Overview == The NAN Data Transport System (NDTS) was engineered for robust deployment on NMR spectrometer workstations. Its core design objectives were to: (1) avoid disrupting data acquisition; (2) provide automatic harvesting with minimal user intervention; (3) operate across a broad range of platforms; and (4) guarantee secure and reliable data transfer. === Supported Environments === NDTS operates across a wide array of spectrometer setups, including Bruker TopS..."

2025-07-17T16:25:23Z

Created page with "== Overview == The NAN Data Transport System (NDTS) was engineered for robust deployment on NMR spectrometer workstations. Its core design objectives were to: (1) avoid disrupting data acquisition; (2) provide automatic harvesting with minimal user intervention; (3) operate across a broad range of platforms; and (4) guarantee secure and reliable data transfer. === Supported Environments === NDTS operates across a wide array of spectrometer setups, including Bruker TopS..."

New page

== Overview ==

The NAN Data Transport System (NDTS) was engineered for robust deployment on NMR spectrometer workstations. Its core design objectives were to: (1) avoid disrupting data acquisition; (2) provide automatic harvesting with minimal user intervention; (3) operate across a broad range of platforms; and (4) guarantee secure and reliable data transfer.

=== Supported Environments ===
NDTS operates across a wide array of spectrometer setups, including Bruker TopSpin (v3.x and v4.x), VnmrJ, and OpenVnmrJ. These systems run on diverse operating systems such as RedHat, CentOS, Ubuntu, AlmaLinux, and Windows 10, covering nearly three decades of variability in hardware, libraries, GUI toolkits, and security policies.

=== NDTS Daemon ===
The NDTS daemon is implemented in '''C++98''', chosen for its broad compatibility and minimal reliance on external libraries. To simplify deployment, the binary is statically linked, avoiding dynamic dependencies. Legacy Python environments and Java were excluded to reduce compatibility issues and runtime overhead. Additional work was required to accommodate '''Windows''' systems, which lack POSIX compliance.

The daemon performs several key functions:
# Detects the launch of TopSpin or VnmrJ, identifying the user and confirming control of the spectrometer.
# Monitors acquisition activity:
## '''VnmrJ''': Listens for InfoProc network events signaling state transitions (e.g., idle to acquiring).
## '''TopSpin 3.x''': Uses '''inotify''' (or '''ReadDirectoryChangesW''' on Windows) to watch accounting file changes; detects experiment completion.
## '''TopSpin 4.x''': Reads a shared memory (shmem) file in the user-specific TopSpin directory to capture both acquisition start and end.
# Decides whether to harvest an experiment using the following criteria:
## The user is authorized for harvesting.
## GUI toggle for harvesting is enabled.
## The directory contains a valid '''fid''' or '''ser''' file.
## Experiment job number is below a configurable threshold (used to exclude test runs or maintenance).
## Total acquisition time exceeds 1 second.
## The pulse program is not one of: '''topshim''', '''rga''', '''wobb''', or '''gs''' (TopSpin only).
# Transfers data (including pulse program and probe details) to the NDTS Gateway. If transfer fails, data is cached locally and retried. All actions are logged with full auditing.
# Every 10 minutes, the daemon:
## Retries any failed data uploads.
## Sends a heartbeat signal.
## Requests updated metadata (users, projects, studies, and samples).
## Detects if the GUI has closed and resets NDTS defaults.

In manual harvesting mode, most automated checks are bypassed. Previously harvested datasets are skipped unless the '''Force''' option is explicitly enabled.

=== NDTS GUI ===
The NDTS graphical interface is built using '''Tcl/Tk''', ensuring compatibility with both legacy and modern versions (pre/post Tcl 8.4). It connects to the daemon over a local TCP/IP port. Upon exit, the daemon resets any user-specific settings. While multiple users may launch the GUI, active settings apply only to the current spectrometer user.

=== NDTS Gateway ===
Each facility includes a '''secure NDTS Gateway''', located on the same subnet as its spectrometer systems. Written in '''Python 3''', the gateway accepts data from authorized IP addresses using a restricted socket protocol that allows no command execution and is hardened against buffer overflows.

Gateway responsibilities:
* Forwards outbound-only requests to the NAN receiver, including:
** Heartbeats (fire-and-forget)
** Metadata requests (checksum-based)
** Experiment uploads (acknowledged with success/failure)
* Retries failed uploads but not failed heartbeat or metadata fetches.
* Uses HTTPS (port 443) with certificate authentication and 10-minute access tokens. Expired tokens are replaced automatically.
* Transfers data using '''rsync over SSH''' (port 22), ensuring encryption and integrity via checksums.
* Only outbound ports (443 and 22) are needed; no inbound access is required. Access may be limited to a fixed receiver IP.

=== NAN Receiver ===
The '''NAN Receiver''' ingests data from facility gateways into a temporary staging area. If any payload is malformed, the gateway is notified and will retry submission until successful. The receiver is a Flask-based, Python 3 service distributed across multiple servers to support horizontal scaling.

A dedicated parser service:
* Processes newly received datasets immediately.
* Periodically re-evaluates previously failed or missed experiments (every 10 minutes).
* Supports bulk re-processing when metadata schemas change.

=== Data Lifecycle ===
NDTS ensures durability and high availability through a multi-tiered storage strategy:
* '''Staging Storage''': Real-time replicated using '''Qumulo''' (https://qumulo.com); purged after successful archiving.
* '''Primary Archive''': Hosted on '''Dell EMC Isilon''' (https://infohub.delltechnologies.com/...).
* '''Disaster Recovery''': WORM-enabled '''S3 bucket''' on '''HP Scality''' (https://www.scality.com), geo-replicated across four locations.
* '''Metadata''': Stored in '''PostgreSQL''', with live replication to a separate data center. Daily backups enable full restore if needed.

This design provides a secure, scalable, and fault-tolerant system for managing spectrometer data across the NAN federation.

← Older revision		Revision as of 16:26, 17 July 2025
Line 3:		Line 3:
	The NAN Data Transport System (NDTS) was engineered for robust deployment on NMR spectrometer workstations. Its core design objectives were to: (1) avoid disrupting data acquisition; (2) provide automatic harvesting with minimal user intervention; (3) operate across a broad range of platforms; and (4) guarantee secure and reliable data transfer.		The NAN Data Transport System (NDTS) was engineered for robust deployment on NMR spectrometer workstations. Its core design objectives were to: (1) avoid disrupting data acquisition; (2) provide automatic harvesting with minimal user intervention; (3) operate across a broad range of platforms; and (4) guarantee secure and reliable data transfer.

	=== Supported Environments ===		== Supported Environments ==
	NDTS operates across a wide array of spectrometer setups, including Bruker TopSpin (v3.x and v4.x), VnmrJ, and OpenVnmrJ. These systems run on diverse operating systems such as RedHat, CentOS, Ubuntu, AlmaLinux, and Windows 10, covering nearly three decades of variability in hardware, libraries, GUI toolkits, and security policies.		NDTS operates across a wide array of spectrometer setups, including Bruker TopSpin (v3.x and v4.x), VnmrJ, and OpenVnmrJ. These systems run on diverse operating systems such as RedHat, CentOS, Ubuntu, AlmaLinux, and Windows 10, covering nearly three decades of variability in hardware, libraries, GUI toolkits, and security policies.

	=== NDTS Daemon ===		== NDTS Daemon ==
	The NDTS daemon is implemented in '''C++98''', chosen for its broad compatibility and minimal reliance on external libraries. To simplify deployment, the binary is statically linked, avoiding dynamic dependencies. Legacy Python environments and Java were excluded to reduce compatibility issues and runtime overhead. Additional work was required to accommodate '''Windows''' systems, which lack POSIX compliance.		The NDTS daemon is implemented in '''C++98''', chosen for its broad compatibility and minimal reliance on external libraries. To simplify deployment, the binary is statically linked, avoiding dynamic dependencies. Legacy Python environments and Java were excluded to reduce compatibility issues and runtime overhead. Additional work was required to accommodate '''Windows''' systems, which lack POSIX compliance.

Line 12:		Line 12:
	# Detects the launch of TopSpin or VnmrJ, identifying the user and confirming control of the spectrometer.		# Detects the launch of TopSpin or VnmrJ, identifying the user and confirming control of the spectrometer.
	# Monitors acquisition activity:		# Monitors acquisition activity:
	## '''VnmrJ''': Listens for InfoProc network events signaling state transitions (e.g., idle to acquiring).		#* '''VnmrJ''': Listens for InfoProc network events signaling state transitions (e.g., idle to acquiring).
	## '''TopSpin 3.x''': Uses '''inotify''' (or '''ReadDirectoryChangesW''' on Windows) to watch accounting file changes; detects experiment completion.		#* '''TopSpin 3.x''': Uses '''inotify''' (or '''ReadDirectoryChangesW''' on Windows) to watch accounting file changes; detects experiment completion.
	## '''TopSpin 4.x''': Reads a shared memory (shmem) file in the user-specific TopSpin directory to capture both acquisition start and end.		#* '''TopSpin 4.x''': Reads a shared memory (shmem) file in the user-specific TopSpin directory to capture both acquisition start and end.
	# Decides whether to harvest an experiment using the following criteria:		# Decides whether to harvest an experiment using the following criteria:
	## The user is authorized for harvesting.		#* The user is authorized for harvesting.
	## GUI toggle for harvesting is enabled.		#* GUI toggle for harvesting is enabled.
	## The directory contains a valid '''fid''' or '''ser''' file.		#* The directory contains a valid '''fid''' or '''ser''' file.
	## Experiment job number is below a configurable threshold (used to exclude test runs or maintenance).		#* Experiment job number is below a configurable threshold (used to exclude test runs or maintenance).
	## Total acquisition time exceeds 1 second.		#* Total acquisition time exceeds 1 second.
	## The pulse program is not one of: '''topshim''', '''rga''', '''wobb''', or '''gs''' (TopSpin only).		#* The pulse program is not one of: '''topshim''', '''rga''', '''wobb''', or '''gs''' (TopSpin only).
	# Transfers data (including pulse program and probe details) to the NDTS Gateway. If transfer fails, data is cached locally and retried. All actions are logged with full auditing.		# Transfers data (including pulse program and probe details) to the NDTS Gateway. If transfer fails, data is cached locally and retried. All actions are logged with full auditing.
	# Every 10 minutes, the daemon:		# Every 10 minutes, the daemon:
	## Retries any failed data uploads.		#* Retries any failed data uploads.
	## Sends a heartbeat signal.		#* Sends a heartbeat signal.
	## Requests updated metadata (users, projects, studies, and samples).		#* Requests updated metadata (users, projects, studies, and samples).
	## Detects if the GUI has closed and resets NDTS defaults.		#* Detects if the GUI has closed and resets NDTS defaults.

	In manual harvesting mode, most automated checks are bypassed. Previously harvested datasets are skipped unless the '''Force''' option is explicitly enabled.		In manual harvesting mode, most automated checks are bypassed. Previously harvested datasets are skipped unless the '''Force''' option is explicitly enabled.

	=== NDTS GUI ===		== NDTS GUI ==
	The NDTS graphical interface is built using '''Tcl/Tk''', ensuring compatibility with both legacy and modern versions (pre/post Tcl 8.4). It connects to the daemon over a local TCP/IP port. Upon exit, the daemon resets any user-specific settings. While multiple users may launch the GUI, active settings apply only to the current spectrometer user.		The NDTS graphical interface is built using '''Tcl/Tk''', ensuring compatibility with both legacy and modern versions (pre/post Tcl 8.4). It connects to the daemon over a local TCP/IP port. Upon exit, the daemon resets any user-specific settings. While multiple users may launch the GUI, active settings apply only to the current spectrometer user.

	=== NDTS Gateway ===		== NDTS Gateway ==
	Each facility includes a '''secure NDTS Gateway''', located on the same subnet as its spectrometer systems. Written in '''Python 3''', the gateway accepts data from authorized IP addresses using a restricted socket protocol that allows no command execution and is hardened against buffer overflows.		Each facility includes a '''secure NDTS Gateway''', located on the same subnet as its spectrometer systems. Written in '''Python 3''', the gateway accepts data from authorized IP addresses using a restricted socket protocol that allows no command execution and is hardened against buffer overflows.

Line 47:		Line 47:
	* Only outbound ports (443 and 22) are needed; no inbound access is required. Access may be limited to a fixed receiver IP.		* Only outbound ports (443 and 22) are needed; no inbound access is required. Access may be limited to a fixed receiver IP.

	=== NAN Receiver ===		== NAN Receiver ==
	The '''NAN Receiver''' ingests data from facility gateways into a temporary staging area. If any payload is malformed, the gateway is notified and will retry submission until successful. The receiver is a Flask-based, Python 3 service distributed across multiple servers to support horizontal scaling.		The '''NAN Receiver''' ingests data from facility gateways into a temporary staging area. If any payload is malformed, the gateway is notified and will retry submission until successful. The receiver is a Flask-based, Python 3 service distributed across multiple servers to support horizontal scaling.

Line 55:		Line 55:
	* Supports bulk re-processing when metadata schemas change.		* Supports bulk re-processing when metadata schemas change.

	=== Data Lifecycle ===		== Data Lifecycle ==
	NDTS ensures durability and high availability through a multi-tiered storage strategy:		NDTS ensures durability and high availability through a multi-tiered storage strategy:
	* '''Staging Storage''': Real-time replicated using '''Qumulo''' (https://qumulo.com); purged after successful archiving.		* '''Staging Storage''': Real-time replicated using '''Qumulo''' (https://qumulo.com); purged after successful archiving.