NAN Data Transport System - Revision history

Mmaciejewski at 16:24, 17 July 2025

2025-07-17T16:24:55Z

← Older revision		Revision as of 16:24, 17 July 2025
Line 5:		Line 5:
	== Overview ==		== Overview ==
	The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data.		The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data.

			See the [[NDTS Technical Implementation]] page for detailed information on how NDTS functions and some of the design decisions.

	== NDTS Components ==		== NDTS Components ==

Mmaciejewski: /* Data Flow Summary */

2025-07-17T16:18:52Z

Data Flow Summary

Mmaciejewski: /* Data Flow Summary */

2025-07-17T16:17:55Z

Data Flow Summary

← Older revision		Revision as of 16:17, 17 July 2025
Line 50:		Line 50:

	=== Experimental Data (Spectrometer → NAN Archive) ===		=== Experimental Data (Spectrometer → NAN Archive) ===
			Datasets automatically or manually harvested are processed and sent to the NAN archive in the following manner:
	# A user completes an acquisition on a spectrometer.		# A user completes an acquisition on a spectrometer.
	# The '''Daemon''' detects the completed experiment and sends it to the '''Gateway'''.		# The '''Daemon''' detects the completed experiment and sends it to the '''Gateway'''.
Line 68:		Line 69:

	The '''Daemon''' uses the updated metadata to set NDTS defaults for NAN users and to present information in the NDTS GUI.		The '''Daemon''' uses the updated metadata to set NDTS defaults for NAN users and to present information in the NDTS GUI.

			=== Heartbeats (Spectrometer → NAN Archive) ===
			At timed intervals (10 minutes) heartbeats from the spectrometer with reports on NDTS and instrument status are sent.

			# The Daemon packages heartbeat information and sends it to the Gateway
			# The '''Gateway''' transmits the data to the '''Receiver''' at UCHC.
			# The Receiver processes the heartbeat data and transmits it to the Elastic database
			# The heartbeat information is summarized and visualized on the virtual Network Operations Center (vNOC).

	== Facility Manager NDTS Responsibilities ==		== Facility Manager NDTS Responsibilities ==

Mmaciejewski: /* Data Flow Summary */

2025-07-17T15:50:36Z

Data Flow Summary

← Older revision		Revision as of 15:50, 17 July 2025
Line 49:		Line 49:
	== Data Flow Summary ==		== Data Flow Summary ==

	=== Experimental ~~data~~ (Spectrometer ~~-->~~ NAN Archive) ===		=== Experimental Data (Spectrometer → NAN Archive) ===
	# A user completes an acquisition on a spectrometer.		# A user completes an acquisition on a spectrometer.
	# The Daemon detects the completed experiment and sends it to the Gateway.		# The '''Daemon''' detects the completed experiment and sends it to the '''Gateway'''.
	# The Gateway transmits the data to the Receiver at UCHC.		# The '''Gateway''' transmits the data to the '''Receiver''' at UCHC.
	# The Receiver accepts the data and hands it off to the Parser.		# The '''Receiver''' accepts the data and hands it off to the '''Parser'''.
	# The Parser extracts metadata and stores it in the PostgreSQL and Elasticsearch databases.		# The '''Parser''' extracts metadata and stores it in the PostgreSQL and Elasticsearch databases.
	# The experiment data is stored in primary storage and backed up to disaster recovery storage.		# The experiment data is stored in primary storage and backed up to disaster recovery storage.
	# The data becomes visible in the NAN Portal (e.g., Data Browser, vNOC) within seconds.		# The data becomes visible in the NAN Portal (e.g., Data Browser, vNOC) within seconds.
Line 60:		Line 60:
	Failures at any stage result in data being spooled locally and retried automatically.		Failures at any stage result in data being spooled locally and retried automatically.

	=== User and ~~probe metadata~~ (NAN Archive ~~-->~~ Spectrometer) ===		=== User and Probe Metadata (NAN Archive → Spectrometer) ===
	Metadata includes facility users, their projects, studies, samples, NDTS default settings, mappings of local workstation users to NAN users, and the currently installed probe. These data are packaged and staged for retrieval by the ~~gateway~~.		Metadata includes facility users, their projects, studies, samples, NDTS default settings, mappings of local workstation users to NAN users, and the currently installed probe. These data are packaged and staged for retrieval by the '''Gateway'''.

	# ~~The~~ metadata ~~destined~~ for the spectrometers ~~is packaged and staged by a service on the Receiver~~ every 10 minutes.		# A service on the '''Receiver''' packages and stages metadata for the spectrometers every 10 minutes.
	# The Gateway ~~examines~~ the checksum of the metadata ~~package compared to~~ current version ~~on the Gateway~~. If different ~~a new metadata~~ package ~~is retrieved - if not~~, ~~the~~ retrieval is skipped. This ~~is done~~ every 10 minutes		# The '''Gateway''' checks the checksum of the staged metadata against its current version. If different, it retrieves the updated package; otherwise, retrieval is skipped. This check occurs every 10 minutes.
	# The spectrometer ~~repeats~~ the process ~~like the Gateway and will pull~~ a new ~~metadata~~ package ~~every 10 minutes as long as there~~ are ~~changes~~		# The spectrometer performs the same check-and-retrieve process every 10 minutes, retrieving a new package only if changes are detected.

	The ~~daemon~~ uses the ~~new~~ metadata to set NDTS defaults for NAN users and ~~presenting the~~ information to the NDTS GUI.		The '''Daemon''' uses the updated metadata to set NDTS defaults for NAN users and to present information in the NDTS GUI.

	== Facility Manager NDTS Responsibilities ==		== Facility Manager NDTS Responsibilities ==

Mmaciejewski at 15:49, 17 July 2025

2025-07-17T15:49:12Z

← Older revision		Revision as of 15:49, 17 July 2025
Line 48:		Line 48:

	== Data Flow Summary ==		== Data Flow Summary ==

			=== Experimental data (Spectrometer --> NAN Archive) ===
	# A user completes an acquisition on a spectrometer.		# A user completes an acquisition on a spectrometer.
	# The Daemon detects the completed experiment and sends it to the Gateway.		# The Daemon detects the completed experiment and sends it to the Gateway.
Line 58:		Line 60:
	Failures at any stage result in data being spooled locally and retried automatically.		Failures at any stage result in data being spooled locally and retried automatically.

	== Facility Manager Responsibilities ==		=== User and probe metadata (NAN Archive --> Spectrometer) ===
			Metadata includes facility users, their projects, studies, samples, NDTS default settings, mappings of local workstation users to NAN users, and the currently installed probe. These data are packaged and staged for retrieval by the gateway.

			# The metadata destined for the spectrometers is packaged and staged by a service on the Receiver every 10 minutes.
			# The Gateway examines the checksum of the metadata package compared to current version on the Gateway. If different a new metadata package is retrieved - if not, the retrieval is skipped. This is done every 10 minutes
			# The spectrometer repeats the process like the Gateway and will pull a new metadata package every 10 minutes as long as there are changes

			The daemon uses the new metadata to set NDTS defaults for NAN users and presenting the information to the NDTS GUI.

			== Facility Manager NDTS Responsibilities ==
	Facility Managers are expected to:		Facility Managers are expected to:
	* Purchase the Gateway computer and install a modern Linux (preferably Ubuntu / Xubuntu / Mint or other Debian based OS)		* Purchase the Gateway computer and install a modern Linux (preferably Ubuntu / Xubuntu / Mint or other Debian based OS)
			* Keep the Gateway OS up-to-date with security patches
	* Install and configure [[NDTS Gateway Installation\|Gateway]] and [[NDTS Daemon Installation\|Daemon]] software		* Install and configure [[NDTS Gateway Installation\|Gateway]] and [[NDTS Daemon Installation\|Daemon]] software
			* Update Gateway and Daemon software within a reasonable time-frame of new releases being made available
	* Manage facility users through the [[Facility Dashboards\|Facility Dashboard]]		* Manage facility users through the [[Facility Dashboards\|Facility Dashboard]]
	* [[Dataset reassignment\|Reassign]] “unselected” or misattributed data through the [[Datasets\|Dataset Browser]]		* [[Dataset reassignment\|Reassign]] “unselected” or misattributed data through the [[Datasets\|Dataset Browser]]

Mmaciejewski at 21:06, 16 July 2025

2025-07-16T21:06:50Z

← Older revision		Revision as of 21:06, 16 July 2025
Line 63:		Line 63:
	* Install and configure [[NDTS Gateway Installation\|Gateway]] and [[NDTS Daemon Installation\|Daemon]] software		* Install and configure [[NDTS Gateway Installation\|Gateway]] and [[NDTS Daemon Installation\|Daemon]] software
	* Manage facility users through the [[Facility Dashboards\|Facility Dashboard]]		* Manage facility users through the [[Facility Dashboards\|Facility Dashboard]]
	* Reassign “unselected” or misattributed data through the [[Datasets\|Dataset Browser]]		* [[Dataset reassignment\|Reassign]] “unselected” or misattributed data through the [[Datasets\|Dataset Browser]]
	* Monitor the health of NDTS for their facility, including heartbeats, through the virtual NAN Operating Center (vNOC)		* Monitor the health of NDTS for their facility, including heartbeats, through the virtual NAN Operating Center (vNOC)

Mmaciejewski: /* Security */

2025-06-25T15:30:38Z

Security

← Older revision		Revision as of 15:30, 25 June 2025
Line 67:		Line 67:

	== Security ==		== Security ==
	Out-of-date operating systems on spectrometer workstations may lack modern encryption. To mitigate this risk, NDTS employs a dedicated '''Gateway''' computer between the workstations and the NDTS '''Receiver'''. The Gateway runs a current Linux distribution, and users are expected to apply security updates promptly.

	Because the Gateway resides on the same internal network as the workstations, dataset transfers from a workstation to the Gateway occur over an unencrypted channel; this local scope generally makes encryption unnecessary.		* Out-of-date operating systems on spectrometer workstations may lack modern encryption. To mitigate this risk, NDTS employs a dedicated '''Gateway''' computer between the workstations and the NDTS '''Receiver'''. The Gateway runs a current Linux distribution, and users are expected to apply security updates promptly.
			* Because the Gateway resides on the same internal network as the workstations, dataset transfers from a workstation to the Gateway occur over an unencrypted channel; this local scope generally makes encryption unnecessary.
	All outbound communication originates from the Gateway; NAN datacenter services never initiate connections to facility Gateways. Transfers from the Gateway to the Receiver are fully encrypted, and mutual TLS certificates ensure the Gateway is connected to the correct Receiver. Checksums protect every transmission, and any failed ~~transfer—either~~ workstation-to-Gateway or Gateway-to-~~Receiver—is~~ queued locally for automatic retry.		* All outbound communication originates from the Gateway; NAN datacenter services never initiate connections to facility Gateways. Transfers from the Gateway to the Receiver are fully encrypted, and mutual TLS certificates ensure the Gateway is connected to the correct Receiver. Checksums protect every transmission, and any failed transfer, either (workstation-to-Gateway or Gateway-to-Receiver) is queued locally for automatic retry.
			* Upon arrival at the Receiver, each dataset is replicated across two independent storage systems. After ingestion, the data is stored redundantly in two additional locations, each offering high durability.
	Upon arrival at the Receiver, each dataset is replicated across two independent storage systems. After ingestion, the data is stored redundantly in two additional locations, each offering high durability.

Mmaciejewski: /* Security */

2025-06-25T15:28:41Z

Security

← Older revision		Revision as of 15:28, 25 June 2025
Line 67:		Line 67:

	== Security ==		== Security ==
	~~It is not uncommon for the~~ operating systems on ~~the workstation spectrometers to be out of date and possibly not utilizing the latest~~ encryption ~~technology~~. ~~Because of~~ this NDTS ~~utilizes~~ a Gateway computer ~~as an intermediary~~ between the ~~workstation computers~~ and the NDTS Receiver. The Gateway is a ~~dedicated computer running an up-to-date version of~~ Linux and ~~we encourage~~ users to apply ~~timely~~ security updates ~~on the Gateway~~.		Out-of-date operating systems on spectrometer workstations may lack modern encryption. To mitigate this risk, NDTS employs a dedicated '''Gateway''' computer between the workstations and the NDTS '''Receiver'''. The Gateway runs a current Linux distribution, and users are expected to apply security updates promptly.

	~~Transmission of datasets from the spectrometer workstations to the Gateway are not encrypted, but as~~ the Gateway ~~is co-located within~~ the same network as the ~~spectrometer~~ workstation ~~computers~~ this is generally ~~not an issue~~.		Because the Gateway resides on the same internal network as the workstations, dataset transfers from a workstation to the Gateway occur over an unencrypted channel; this local scope generally makes encryption unnecessary.

	~~Communication between~~ the Gateway ~~and the Receiver is always initiated by the Gateway and thus is outbound traffic.~~ NAN services ~~in the datacenter~~ never ~~attempt~~ to ~~communicate with~~ facility ~~Gateway computers~~. Transfers from the Gateway to the Receiver are fully encrypted and ~~the Gateway and~~ certificates ~~are used to~~ ensure ~~that~~ the Gateway is ~~communicating with~~ the ~~proper~~ Receiver. Checksums ~~are utilized for all transmissions to ensure there was no issue with the transfer. Any~~ failed ~~transmissions from the~~ workstation ~~computers~~ to ~~the~~ Gateway, or ~~the~~ Gateway to ~~the Receiver are spooled~~ locally ~~to be retransmitted later~~.		All outbound communication originates from the Gateway; NAN datacenter services never initiate connections to facility Gateways. Transfers from the Gateway to the Receiver are fully encrypted, and mutual TLS certificates ensure the Gateway is connected to the correct Receiver. Checksums protect every transmission, and any failed transfer—either workstation-to-Gateway or Gateway-to-Receiver—is queued locally for automatic retry.

	~~Once datasets arrive~~ at the Receiver ~~their files are~~ replicated across two storage systems ~~and once ingested~~ the data is ~~again~~ stored in two locations, ~~both with~~ high ~~levels of data~~ durability.		Upon arrival at the Receiver, each dataset is replicated across two independent storage systems. After ingestion, the data is stored redundantly in two additional locations, each offering high durability.

Mmaciejewski: /* Security */

2025-06-25T15:27:02Z

Security

← Older revision		Revision as of 15:27, 25 June 2025
Line 67:		Line 67:

	== Security ==		== Security ==
			It is not uncommon for the operating systems on the workstation spectrometers to be out of date and possibly not utilizing the latest encryption technology. Because of this NDTS utilizes a Gateway computer as an intermediary between the workstation computers and the NDTS Receiver. The Gateway is a dedicated computer running an up-to-date version of Linux and we encourage users to apply timely security updates on the Gateway.

			Transmission of datasets from the spectrometer workstations to the Gateway are not encrypted, but as the Gateway is co-located within the same network as the spectrometer workstation computers this is generally not an issue.

			Communication between the Gateway and the Receiver is always initiated by the Gateway and thus is outbound traffic. NAN services in the datacenter never attempt to communicate with facility Gateway computers. Transfers from the Gateway to the Receiver are fully encrypted and the Gateway and certificates are used to ensure that the Gateway is communicating with the proper Receiver. Checksums are utilized for all transmissions to ensure there was no issue with the transfer. Any failed transmissions from the workstation computers to the Gateway, or the Gateway to the Receiver are spooled locally to be retransmitted later.

			Once datasets arrive at the Receiver their files are replicated across two storage systems and once ingested the data is again stored in two locations, both with high levels of data durability.

Mmaciejewski at 14:57, 25 June 2025

2025-06-25T14:57:53Z

← Older revision		Revision as of 14:57, 25 June 2025
Line 38:		Line 38:
	\|-		\|-
	\| '''Primary Storage'''\|\| UCHC Data Center \|\|		\| '''Primary Storage'''\|\| UCHC Data Center \|\|
	* Stores ~~canonical~~ copies of all collected experimental data		* Stores copies of all collected experimental data
	\|-		\|-
	\| '''Disaster Recovery Storage'''\|\| Geo-dispersed \|\|		\| '''Disaster Recovery Storage'''\|\| Geo-dispersed \|\|
Line 64:		Line 64:
	* Manage facility users through the [[Facility Dashboards\|Facility Dashboard]]		* Manage facility users through the [[Facility Dashboards\|Facility Dashboard]]
	* Reassign “unselected” or misattributed data through the [[Datasets\|Dataset Browser]]		* Reassign “unselected” or misattributed data through the [[Datasets\|Dataset Browser]]
	* Monitor the health of NDTS for their facility, including heartbeats, through the virtual NAN Operating Center (vNOC)~~<br />~~		* Monitor the health of NDTS for their facility, including heartbeats, through the virtual NAN Operating Center (vNOC)
	~~----~~

	~~<span style~~=~~"font-size:90%">For assistance, contact the NAN Repository administrator or submit a ticket from the Facility Dashboard.</span>~~		== Security ==

← Older revision		Revision as of 16:18, 17 July 2025
Line 50:		Line 50:

	=== Experimental Data (Spectrometer → NAN Archive) ===		=== Experimental Data (Spectrometer → NAN Archive) ===
	Datasets automatically or manually harvested are processed and sent to the NAN ~~archive in the following manner~~:		Datasets that are automatically or manually harvested are processed and sent to the NAN Archive as follows:
	# A user completes an acquisition on a spectrometer.		# A user completes an acquisition on a spectrometer.
	# The '''Daemon''' detects the completed experiment and sends it to the '''Gateway'''.		# The '''Daemon''' detects the completed experiment and sends it to the '''Gateway'''.
Line 65:		Line 65:

	# A service on the '''Receiver''' packages and stages metadata for the spectrometers every 10 minutes.		# A service on the '''Receiver''' packages and stages metadata for the spectrometers every 10 minutes.
	# The '''Gateway''' ~~checks~~ the checksum of the staged metadata against its current version. If different, it retrieves the updated package; otherwise, retrieval is skipped. This check occurs every 10 minutes.		# The '''Gateway''' compares the checksum of the staged metadata against its current version. If different, it retrieves the updated package; otherwise, retrieval is skipped. This check occurs every 10 minutes.
	# The spectrometer performs the same check~~-and-retrieve process~~ every 10 minutes~~, retrieving~~ a new package only if changes are detected.		# The spectrometer performs the same check every 10 minutes and retrieves a new package only if changes are detected.

	The '''Daemon''' uses the updated metadata to set NDTS defaults for NAN users and to present information in the NDTS GUI.		The '''Daemon''' uses the updated metadata to set NDTS defaults for NAN users and present information in the NDTS GUI.

	=== Heartbeats (Spectrometer → NAN Archive) ===		=== Heartbeats (Spectrometer → NAN Archive) ===
	At ~~timed~~ intervals ~~(10 minutes) heartbeats from~~ the spectrometer ~~with~~ reports on NDTS and instrument status ~~are sent~~.		At 10-minute intervals, the spectrometer sends heartbeat reports containing NDTS and instrument status.

	# The Daemon packages heartbeat information and sends it to the Gateway		# The '''Daemon''' packages the heartbeat information and sends it to the '''Gateway'''.
	# The '''Gateway''' transmits the data to the '''Receiver''' at UCHC.		# The '''Gateway''' transmits the data to the '''Receiver''' at UCHC.
	# The Receiver processes the heartbeat data and ~~transmits~~ it to the ~~Elastic~~ database		# The '''Receiver''' processes the heartbeat data and stores it in the Elasticsearch database.
	# ~~The heartbeat~~ information is summarized and visualized on the virtual Network Operations Center (vNOC).		# Heartbeat information is summarized and visualized in the virtual Network Operations Center (vNOC).

	== Facility Manager NDTS Responsibilities ==		== Facility Manager NDTS Responsibilities ==