NAN Data Transport System: Difference between revisions

From Network for Advanced NMR
Jump to navigationJump to search
← Older editNewer edit →
VisualWikitext
No edit summary
No edit summary
Line 1: Line 1:
= NDTS Facility Manager Guide =
= NDTS Facility Manager Guide =


The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data directly from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the NDTS components, ensuring connectivity, and supporting data access for users.
The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data.


== '''NDTS Architecture Overview''' ==
== '''System Architecture and Data Flow''' ==
NDTS consists of two core components:
The NDTS system consists of local and central components working together to collect, transfer, store, and index NMR experiment data.
* '''NDTS Daemon''' – Runs on each spectrometer workstation. Detects completed acquisitions, manages user and experiment metadata, and transfers data to the gateway.
* '''NDTS Gateway''' – Runs on a central server at the facility. Receives data from all workstations and forwards it to the NDTS Receiver at UCHC.


[[File:NDTS_Architecture_Diagram.png|center|800px|NDTS architecture showing Daemon → Gateway → Receiver pipeline]]
{| class="wikitable"
! Component !! Location !! Role
|-
| '''Daemon''' || Spectrometer Workstation || Detects completed experiments, associates metadata, and sends data to the Gateway.
|-
| '''Gateway''' || Facility Server || Receives data from all Daemons and relays it to the Receiver.
|-
| '''Receiver''' || UCHC Data Center || Accepts experiment data and metadata from Gateways.
|-
| '''Parser''' || UCHC Data Center || Extracts parameters, user info, and structural metadata from each dataset.
|-
| '''PostgreSQL Database''' || UCHC Data Center || Stores structured metadata and associations (users, instruments, facilities, samples, etc.).
|-
| '''Primary Storage''' || UCHC Data Center || Stores canonical copies of all collected experimental data.
|-
| '''Disaster Recovery Storage''' || Off-site || Maintains redundant backups of all experiment data.
|-
| '''Elasticsearch Database''' || UCHC Data Center || Indexes experiment metadata for search and access via the web portal and APIs.
|}


== '''Data Flow Summary''' ==
'''Data Flow Summary:'''
# An experiment ends on the spectrometer workstation.
# A user completes an acquisition on a spectrometer.
# The NDTS Daemon detects it, packages the data, and attempts to send it to the Gateway.
# The Daemon detects the completed experiment and sends it to the Gateway.
# The Gateway relays the data to the Receiver.
# The Gateway transmits the data to the Receiver at UCHC.
# Once received, the data becomes available in the NAN Repository within seconds.
# The Receiver accepts the data and hands it off to the Parser.
# The Parser extracts metadata and stores it in the PostgreSQL and Elasticsearch databases.
# The experiment data is stored in primary storage and backed up to disaster recovery storage.
# The data becomes visible in the NAN Portal (e.g., Data Browser, vNOC) within seconds.


If a connection fails at any stage, data is spooled and retried until successful.
Failures at any stage result in data being spooled locally and retried automatically.


== '''NDTS Responsibilities for Facility Managers''' ==
== '''Facility Manager Responsibilities''' ==
Facility Managers are expected to:
Facility Managers are expected to:
* Install and configure NDTS Gateway and Daemon software
* Install and configure Gateway and Daemon software
* Monitor daemon heartbeat and transmission status
* Monitor daemon heartbeat and transmission status
* Support users in selecting the correct metadata (e.g., NMRhub usernames)
* Support users in selecting correct metadata (e.g., NMRhub usernames, projects, samples)
* Reassign “unselected” or misattributed data
* Reassign “unselected” or misattributed data
* Use the Facility Dashboard to review transmission health and harvested data
* Use the Facility Dashboard to review health status and harvested experiments


== '''User Guide Sections''' ==
== '''User Guide Sections''' ==
Use the links below to access detailed instructions for each NDTS task or component:
Use the links below to access detailed instructions for each component or task:


* [[NDTS_Installation|Installing and Updating NDTS Components]]
* [[NDTS_Installation|Installing and Updating Daemon and Gateway]]
* [[NDTS_Daemon_Configuration|Configuring the NDTS Daemon]]
* [[NDTS_Daemon_Configuration|Configuring the Daemon]]
* [[NDTS_Daemon_Operation|Running and Monitoring the Daemon]]
* [[NDTS_Daemon_Operation|Running and Monitoring the Daemon]]
* [[NDTS_Workstation_UI|Using the NDTS Workstation UI]]
* [[NDTS_Workstation_UI|Using the Workstation UI]]
* [[NDTS_Experiment_Harvesting|Understanding Experiment Harvesting]]
* [[NDTS_Experiment_Harvesting|Understanding Experiment Harvesting]]
* [[NDTS_Gateway_Operation|Gateway Operation and Management]]
* [[NDTS_Gateway_Operation|Gateway Operation and Management]]
* [[NDTS_Data_Access|Accessing Collected Data]]
* [[NDTS_Data_Access|Accessing Collected Data]]
* [[NDTS_Logging|NDTS Logging and Audit Trail]]
* [[NDTS_Logging|Logging and Audit Trail]]
* [[NDTS_Appendix|Appendix and Troubleshooting]]
* [[NDTS_Appendix|Appendix and Troubleshooting]]


----
----


<span style="font-size:90%">Questions? Contact the NAN Repository administrator or submit a ticket from the Facility Dashboard.</span>
<span style="font-size:90%">For assistance, contact the NAN Repository administrator or submit a ticket from the Facility Dashboard.</span>
 
 
[[Category:NDTS]]

Revision as of 17:32, 2 June 2025

NDTS Facility Manager Guide

The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data.

System Architecture and Data Flow

The NDTS system consists of local and central components working together to collect, transfer, store, and index NMR experiment data.

Component Location Role
Daemon Spectrometer Workstation Detects completed experiments, associates metadata, and sends data to the Gateway.
Gateway Facility Server Receives data from all Daemons and relays it to the Receiver.
Receiver UCHC Data Center Accepts experiment data and metadata from Gateways.
Parser UCHC Data Center Extracts parameters, user info, and structural metadata from each dataset.
PostgreSQL Database UCHC Data Center Stores structured metadata and associations (users, instruments, facilities, samples, etc.).
Primary Storage UCHC Data Center Stores canonical copies of all collected experimental data.
Disaster Recovery Storage Off-site Maintains redundant backups of all experiment data.
Elasticsearch Database UCHC Data Center Indexes experiment metadata for search and access via the web portal and APIs.

Data Flow Summary:

  1. A user completes an acquisition on a spectrometer.
  2. The Daemon detects the completed experiment and sends it to the Gateway.
  3. The Gateway transmits the data to the Receiver at UCHC.
  4. The Receiver accepts the data and hands it off to the Parser.
  5. The Parser extracts metadata and stores it in the PostgreSQL and Elasticsearch databases.
  6. The experiment data is stored in primary storage and backed up to disaster recovery storage.
  7. The data becomes visible in the NAN Portal (e.g., Data Browser, vNOC) within seconds.

Failures at any stage result in data being spooled locally and retried automatically.

Facility Manager Responsibilities

Facility Managers are expected to:

  • Install and configure Gateway and Daemon software
  • Monitor daemon heartbeat and transmission status
  • Support users in selecting correct metadata (e.g., NMRhub usernames, projects, samples)
  • Reassign “unselected” or misattributed data
  • Use the Facility Dashboard to review health status and harvested experiments

User Guide Sections

Use the links below to access detailed instructions for each component or task:

For assistance, contact the NAN Repository administrator or submit a ticket from the Facility Dashboard.

Retrieved from "https://wiki.usnan.org/index.php?title=NAN_Data_Transport_System&oldid=543"