NAN Data Transport System: Difference between revisions
From Network for Advanced NMR
Jump to navigationJump to search
Mmaciejewski (talk | contribs) No edit summary |
Mmaciejewski (talk | contribs) |
||
| Line 1: | Line 1: | ||
= | == '''Overview''' == | ||
The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data. | The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data. | ||
== '''System Architecture and Data Flow''' == | == '''System Architecture and Data Flow''' == | ||
The NDTS system consists of local and central components working together to collect, transfer, store, and index NMR | The NDTS system consists of local and central components working together to collect, transfer, store, and index NMR datasets | ||
{| class="wikitable" | {| class="wikitable" | ||
| Line 10: | Line 9: | ||
|- | |- | ||
| '''Daemon''' || Spectrometer Workstation || Detects completed experiments, associates metadata, and sends data to the Gateway. | | '''Daemon''' || Spectrometer Workstation || Detects completed experiments, associates metadata, and sends data to the Gateway. | ||
Sends heartbeat information to the Gateway | |||
Pull user information from the Gateway | |||
|- | |- | ||
| '''Gateway''' || | | '''Gateway''' || Within NMR facility network || Receives data from all Daemons and relays it to the Receiver. Pulls user information from the NAN Receiver | ||
|- | |- | ||
| '''Receiver''' || UCHC Data Center || Accepts experiment data and metadata from Gateways | | '''Receiver''' || UCHC Data Center || Accepts experiment data and metadata from Gateways | ||
Packages user information destined for the spectrometer workstations | |||
|- | |- | ||
| '''Parser''' || UCHC Data Center || | | '''Parser''' || UCHC Data Center || Parses datasets to extract additional metadata, writes database entries, and stores dataset files to primary and disaster recovery storage appliances | ||
|- | |- | ||
| '''PostgreSQL Database''' || UCHC Data Center || Stores structured metadata | | '''PostgreSQL Database''' || UCHC Data Center || Stores datasets and their structured metadata | ||
|- | |- | ||
| '''Primary Storage''' || UCHC Data Center || Stores canonical copies of all collected experimental data | | '''Primary Storage''' || UCHC Data Center || Stores canonical copies of all collected experimental data | ||
|- | |- | ||
| '''Disaster Recovery Storage''' || | | '''Disaster Recovery Storage''' || Geo-dispersed || Maintains redundant backups of all experimental data on a WORM S3 bucket | ||
|- | |- | ||
| '''Elasticsearch Database''' || UCHC Data Center || Indexes | | '''Elasticsearch Database''' || UCHC Data Center || Indexes statistics about harvested datasets and heartbeat information for visualization from the virtual NAN Operation Center (vNOC) | ||
|} | |} | ||
'''Data Flow Summary:''' | === '''Data Flow Summary:''' === | ||
# A user completes an acquisition on a spectrometer. | # A user completes an acquisition on a spectrometer. | ||
# The Daemon detects the completed experiment and sends it to the Gateway. | # The Daemon detects the completed experiment and sends it to the Gateway. | ||
Revision as of 17:42, 2 June 2025
Overview
The Network for Advanced NMR Data Transport System (NDTS) enables automated harvesting of NMR acquisition data from spectrometer workstations and delivers it securely to the NAN Repository. Facility Managers are responsible for installing and managing the local components of the system, ensuring connectivity, and supporting user access to collected data.
System Architecture and Data Flow
The NDTS system consists of local and central components working together to collect, transfer, store, and index NMR datasets
| Component | Location | Role |
|---|---|---|
| Daemon | Spectrometer Workstation | Detects completed experiments, associates metadata, and sends data to the Gateway.
Sends heartbeat information to the Gateway Pull user information from the Gateway |
| Gateway | Within NMR facility network | Receives data from all Daemons and relays it to the Receiver. Pulls user information from the NAN Receiver |
| Receiver | UCHC Data Center | Accepts experiment data and metadata from Gateways
Packages user information destined for the spectrometer workstations |
| Parser | UCHC Data Center | Parses datasets to extract additional metadata, writes database entries, and stores dataset files to primary and disaster recovery storage appliances |
| PostgreSQL Database | UCHC Data Center | Stores datasets and their structured metadata |
| Primary Storage | UCHC Data Center | Stores canonical copies of all collected experimental data |
| Disaster Recovery Storage | Geo-dispersed | Maintains redundant backups of all experimental data on a WORM S3 bucket |
| Elasticsearch Database | UCHC Data Center | Indexes statistics about harvested datasets and heartbeat information for visualization from the virtual NAN Operation Center (vNOC) |
Data Flow Summary:
- A user completes an acquisition on a spectrometer.
- The Daemon detects the completed experiment and sends it to the Gateway.
- The Gateway transmits the data to the Receiver at UCHC.
- The Receiver accepts the data and hands it off to the Parser.
- The Parser extracts metadata and stores it in the PostgreSQL and Elasticsearch databases.
- The experiment data is stored in primary storage and backed up to disaster recovery storage.
- The data becomes visible in the NAN Portal (e.g., Data Browser, vNOC) within seconds.
Failures at any stage result in data being spooled locally and retried automatically.
Facility Manager Responsibilities
Facility Managers are expected to:
- Install and configure Gateway and Daemon software
- Monitor daemon heartbeat and transmission status
- Support users in selecting correct metadata (e.g., NMRhub usernames, projects, samples)
- Reassign “unselected” or misattributed data
- Use the Facility Dashboard to review health status and harvested experiments
User Guide Sections
Use the links below to access detailed instructions for each component or task:
- Installing and Updating Daemon and Gateway
- Configuring the Daemon
- Running and Monitoring the Daemon
- Using the Workstation UI
- Understanding Experiment Harvesting
- Gateway Operation and Management
- Accessing Collected Data
- Logging and Audit Trail
- Appendix and Troubleshooting
For assistance, contact the NAN Repository administrator or submit a ticket from the Facility Dashboard.
