NDTS Gateway Installation: Difference between revisions
From Network for Advanced NMR
Jump to navigationJump to search
Mmaciejewski (talk | contribs) |
Mmaciejewski (talk | contribs) No edit summary |
||
| (One intermediate revision by the same user not shown) | |||
| Line 1: | Line 1: | ||
== | {{NDTS_Navbox}} | ||
== Gateway Installation == | |||
=== Prerequisites === | === Prerequisites === | ||
| Line 47: | Line 49: | ||
===== [[NDTS Gateway Update Example]] ===== | ===== [[NDTS Gateway Update Example]] ===== | ||
== | == NDTS Gateway Configuration == | ||
The following are optional configuration values that can be specified in the <code>/opt/nandt- gateway/gateway_configuration.json</code> file. The file must adhere to proper JSON formatting: | The following are optional configuration values that can be specified in the <code>/opt/nandt- gateway/gateway_configuration.json</code> file. The file must adhere to proper JSON formatting: | ||
| Line 123: | Line 125: | ||
|} | |} | ||
== | == NDTS Gateway Components == | ||
The NDTS Gateway consists of several key components installed under <code>/opt/nandt-gateway</code>, unless otherwise noted. Each plays a specific role in data transfer, configuration, or monitoring. | The NDTS Gateway consists of several key components installed under <code>/opt/nandt-gateway</code>, unless otherwise noted. Each plays a specific role in data transfer, configuration, or monitoring. | ||
| Line 149: | Line 151: | ||
All other files int the <code>nandt-gateway</code> directory are for the operation of the gateway, and are not of general interest | All other files int the <code>nandt-gateway</code> directory are for the operation of the gateway, and are not of general interest | ||
== | == Normal Operation of the Gateway == | ||
=== Experiment Data Flow === | === Experiment Data Flow === | ||
| Line 179: | Line 181: | ||
* When an NDTS daemon requests the user list, the gateway compares checksums; if they differ, it sends the current <code>users.dat</code> contents to the daemon | * When an NDTS daemon requests the user list, the gateway compares checksums; if they differ, it sends the current <code>users.dat</code> contents to the daemon | ||
== | == NDTS Gateway Logging == | ||
Logging level are controlled by the logging_level parameter in the JSON configuration file (see [[NDTS Gateway Installation#Configuration Options|above]]) | Logging level are controlled by the logging_level parameter in the JSON configuration file (see [[NDTS Gateway Installation#Configuration Options|above]]) | ||
Latest revision as of 19:56, 9 June 2025
Navigation of NDTS Documentation
- NAN Data Transport System
- NDTS Overview
- NDTS Installation
- TopSpin 3.x Requirement
- Managing, Monitoring, and Logging
- NDTS Usage Guides
Gateway Installation
Prerequisites
- A Gateway UUID supplied by the NAN Repository administrator
- A Linux host (Ubuntu 20.04 or later recommended) that should be on the same network as the spectrometer workstations
- TCP port 60195 open inbound from those workstations
Installation Steps
# Copy the .gz file to the gateway computer tar xf data-transport-gateway-<version>.tar.gz cd dtgateway-installer sudo ./install.sh # → You will be prompted only for the Gateway UUID
Firewall
Open TCP port 60195 on the Gateway to all spectrometer workstations; no other inbound ports are required.
Verifying the Service
Note that the installer automatically runs systemctl start data-transport-gateway after installation, but it is always good to check when installing the software that the service started.
sudo systemctl status data-transport-gateway # should show “active (running)” sudo systemctl start data-transport-gateway # start if not running
Updating the Gateway
When installing the Gateway software the installation will auto-detect if another version exists. The installation will proceed as above, but without questions as it will find the previous configuration file and use those values.
# Run the same installer script from a newer package cd dtgateway-installer sudo ./install.sh # auto-detects an existing install; no prompts
Note that the installer automatically runs systemctl restart data-transport-gateway after upgrading, but it is always good to check when installing the software that the service started. See Verifying the Service for details
Uninstalling the Gateway
Download the latest version of the Gateway software, unpack, and run the uninstall.sh script
sudo ./uninstall.sh # script is included inside every package
Rolling Back the Gateway
Obtain the desired older package and run its install.sh
Rolling back more than one version is not recommended.
Installation and Update Examples
NDTS Gateway Install Example
NDTS Gateway Update Example
NDTS Gateway Configuration
The following are optional configuration values that can be specified in the /opt/nandt- gateway/gateway_configuration.json file. The file must adhere to proper JSON formatting:
Configuration Options
| Option | Description | Default |
|---|---|---|
| log_file_path | Path to write the gateway log file | /opt/nandt-gateway/nan-gateway.log |
| logging_level | Minimum severity written to the log (fatal → debug) | info |
| logging_format | Python logging format string for each line |
%(asctime)s %(levelname)s %(module)s [%(process)d]: %(message)s |
| base_output_path | Directory used for incoming experiment data | /opt/nandt-gateway/data/received |
| https_endpoint | HTTPS endpoint of the NDTS Receiver | https://receiver.usnan.org |
| retry_limit | Attempts to forward each dataset to the Receiver | 5 |
| retry_delay | Seconds to wait between retry attempts | 300 |
| rsync_username | SSH/rsync account on the Receiver | nandt |
| rsync_node | Node to which experiment files are rsync’d | receiver.usnan.org |
| request_queue_size | Outstanding daemon connections accepted before new ones are rejected. Increase this value if the gateway is rejecting requests from spectrometers evident in the workstation daemon log file showing network communication errors | 50 |
Example JSON configuration file
{
"log_file_path": "/opt/nandt-gateway/nan-gateway.log",
"logging_level": "info",
"logging_format": "%(asctime)s %(levelname)s %(module)s [%(process)d]: %(message)s",
"base_output_path": "/opt/nandt-gateway/data/received",
"https_endpoint": "https://receiver.usnan.org",
"retry_limit": 5,
"retry_delay": 300,
"rsync_username": "nandt",
"rsync_node": "receiver.usnan.org",
"request_queue_size": 50
}
Logging Levels
Nothing below the set log level will be logged.
| Level | Description | verbosity | default |
|---|---|---|---|
| fatal | Critical events only | very low | |
| error | Problems that must be addressed for normal operation | low | |
| warning | Unusual events that generally require no action | medium | |
| info | Unusual or noteworthy events (recommended for production) | high | yes |
| debug | Detailed diagnostic messages useful for troubleshooting, but should not be left on for normal operation | extremely high |
NDTS Gateway Components
The NDTS Gateway consists of several key components installed under /opt/nandt-gateway, unless otherwise noted. Each plays a specific role in data transfer, configuration, or monitoring.
| File or Directory | Description |
|---|---|
main.py |
Entry point for the gateway's Python-based service |
/etc/systemd/system/data-transport-gateway.service |
systemd unit file used to control the gateway service |
nan-gateway.log |
Runtime log file (location can be changed via log_file_path in config)
|
gateway_configuration.json |
Gateway configuration file containing operational parameters |
data/received/ |
Temporary storage location for incoming dataset files until data is successfully forwarded to the receiver |
data/retransmitter.pid |
Contains the PID of the process that is attempting to re-send data that previously failed to be sent to the receiver. This is used internally by the gateway software to manage retransmitting data |
data/users.dat |
Cached list of user/sample/project/study data pulled from the receiver |
data/user_checksum.dat |
Checksum file used to determine when the users.dat file should be refreshed |
All other files int the nandt-gateway directory are for the operation of the gateway, and are not of general interest
Normal Operation of the Gateway
Experiment Data Flow
- The gateway listens on its server port for connections from an NDTS daemon
- When a daemon indicates it has experiment data, the gateway opens a dedicated socket and saves the incoming files to the base output path (default
/opt/nandt-gateway/data/received) - After a complete, error-free transfer, the daemon marks the experiment as sent and will not resend it
- The gateway immediately spawns a Transmitter process—one per experiment—to forward the files to the NDTS Receiver
- A Transmitter retries up to
retry_limittimes, waitingretry_delayseconds between attempts - On success, the experiment directory is deleted from the base output path; on exhaustion of retries the Transmitter exits and the data become eligible for Retransmit
Retransmit
- At start-up and after each heartbeat (see below), the gateway launches a Retransmitter
- The Retransmitter scans
…/data/receivedfor unsent experiments - For any experiment whose original retry window has elapsed, it spawns a new Transmitter and repeats the transmit workflow until success or manual deletion
- Before creating a Retransmitter, the gateway checks for
{base_output_path}/../retransmitter.pid; if that PID is active, no new Retransmitter is started
Heartbeat Forwarding
- The gateway does not send independent heartbeats
- Upon receiving a daemon heartbeat, it spawns a Heartbeat Transmitter that forwards the message to the NDTS Receiver, appending:
* Local datetime * UTC datetime * Gateway UUID
- If transmission fails, the heartbeat is dropped and the Heartbeat Transmitter exits
User-List Refresh
- Every 15 seconds the gateway spawns a Refresh Users process to request an updated user list (users, samples, projects, studies, etc.) from the Receiver
- The request includes the checksum stored in
{base_output_path}/../user_checksum.dat - If the checksum is outdated, the Receiver returns a new list, which the gateway saves to
{base_output_path}/../users.datand updates the checksum file - When an NDTS daemon requests the user list, the gateway compares checksums; if they differ, it sends the current
users.datcontents to the daemon
NDTS Gateway Logging
Logging level are controlled by the logging_level parameter in the JSON configuration file (see above)
Example log excerpt:
2022-06-06 09:22:23,130 INFO main [61393]: Starting gateway retransmitter instance... 2022-06-06 09:22:23,130 INFO main [61393]: Experiment file path = /opt/nandt-gateway/data/received 2022-06-06 09:22:23,132 INFO main [61393]: Starting gateway receiver network endpoint... 2022-06-06 09:22:23,133 INFO GatewayReceiver [61393]: Starting file receiver... 2022-06-06 09:22:23,133 INFO GatewayRetransmitter [61405]: In GatewayRetransmit handle() method... 2022-06-06 09:22:23,134 INFO GatewayRetransmitter [61405]: No experiments to retransmit -- exiting... 2022-06-06 09:22:23,134 INFO GatewayRetransmitter [61405]: GatewayRetransmit handle() method completed... 2022-06-06 09:23:12,829 INFO GatewayTransmitter [61512]: Heartbeat sent to https://receiver.usnan.org/heartbeat
From the log file note:
- The first three lines show the spawning of the retransmitter.
- The line that starts with “INFO GatewayReceiver” shows the file receiver starting, which means that it is awaiting connects from NDTS Daemons.
- The lines that contain “INFO GatewayRetransmitter” show that there are no experiments to retransmit, so the retransmitter exits.
- The last line shows that a heartbeat came in from an NDTS Daemon, and the Gateway is forwarding it to the Receiver.
