NDTS Gateway Installation

From Network for Advanced NMR
Revision as of 18:32, 3 June 2025 by Mmaciejewski (talk | contribs)
Jump to navigationJump to search

Gateway Installation

Prerequisites

  • A Gateway UUID supplied by the NAN Repository administrator
  • A Linux host (Ubuntu 20.04 or later recommended) that should be on the same network as the spectrometer workstations
  • TCP port 60195 open inbound from those workstations

Installation Steps

# Copy the .gz file to the gateway computer
tar xf data-transport-gateway-<version>.tar.gz
cd dtgateway-installer
sudo ./install.sh
# → You will be prompted only for the Gateway UUID

Firewall

Open TCP port 60195 on the Gateway to all spectrometer workstations; no other inbound ports are required.

Verifying the Service

Note that the installer automatically runs systemctl start data-transport-gateway after installation, but it is always good to check when installing the software that the service started.

sudo systemctl status data-transport-gateway      # should show “active (running)”
sudo systemctl start  data-transport-gateway      # start if not running

Updating the Gateway

When installing the Gateway software the installation will auto-detect if another version exists. The installation will proceed as above, but without questions as it will find the previous configuration file and use those values.

# Run the same installer script from a newer package
cd dtgateway-installer
sudo ./install.sh             # auto-detects an existing install; no prompts

Note that the installer automatically runs systemctl restart data-transport-gateway after upgrading, but it is always good to check when installing the software that the service started. See Verifying the Service for details

Uninstalling the Gateway

Download the latest version of the Gateway software, unpack, and run the uninstall.sh script

sudo ./uninstall.sh           # script is included inside every package

Rolling Back the Gateway

Obtain the desired older package and run its install.sh

Rolling back more than one version is not recommended.

NDTS Gateway Configuration

The following are optional configuration values that can be specified in the /opt/nandt- gateway/gateway_configuration.json file. The file must adhere to proper JSON formatting:

Configuration Options

Option Description Default
log_file_path Path to write the gateway log file /opt/nandt-gateway/nan-gateway.log
logging_level Minimum severity written to the log (fatal → debug) info
logging_format Python logging format string for each line %(asctime)s %(levelname)s %(module)s [%(process)d]: %(message)s
base_output_path Directory used for incoming experiment data /opt/nandt-gateway/data/received
https_endpoint HTTPS endpoint of the NDTS Receiver https://receiver.usnan.org
retry_limit Attempts to forward each dataset to the Receiver 5
retry_delay Seconds to wait between retry attempts 300
rsync_username SSH/rsync account on the Receiver nandt
rsync_node Node to which experiment files are rsync’d receiver.usnan.org
request_queue_size Outstanding daemon connections accepted before new ones are rejected. Increase this value if the gateway is rejecting requests from spectrometers evident in the workstation daemon log file showing network communication errors 50

Example JSON configuration file

{
  "log_file_path": "/opt/nandt-gateway/nan-gateway.log",
  "logging_level": "info",
  "logging_format": "%(asctime)s %(levelname)s %(module)s [%(process)d]: %(message)s",
  "base_output_path": "/opt/nandt-gateway/data/received",
  "https_endpoint": "https://receiver.usnan.org",
  "retry_limit": 5,
  "retry_delay": 300,
  "rsync_username": "nandt",
  "rsync_node": "receiver.usnan.org",
  "request_queue_size": 50
}

Logging Levels

Nothing below the set log level will be logged.

Level Description verbosity default
fatal Critical events only very low
error Problems that must be addressed for normal operation low
warning Unusual events that generally require no action medium
info Unusual or noteworthy events (recommended for production) high yes
debug Detailed diagnostic messages useful for troubleshooting, but should not be left on for normal operation extremely high

NDTS Gateway Components

The NDTS Gateway consists of several key components installed under /opt/nandt-gateway, unless otherwise noted. Each plays a specific role in data transfer, configuration, or monitoring.

File or Directory Description
main.py Entry point for the gateway's Python-based service
/etc/systemd/system/data-transport-gateway.service systemd unit file used to control the gateway service
nan-gateway.log Runtime log file (location can be changed via log_file_path in config)
gateway_configuration.json Gateway configuration file containing operational parameters
data/received/ Temporary storage location for incoming dataset files until data is successfully forwarded to the receiver
data/retransmitter.pid Contains the PID of the process that is attempting to re-send data that previously failed to be sent to the receiver. This is used internally by the gateway software to manage retransmitting data
data/users.dat Cached list of user/sample/project/study data pulled from the receiver
data/user_checksum.dat Checksum file used to determine when the users.dat file should be refreshed

All other files int the nandt-gateway directory are for the operation of the gateway, and are not of general interest

NDTS Gateway Operation

  • Experiment transfer – Workstation daemons connect on TCP 60195, announce a dataset, and stream it to received/. Successful transfers spawn a Transmitter process that forwards the data to the receiver; on success the local copy is deleted.
  • Retransmit – At start-up and after each daemon heartbeat, a Retransmitter scans received/. If a dataset’s original retry window has expired, it launches a new transmitter until delivery succeeds or the file is manually removed.
  • Heartbeat forwarding – Every daemon heartbeat is wrapped with the gateway UUID, local and UTC timestamps, then sent to the receiver by a short-lived Heartbeat Transmitter
  • User list propagation – Every 15s the gateway requests an updated list from the receiver; if the checksum differs it saves the new list to users.dat and serves it to workstations on demand.

NDTS Gateway Logging

Logging level are controlled by the logging_level parameter in the JSON configuration file (see above)

Example log excerpt: 

2022-06-06 09:22:23,130 INFO main [61393]: Starting gateway retransmitter instance...
2022-06-06 09:22:23,130 INFO main [61393]: Experiment file path = /opt/nandt-gateway/data/received
2022-06-06 09:22:23,132 INFO main [61393]: Starting gateway receiver network endpoint...
2022-06-06 09:22:23,133 INFO GatewayReceiver [61393]: Starting file receiver...
2022-06-06 09:22:23,133 INFO GatewayRetransmitter [61405]: In GatewayRetransmit handle() method...
2022-06-06 09:22:23,134 INFO GatewayRetransmitter [61405]: No experiments to retransmit -- exiting...
2022-06-06 09:22:23,134 INFO GatewayRetransmitter [61405]: GatewayRetransmit handle() method completed...
2022-06-06 09:23:12,829 INFO GatewayTransmitter [61512]: Heartbeat sent to https://receiver.usnan.org/heartbeat

From the log file note:

  • The first three lines show the spawning of the retransmitter.
  • The line that starts with “INFO GatewayReceiver” shows the file receiver starting, which means that it is awaiting connects from NDTS Daemons.
  • The lines that contain “INFO GatewayRetransmitter” show that there are no experiments to retransmit, so the retransmitter exits.
  • The last line shows that a heartbeat came in from an NDTS Daemon, and the Gateway is forwarding it to the Receiver.
Retrieved from "https://wiki.usnan.org/index.php?title=NDTS_Gateway_Installation&oldid=630"