NDTS Experiment Harvesting - Revision history

Mmaciejewski at 20:05, 9 June 2025

2025-06-09T20:05:56Z

← Older revision		Revision as of 20:05, 9 June 2025
Line 1:		Line 1:
			{{NDTS_Navbox}}

	== Overview ==		== Overview ==

Mmaciejewski: /* Limitations */

2025-06-09T19:27:05Z

Limitations

← Older revision		Revision as of 19:27, 9 June 2025
Line 59:		Line 59:

	== Limitations ==		== Limitations ==
	~~NDTS is off~~

	paropt		* If the NDTS daemon is not running no datasets will be harvested.
			* For Bruker paropt arrayed experiments the experimental directory is often overwritten faster than NDTS can harvest and thus datasets collected with paropt are often corrupted. In addition, if the paropt data is saved as a 2D parameter set (but one which did not run) the daemon will not know of its existence and not collect it. We recommend running paropt experiments in job numbers higher than the configured value for data harvesting so that these datasets may be ignored.
	missing ~~critical data~~		* A dataset that is missing key information, such as a fid or ser file or parameter sets will be ignored.

Mmaciejewski: /* What is harvested */

2025-06-09T19:19:27Z

What is harvested

@@ Line 33: / Line 33: @@
 == What is harvested ==
 == What if gateway is unreachable ==
 == Limitations ==

Mmaciejewski: /* How NDTS determines if a dataset should be harvested */

2025-06-09T19:01:21Z

How NDTS determines if a dataset should be harvested

← Older revision		Revision as of 19:01, 9 June 2025
Line 30:		Line 30:
	Note that the NDTS daemon must be running. The daemon will not detect datasets that were collected while the daemon was disabled. The daemon is configured to be running at all times, so hopefully this is a rare event. If the daemon were disabled, the manual harvesting button from the NDTS-GUI may be used to harvested the missed dataset(s).		Note that the NDTS daemon must be running. The daemon will not detect datasets that were collected while the daemon was disabled. The daemon is configured to be running at all times, so hopefully this is a rare event. If the daemon were disabled, the manual harvesting button from the NDTS-GUI may be used to harvested the missed dataset(s).

	NDTS writes a detailed audit log that lists all experiments run on the spectrometer and a status of the transfer (e.g. sent, spooled, skipped). ~~See the~~ [[NDTS Daemon Operation#(B) Experiment Transfer Audit File\|~~NDTS audit file~~]]		NDTS writes a detailed audit log that lists all experiments run on the spectrometer and a status of the transfer (e.g. sent, spooled, skipped). Click [[NDTS Daemon Operation#(B) Experiment Transfer Audit File\|here]] for details.

	== What is harvested ==		== What is harvested ==

Mmaciejewski: /* How NDTS determines if a dataset should be harvested */

2025-06-09T18:59:44Z

How NDTS determines if a dataset should be harvested

← Older revision		Revision as of 18:59, 9 June 2025
Line 27:		Line 27:
	* the experiment job number is below the configured threshold set by the facility manager and displayed in the NDTS-GUI (this allows experiments, such as pulse calibrations, to easily be skipped by users without the need to toggle the harvesting status)		* the experiment job number is below the configured threshold set by the facility manager and displayed in the NDTS-GUI (this allows experiments, such as pulse calibrations, to easily be skipped by users without the need to toggle the harvesting status)
	* the acquisition time exceeds 1 second (VNMRj often records a less than 1 second dataset at the beginning of a new dataset which contains garbage data. This requirement avoids those from being harvested).		* the acquisition time exceeds 1 second (VNMRj often records a less than 1 second dataset at the beginning of a new dataset which contains garbage data. This requirement avoids those from being harvested).
	* pulse program is not topshim, rga, wobb, or gs (this avoids harvesting datasets involved in tuning, shimming, receiver gain settings, and		* pulse program is not topshim, rga, wobb, or gs (this avoids harvesting datasets involved in tuning, shimming, receiver gain settings, and real-time parameter optimization)
			Note that the NDTS daemon must be running. The daemon will not detect datasets that were collected while the daemon was disabled. The daemon is configured to be running at all times, so hopefully this is a rare event. If the daemon were disabled, the manual harvesting button from the NDTS-GUI may be used to harvested the missed dataset(s).

	~~==== NDTS Audit File ====~~		NDTS writes a detailed audit log that lists all experiments run on the spectrometer and a status of the transfer (e.g. sent, spooled, skipped). See the [[NDTS Daemon Operation#(B) Experiment Transfer Audit File\|NDTS audit file]]
	~~The [[NDTS Daemon Operation#(B) Experiment Transfer Audit File\|~~NDTS audit ~~file]] will show detailed records of every dataset~~ that the ~~daemon detected~~ and ~~what what the~~ status of the transfer is (e.g. sent, spooled, skipped).

	== What is harvested ==		== What is harvested ==

Mmaciejewski: /* Datasets that are skipped automatically */

2025-06-09T18:53:17Z

Datasets that are skipped automatically

← Older revision		Revision as of 18:53, 9 June 2025
Line 19:		Line 19:
	Experimental datasets with the same experiment name (e.g. <code>exp''N''</code> for VnmrJ, and <code>data_directory/''N''</code> for TopSpin) and the same sessionID are all tagged as ''redundant'' except for the most recent experiment which is tagged ''preferred''. The ''preferred'' / ''redundanat'' status can later be altered by the user from the [[Datasets\|Dataset Browser]].		Experimental datasets with the same experiment name (e.g. <code>exp''N''</code> for VnmrJ, and <code>data_directory/''N''</code> for TopSpin) and the same sessionID are all tagged as ''redundant'' except for the most recent experiment which is tagged ''preferred''. The ''preferred'' / ''redundanat'' status can later be altered by the user from the [[Datasets\|Dataset Browser]].

	== ~~Datasets~~ that ~~are~~ skipped ~~automatically ==~~		== How NDTS determines if a dataset should be harvested ==
			The NDTS daemon follows a logical workflow to determine if an experiment that completes should be harvested. Data will be harvested if:

			* the user is configured for harvesting (a user, who the facility manager has set to never collect from the Manage Facility Users tool, will not have their data harvest regardless of other NDTS settings)
			* the NDTS-GUI harvest toggle is "on"
			* the dataset contains valid files (fid / ser), (propar / acqus*)
			* the experiment job number is below the configured threshold set by the facility manager and displayed in the NDTS-GUI (this allows experiments, such as pulse calibrations, to easily be skipped by users without the need to toggle the harvesting status)
			* the acquisition time exceeds 1 second (VNMRj often records a less than 1 second dataset at the beginning of a new dataset which contains garbage data. This requirement avoids those from being harvested).
			* pulse program is not topshim, rga, wobb, or gs (this avoids harvesting datasets involved in tuning, shimming, receiver gain settings, and

	==== NDTS Audit File ====		==== NDTS Audit File ====

Mmaciejewski: /* Overview */

2025-06-09T18:07:49Z

Overview

← Older revision		Revision as of 18:07, 9 June 2025
Line 11:		Line 11:

	== Sessions ==		== Sessions ==
			Each dataset harvested is associated with a sessionID. New sessions are created when:

			* a user logs in to the workstation
			* when a user starts the spectrometer software that controls the spectrometer (VNMRj or TopSpin)
			* when the NAN User is changed in the NDTS-GUI

			Experimental datasets with the same experiment name (e.g. <code>exp''N''</code> for VnmrJ, and <code>data_directory/''N''</code> for TopSpin) and the same sessionID are all tagged as ''redundant'' except for the most recent experiment which is tagged ''preferred''. The ''preferred'' / ''redundanat'' status can later be altered by the user from the [[Datasets\|Dataset Browser]].

	== Datasets that are skipped automatically ==		== Datasets that are skipped automatically ==

Mmaciejewski: /* Identifying acquisition start and end events */

2025-06-09T17:45:05Z

Identifying acquisition start and end events

← Older revision		Revision as of 17:45, 9 June 2025
Line 5:		Line 5:

	* '''VNMRj''' - the daemon listens to InfoProc network messages to detect transitions in status (e.g. idle to acquiring)		* '''VNMRj''' - the daemon listens to InfoProc network messages to detect transitions in status (e.g. idle to acquiring)
	* '''TopSpin 3.x''' - the daemon uses inotify (ReadDirectoryChanges on Windows) to monitor changes to the accounting file		* '''TopSpin 3.x''' - the daemon uses inotify (ReadDirectoryChanges on Windows) to monitor changes to the accounting file. See additional details on how NDTS utilizes the TopSpin accounting file [[Monitoring TopSpin Accounting File\|here]]:
	** accounting must be enabled on a per user basis		** accounting must be enabled on a per user basis. See [[Enabling TopSpin Accounting\|enabling TopSpin accouting]] for more details.
	** only detects experiment end events. The experiment start time is determined after the experiment is complete		** only detects experiment end events. The experiment start time is determined after the experiment is complete
	* '''TopSpin 4.x''' - the daemon monitors the shmem file in the user-specific TopSpin directory for start and end events		* '''TopSpin 4.x''' - the daemon monitors the <code>shmem</code> file in the user-specific TopSpin directory (<code>{topspin_directory}/prog/curdir/{user}/shmem</code>) for ''start'' and ''end'' events

	== Sessions ==		== Sessions ==

Mmaciejewski at 16:40, 9 June 2025

2025-06-09T16:40:05Z

← Older revision		Revision as of 16:40, 9 June 2025
Line 13:		Line 13:

	== Datasets that are skipped automatically ==		== Datasets that are skipped automatically ==
	(~~log~~ file)
			==== NDTS Audit File ====
			The [[NDTS Daemon Operation#(B) Experiment Transfer Audit File\|NDTS audit file]] will show detailed records of every dataset that the daemon detected and what what the status of the transfer is (e.g. sent, spooled, skipped).

	== What is harvested ==		== What is harvested ==

	== What if gateway is unreachable ==		== What if gateway is unreachable ==

			== Limitations ==
			NDTS is off

			paropt

			missing critical data

Mmaciejewski: Created page with "== Overview == == Identifying acquisition start and end events == The daemon detects when TopSpin or VNMRj is launched, who launched it, and whether it controls the spectrometer. How experiment start and end events are detected varies depending on the software version controlling the spectrometer: * '''VNMRj''' - the daemon listens to InfoProc network messages to detect transitions in status (e.g. idle to acquiring) * '''TopSpin 3.x''' - the daemon uses inotify (ReadDi..."

2025-06-09T15:44:57Z

Created page with "== Overview == == Identifying acquisition start and end events == The daemon detects when TopSpin or VNMRj is launched, who launched it, and whether it controls the spectrometer. How experiment start and end events are detected varies depending on the software version controlling the spectrometer: * '''VNMRj''' - the daemon listens to InfoProc network messages to detect transitions in status (e.g. idle to acquiring) * '''TopSpin 3.x''' - the daemon uses inotify (ReadDi..."

New page

== Overview ==

== Identifying acquisition start and end events ==
The daemon detects when TopSpin or VNMRj is launched, who launched it, and whether it controls the spectrometer. How experiment start and end events are detected varies depending on the software version controlling the spectrometer:

* '''VNMRj''' - the daemon listens to InfoProc network messages to detect transitions in status (e.g. idle to acquiring)
* '''TopSpin 3.x''' - the daemon uses inotify (ReadDirectoryChanges on Windows) to monitor changes to the accounting file
** accounting must be enabled on a per user basis
** only detects experiment end events. The experiment start time is determined after the experiment is complete
* '''TopSpin 4.x''' - the daemon monitors the shmem file in the user-specific TopSpin directory for start and end events

== Sessions ==

== Datasets that are skipped automatically ==
(log file)

== What is harvested ==

== What if gateway is unreachable ==