Mmaciejewski at 17:05, 10 June 2025

2025-06-10T17:05:29Z

← Older revision		Revision as of 17:05, 10 June 2025
Line 1:		Line 1:
	~~<span style="display:inline-block; margin-bottom:1em;">[[~~Datasets~~\|← Back to Dataset Browser]]</span>~~		{{Datasets}}

	= Arbitrary Data Upload =		= Arbitrary Data Upload =

Mmaciejewski: Created page with "← Back to Dataset Browser = Arbitrary Data Upload = While NDTS provides automated data harvesting from NAN-connected instruments, there are cases where data must be uploaded manually. The '''Arbitrary Data Upload Tool''' supports this need, offering a structured workflow for uploading datasets that were not harvested in real time. This tool is useful when: * Automatic NDTS harvesting was unin..."

2025-05-28T18:54:03Z

Created page with "<span style="display:inline-block; margin-bottom:1em;">← Back to Dataset Browser</span> = Arbitrary Data Upload = While NDTS provides automated data harvesting from NAN-connected instruments, there are cases where data must be uploaded manually. The '''Arbitrary Data Upload Tool''' supports this need, offering a structured workflow for uploading datasets that were not harvested in real time. This tool is useful when: * Automatic NDTS harvesting was unin..."

New page

<span style="display:inline-block; margin-bottom:1em;">[[Datasets|← Back to Dataset Browser]]</span>

= Arbitrary Data Upload =

While NDTS provides automated data harvesting from NAN-connected instruments, there are cases where data must be uploaded manually. The '''Arbitrary Data Upload Tool''' supports this need, offering a structured workflow for uploading datasets that were not harvested in real time.

This tool is useful when:
* Automatic NDTS harvesting was unintentionally disabled
* Data was collected prior to instrument connection to NAN
* Legacy datasets are stored on spectrometer workstations or lab computers

== Limitations of NDTS Manual Harvesting ==

The NDTS GUI includes a manual upload feature for harvesting individual experiments. However, this feature is not designed for bulk uploads or complex directory structures. The Arbitrary Data Upload Tool addresses this limitation by enabling users to submit archives containing multiple datasets.

== Upload Workflow ==

The Arbitrary Data Upload process consists of the following steps:

# Users create a `.tar` or `.zip` archive containing the data directory. The archive may include nested subdirectories.
# The archive file is uploaded to the NAN portal using a user's NAN account.
# A background service on the portal scans the uploaded file and identifies valid experimental datasets.
# Identified datasets are displayed in a '''staging table''', which resembles the standard Data Browser. Users may fill in required metadata fields and edit existing values.
# Once datasets are prepared, users select which ones to finalize and upload to the NAN archive.
# Selected datasets are parsed and fully integrated into the archive.

== Features and Safeguards ==

* The staging area is cached, allowing users to pause and return later. Sessions persist across logins.
* A full history of all arbitrary uploads is maintained per user.
* The system detects duplicates based on unique identifiers and alerts users to prevent redundant entries.
* Datasets that do not meet minimum completeness standards—such as missing critical files or metadata—are rejected.
* Only datasets from instruments registered in the NAN portal are accepted.
* Successfully uploaded datasets are marked with a '''Transfer Mode''' of `arbitrary`.

This workflow ensures consistency, quality control, and traceability of data manually added to the NAN archive.

Arbitrary Dataset Upload - Revision history

Mmaciejewski at 17:05, 10 June 2025