Datasets - Revision history

Mmaciejewski: /* Upload Datasets */

2025-07-03T15:18:22Z

Upload Datasets

← Older revision		Revision as of 15:18, 3 July 2025
Line 218:		Line 218:

	See the [[Arbitrary Dataset Upload]] page for details		See the [[Arbitrary Dataset Upload]] page for details

	[[Category:Data Browser]]		[[Category:Data Browser]]
			[[Category:Datasets]]

Apozhidaeva at 18:48, 23 June 2025

2025-06-23T18:48:28Z

← Older revision		Revision as of 18:48, 23 June 2025
Line 72:		Line 72:

	* Brings up a pull-down menu to toggle which columns are shown in the dataset table.		* Brings up a pull-down menu to toggle which columns are shown in the dataset table.
	* Allows a View to be ~~Created~~, overwritten, or deleted. Note that the columns that are displayed are saved as a user preference and are not tied to a View. A View defines the applied filters and sorts to the columns and is independent on which columns are visible. For example, maybe you had a project where all the datasets were collected between two dates so you define a View to filter only datasets from specific users involved in the project that lie between two dates so that you can quickly see those datasets without the need to reapply the filters.		* Allows a View to be created, overwritten, or deleted. Note that the columns that are displayed are saved as a user preference and are not tied to a View. A View defines the applied filters and sorts to the columns and is independent on which columns are visible. For example, maybe you had a project where all the datasets were collected between two dates so you define a View to filter only datasets from specific users involved in the project that lie between two dates so that you can quickly see those datasets without the need to reapply the filters.

	=== Saved Views ===		=== Saved Views ===

Mmaciejewski: /* Knowledgebase Datasets */

2025-06-20T12:35:26Z

Knowledgebase Datasets

← Older revision		Revision as of 12:35, 20 June 2025
Line 29:		Line 29:
	=== My & Lab Data ===		=== My & Lab Data ===
	: Displays datasets accessible via user- or lab-based permissions (excluding datasets that are visible only due to being public).		: Displays datasets accessible via user- or lab-based permissions (excluding datasets that are visible only due to being public).
	: This section includes a hierarchical organization mirroring a file system:		: This section includes a hierarchical organization mirroring a file system including user defined Collections and Lab controlled [[Project, Studies, and Collections\|Projects --> Studies --> Collections]]
	:* '''My Collections''' – personal collections created by the user.		:* '''My Collections''' – personal collections created by the user.
	:* '''Projects''' – high-level groupings for data organization		:* '''Projects''' – high-level groupings for data organization

Mmaciejewski at 12:31, 20 June 2025

2025-06-20T12:31:32Z

← Older revision		Revision as of 12:31, 20 June 2025
Line 11:		Line 11:
	* '''[[Datasets#Customization Tools\|Customization Tools]]''' in the upper-right corner to configure columns, saved views, and filters.		* '''[[Datasets#Customization Tools\|Customization Tools]]''' in the upper-right corner to configure columns, saved views, and filters.
	* Advanced '''[[Datasets#Selection, Sorting, and Filtering\|Selection, Sorting, and Filtering]]''' of datasets		* Advanced '''[[Datasets#Selection, Sorting, and Filtering\|Selection, Sorting, and Filtering]]''' of datasets
	* Ability to perform [[Datasets#Actions\|Actions]] quickly and easily from a context menu to one or more datasets		* Ability to perform [[Datasets#Actions\|'''Actions''']] quickly and easily from a context menu to one or more datasets
	* An '''[[Datasets#Upload Datasets\|Upload Datasets]]''' button to submit datasets that were not harvested by NDTS.		* An '''[[Datasets#Upload Datasets\|Upload Datasets]]''' button to submit datasets that were not harvested by NDTS.

	== Navigation Pane ==		== Navigation Pane ==
	[[File:Navigation pane.png\|thumb\|Navigation pane]]		[[File:Navigation pane.png\|thumb\|Navigation pane]]
	The Navigation Pane allows users to quickly access datasets across different categories. Unauthenticated users will only see ''All Public Datasets'' and ''Knowledgebase Datasets''.		The Navigation Pane allows users to quickly access datasets across different categories. Unauthenticated users will only see '''''All Public Datasets''''' and '''''Knowledgebase Datasets'''''.

	=== All Datasets ===		=== All Datasets ===

Mmaciejewski at 20:17, 10 June 2025

2025-06-10T20:17:58Z

← Older revision		Revision as of 20:17, 10 June 2025
Line 4:		Line 4:
	= Data Browser: Datasets =		= Data Browser: Datasets =

	The Dataset Browser allows users to explore and manage datasets they are authorized to access. Access permissions are granted either because the dataset is ''public'' or through ''lab-based'', ''user-based'', or ''collaborative'' permissions authorized by a Principal Investigator (PI).		The Dataset Browser allows users to explore and manage datasets they are authorized to access. Access [[Lab Permissions\|permissions]] are granted either because the dataset is ''public'' or through ''lab-based'', ''user-based'', or ''collaborative'' permissions authorized by a Principal Investigator (PI).

	The Dataset Browser includes:		The Dataset Browser includes:

Mmaciejewski at 17:01, 10 June 2025

2025-06-10T17:01:36Z

← Older revision		Revision as of 17:01, 10 June 2025
Line 1:		Line 1:
	{{DataBrowser}}		{{DataBrowser}}
			{{Datasets}}

	= Data Browser: Datasets =		= Data Browser: Datasets =

Mmaciejewski at 16:49, 10 June 2025

2025-06-10T16:49:01Z

← Older revision		Revision as of 16:49, 10 June 2025
Line 1:		Line 1:
	~~<span style="display:inline-block; margin-bottom:1em;">[[Data Browser\|← Back to Data Browser]]</span>~~		{{DataBrowser}}

	= Data Browser: Datasets =		= Data Browser: Datasets =

Mmaciejewski: /* Table Columns */

2025-05-28T18:57:44Z

Table Columns

← Older revision		Revision as of 18:57, 28 May 2025
Line 43:		Line 43:
	Columns represent different metadata fields for the NAN dataset. There is a default list of columns that are displayed, but users can toggle different columns on and off as desired by selecting the wrench icon in the upper right hand corner of the dataset browser. Columns may be re-order by dragging them. The columns to be displayed, along with their order, is saved in the NAN database as a user preference and will persist across sessions, browsers, and computers. See [[Dataset Columns]] for a complete list of columns with a short description and the types of filters that may be applied.		Columns represent different metadata fields for the NAN dataset. There is a default list of columns that are displayed, but users can toggle different columns on and off as desired by selecting the wrench icon in the upper right hand corner of the dataset browser. Columns may be re-order by dragging them. The columns to be displayed, along with their order, is saved in the NAN database as a user preference and will persist across sessions, browsers, and computers. See [[Dataset Columns]] for a complete list of columns with a short description and the types of filters that may be applied.

	=== ~~Displaying~~ Redundant ~~and Preferred Datasets~~ ===		==== Redundant Status Column ====
	* By default, the Data Browser displays only '''Preferred''' datasets to provide a clean and responsive interface.		* By default, the Data Browser displays only '''Preferred''' datasets to provide a clean and responsive interface.
	* Users can identify datasets with redundancies via an icon on the preferred dataset. Clicking the icon opens the full set for review (see Icon badges below)		* Users can identify datasets with redundancies via an icon on the preferred dataset. Clicking the icon opens the full set for review (see Icon badges below)

Mmaciejewski: /* My & Lab Data */

2025-05-28T18:56:34Z

My & Lab Data

← Older revision		Revision as of 18:56, 28 May 2025
Line 32:		Line 32:
	:* '''Projects''' – high-level groupings for data organization		:* '''Projects''' – high-level groupings for data organization
	:** '''Studies''' – reside inside Projects to allow datasets from a given study to be grouped		:** '''Studies''' – reside inside Projects to allow datasets from a given study to be grouped
	:*** '''Collections''' – reside inside Studies to allow fine-grained dataset groupings\		:*** '''Collections''' – reside inside Studies to allow fine-grained dataset groupings

	== Dataset Table ==		== Dataset Table ==

Mmaciejewski: /* Upload Datasets */

2025-05-28T18:54:47Z

Upload Datasets

← Older revision		Revision as of 18:54, 28 May 2025
Line 217:		Line 217:

	See the [[Arbitrary Dataset Upload]] page for details		See the [[Arbitrary Dataset Upload]] page for details

	While NDTS provides automated data harvesting from NAN-connected instruments, there are cases where data must be uploaded manually. The '''Arbitrary Data Upload Tool''' supports this need, offering a structured workflow for uploading datasets that were not harvested in real time.

	~~This tool is useful when:~~
	* Automatic NDTS harvesting was unintentionally disabled
	* Data was collected prior to instrument connection to NAN
	* Legacy datasets are stored on spectrometer workstations or lab computers

	~~=== Limitations of NDTS Manual Harvesting ===~~
	The NDTS GUI includes a manual upload feature for harvesting individual experiments. However, this feature is not designed for bulk uploads or complex directory structures. The Arbitrary Data Upload Tool addresses this limitation by enabling users to submit archives containing multiple datasets.

	~~=== Upload Workflow ===~~
	~~The Arbitrary Data Upload process consists of the following steps:~~

	~~# Users create a `.tar` or `.zip` archive containing the data directory. The archive may include nested subdirectories.~~
	~~# The archive file is uploaded to the NAN portal using a user's NAN account.~~
	~~# A background service on the portal scans the uploaded file and identifies valid experimental datasets.~~
	~~# Identified datasets are displayed in a '''staging table''', which resembles the standard Data Browser. Users may fill in required metadata fields and edit existing values.~~
	~~# Once datasets are prepared, users select which ones to finalize and upload to the NAN archive.~~
	~~# Selected datasets are parsed and fully integrated into the archive.~~

	~~=== Features and Safeguards ===~~
	* The staging area is cached, allowing users to pause and return later. Sessions persist across logins.
	* A full history of all arbitrary uploads is maintained per user.
	* The system detects duplicates based on unique identifiers and alerts users to prevent redundant entries.
	* Datasets that do not meet minimum completeness standards—such as missing critical files or metadata—are rejected.
	* Only datasets from instruments registered in the NAN portal are accepted.
	* Successfully uploaded datasets are marked with a '''Transfer Mode''' of `arbitrary`.

	~~This workflow ensures consistency, quality control, and traceability of data manually added to the NAN archive.~~


	[[Category:Data Browser]]		[[Category:Data Browser]]