View source for Datasets

<span style="display:inline-block; margin-bottom:1em;">[[Data Browser|← Back to Data Browser]]</span>

= Data Browser: Datasets =

The Dataset Browser allows users to explore and manage datasets they are authorized to access. Access permissions are granted either because the dataset is ''public'' or through ''lab-based'', ''user-based'', or ''collaborative'' permissions authorized by a Principal Investigator (PI).

The Dataset Browser includes:
* A '''[[Datasets#Navigation Pane|Navigation Pane]]''' on the left side for switching between dataset views and hierarchical organization of My & Lab data described below.
* A [[Datasets#Dataset Table|'''Dataset''' '''Table''']] for displaying rows of datasets with columns representing metadata that may be sorted and filtered.
* '''[[Datasets#Customization Tools|Customization Tools]]''' in the upper-right corner to configure columns, saved views, and filters.
* Advanced '''[[Datasets#Selection, Sorting, and Filtering|Selection, Sorting, and Filtering]]''' of datasets
* Ability to perform [[Datasets#Actions|Actions]] quickly and easily from a context menu to one or more datasets
* An '''[[Datasets#Upload Datasets|Upload Datasets]]''' button to submit datasets that were not harvested by NDTS.

== Navigation Pane ==
[[File:Navigation pane.png|thumb|Navigation pane]]
The Navigation Pane allows users to quickly access datasets across different categories. Unauthenticated users will only see ''All Public Datasets'' and ''Knowledgebase Datasets''.

=== All Datasets ===
: Displays all datasets the user can access, including public and permission-granted datasets.

=== All Public Datasets ===
: Displays only datasets marked as public.

=== Knowledgebase Datasets ===
: Displays a curated subset of public datasets that are highly annotated and intended to aid users in experimental planning and analysis.

=== My & Lab Data ===
: Displays datasets accessible via user- or lab-based permissions (excluding datasets that are visible only due to being public).
: This section includes a hierarchical organization mirroring a file system:
:* '''My Collections''' – personal collections created by the user.
:* '''Projects''' – high-level groupings for data organization
:** '''Studies''' – reside inside Projects to allow datasets from a given study to be grouped
:*** '''Collections''' – reside inside Studies to allow fine-grained dataset groupings\

== Dataset Table ==
The Dataset Table displays all datasets for which the user has at least ''read access''. 

=== Table Rows ===
Each row highlights a dataset in the NAN archive

=== Table Columns ===
Columns represent different metadata fields for the NAN dataset. There is a default list of columns that are displayed, but users can toggle different columns on and off as desired by selecting the wrench icon in the upper right hand corner of the dataset browser. Columns may be re-order by dragging them. The columns to be displayed, along with their order, is saved in the NAN database as a user preference and will persist across sessions, browsers, and computers. See [[Dataset Columns]] for a complete list of columns with a short description and the types of filters that may be applied.

=== Displaying Redundant and Preferred Datasets ===
* By default, the Data Browser displays only '''Preferred''' datasets to provide a clean and responsive interface.
* Users can identify datasets with redundancies via an icon on the preferred dataset. Clicking the icon opens the full set for review (see Icon badges below)
* A '''Redundant''' column may be enabled in the browser view. When this column is active, the Data Browser shows both preferred and redundant datasets for complete visibility.
==== Display Name / Dataset Name ====
When a dataset is harvested by the NAN Data Transport System it is stored in the NAN database with a unique UUID (hidden from the user) and is given a Dataset Name (non-editable) that matches the experimental directory from the NMR spectrometer.
* VNMRJ: "exp''N"''
* Bruker: "experiment/''N"''
As the Dataset Name is generally not a useful description of the experiment we also create a Display Name that is user editable to allow users to create a more descriptive and meaningful label. When downloading, the dataset is saved using the original '''Dataset Name''' and the the '''Display Name''' is saved in a CSV file within the dataset folder. Note that the Display Name is fixed as the first column of the dataset table and cannot be altered. The Data Name column is not displayed by default, but can be toggled on if desired.

==== Icon badges ====
Icon badges in select columns represent additional information about the dataset and provide navigation links as described here:

* '''Display Name icon badges'''
** A circle with a star represents that the dataset is marked as a "preferred" dataset of a redundant set. In parenthesis will be the total number of datasets in the redundant set. Clicking the star will show all the datasets in the redundant set and provide a breadcrumb to navigate back to the default view.
** A small clock icon indicates that the dataset has been published. Clicking the icon will allow you to navigate to previous published versions or the original dataset.
* '''Sample icons'''
** A link icon indicates that a sample has been linked to the dataset and clicking the link icon will bring you to the Sample Browser filtered on the linked sample.

=== Pagination ===
At the bottom of the table is a '''pagination control'''. Users can move between pages and adjust the number of rows displayed per page: '''25''', '''50''', '''100''', or '''500''' datasets per page

== Customization Tools ==
[[File:Customization setting-tools.png|thumb|Tools to customize and utilize views and filters|400x400px]]

=== Wrench Icon ===

* Brings up a pull-down menu to toggle which columns are shown in the dataset table.
* Allows a View to be Created, overwritten, or deleted. Note that the columns that are displayed are saved as a user preference and are not tied to a View. A View defines the applied filters and sorts to the columns and is independent on which columns are visible. For example, maybe you had a project where all the datasets were collected between two dates so you define a View to filter only datasets from specific users involved in the project that lie between two dates so that you can quickly see those datasets without the need to reapply the filters.

=== Saved Views ===

* Pull-down list of saved views (defined filters and sort)

=== Quick Filters ===
Quick filters apply predefined views to narrow down datasets. Current Quick Filters include:

* '''Successful Datasets Only''' – shows datasets marked as successful.
* '''Hide Failed Datasets''' – hides datasets marked as failed.
* '''My Data''' – datasets owned by the logged-in user.
* '''KB Datasets''' – datasets published in the Knowledgebase.

Note that successful and failed dataset filters rely on proper classification of datasets

=== Remove Filters Icon ===

* When no filters are applied to any columns the icon appears faded and is not selectable
* When not faded and selectable will clear all applied filters and sorts
* When the icon contains an exclamation point it means filters or sorts for a non visible column are active. Pressing the icon will prompt if all filters and sorts should be removed or only those for the non-visible columns.

=== Selection Icon ===

* Shown as a circle with a line through it. The Icon become visible when one or more datasets are selected and pressing it will clear all selections. Can be very handy when datasets are selected, but not visible on the screen.
== Selection, Sorting, and Filtering ==
[[File:Filter on NAN User.png|thumb|Text filter with two rules]]

=== Selection ===

Datasets can be selected by clicking on the ''Display Name''. The Dataset Browser supports multi-selection, with a checkbox next to each Display Name indicating selection status.

To select multiple datasets:
* Hold the ''Ctrl'' key (or ''Cmd'' on Mac) and click on Display Names to toggle individual selections.
* Hold the ''Shift'' key to select a range from the last selected to the current dataset.

'''''IMPORTANT NOTE''''': There is inconsistent behavior when using ''Shift'' and ''Ctrl'' keys with the checkboxes. It is strongly recommended to use the Display Name for selection. Treat checkboxes only as visual indicators.

By default, datasets are sorted by date, with the most recent shown first. Sorting and filtering are available for all columns.

=== Sorting ===

Each column header includes a sort button (up/down arrows). Click once to sort in ascending order; click again to sort in descending order.

=== Filtering ===

Each column header includes a filter icon that opens a filtering dialog. The available filter types depend on the column's data type. The table below summarizes available filters. See [[Dataset Columns]] for which filter type apply to each column.

{| class="wikitable"
|+ Filter Types
! Date !! Boolean !! Text !! Number !! Controlled List<br />''Classification'' !! Controlled List<br />''Transfer Mode'' !! Tags
|-
| equals || yes || equals || equals || || || includes
|-
| before || no || does not equal || does not equal || || || does not include
|-
| after || is unset || contains || greater than || || || 
|-
|  || is set || does not contain || less than || || || 
|-
|  ||  || similar to ||  || || || 
|-
|  ||  || starts with ||  || || || 
|-
|  ||  || ends with ||  || || || 
|-
|  ||  || is unset ||  || || || 
|}

For all filter types except Boolean, users can add multiple filter rules per column. If multiple rules are added, the user must specify whether to "Match All" (AND) or "Match Any" (OR). This setting is ignored if only one rule is applied.

Advanced filters can also span multiple columns. While building complex filters may take effort, users can save views for reuse. See [[Datasets#Customization Tools|Customization Tools]] for details.

== Actions ==
[[File:Dataset Context Menu.png|thumb|364x364px|Context menu for dataset actions]]

The <nowiki>'''</nowiki>Actions<nowiki>'''</nowiki> menu is accessed by right-clicking on a dataset row in the Dataset Browser. For multiple selections, right-click on any of the selected rows to perform bulk actions. Available actions depend on user permissions—actions unavailable to the user will appear grayed out.

Below is a table of available actions with some providing links to a page with additional details.

{| class="wikitable"
|+
! Action
! Bulk Action Capable
! Description
|-
| [[Dataset Editing|View / Edit Dataset]]
| Possibly
| Opens a modal window to view or edit the selected dataset.
|-
| [[Dataset reassignment|Reassign]]
| Yes
| Assigns or reassigns a dataset to a ''NAN user''. Facility managers can reassign datasets to any user without time restrictions. Standard users can reassign within their lab group for up to three months after harvesting.
|-
| [[Download Datasets|Download]]
| Yes
| Downloads datasets in a variety of organizational layouts.
|-
| [[NMRbox Integration]]
| Yes
| Copies a dataset from the NAN archive to the user’s NMRbox home folder in a predefined location. Also enables retrieval of post-acquisition files from NMRbox back into the NAN archive.
|-
| [[Supplemental Data]]
| No
| Adds or views supplemental data associated with a dataset.
|-
| [[Redundant Datasets|Redundancy]]
| Yes
| Sets the dataset’s ''redundancy status'' as “preferred” or “redundant.” By default, the most recent experiment in a redundant set is marked as preferred.
|-
| [[Public and Publishing|Make Public]]
| Yes
| Marks the dataset as publicly available.
|-
| [[Link Sample]]
| Yes
| Links a dataset to a sample.
|-
| [[Dataset Classification|Classification]]
| Yes
| Allows uses to classify datasets from a controlled list. Allows NMR facility managers to target a dataset to be removed from the NAN archive.
|-
| Tags
| Yes
| Allows users to assign arbitrary tags to datasets.
|-
| Notes
| Yes
| Allows users to add notes to datasets.
|-
| Unlink from Collection
| Yes
| Removes a dataset from a collection.
|-
| [[Public and Publishing|Publish]]
| Yes
| Publishes a dataset.
|-
|Copy Dataset Link
|No
|Copies the URL of a dataset to the Clipboard
|}

== Upload Datasets ==

See the [[Arbitrary Dataset Upload]] page for details

While NDTS provides automated data harvesting from NAN-connected instruments, there are cases where data must be uploaded manually. The '''Arbitrary Data Upload Tool''' supports this need, offering a structured workflow for uploading datasets that were not harvested in real time.

This tool is useful when:
* Automatic NDTS harvesting was unintentionally disabled
* Data was collected prior to instrument connection to NAN
* Legacy datasets are stored on spectrometer workstations or lab computers

=== Limitations of NDTS Manual Harvesting ===
The NDTS GUI includes a manual upload feature for harvesting individual experiments. However, this feature is not designed for bulk uploads or complex directory structures. The Arbitrary Data Upload Tool addresses this limitation by enabling users to submit archives containing multiple datasets.

=== Upload Workflow ===
The Arbitrary Data Upload process consists of the following steps:

# Users create a `.tar` or `.zip` archive containing the data directory. The archive may include nested subdirectories.
# The archive file is uploaded to the NAN portal using a user's NAN account.
# A background service on the portal scans the uploaded file and identifies valid experimental datasets.
# Identified datasets are displayed in a '''staging table''', which resembles the standard Data Browser. Users may fill in required metadata fields and edit existing values.
# Once datasets are prepared, users select which ones to finalize and upload to the NAN archive.
# Selected datasets are parsed and fully integrated into the archive.

=== Features and Safeguards ===
* The staging area is cached, allowing users to pause and return later. Sessions persist across logins.
* A full history of all arbitrary uploads is maintained per user.
* The system detects duplicates based on unique identifiers and alerts users to prevent redundant entries.
* Datasets that do not meet minimum completeness standards—such as missing critical files or metadata—are rejected.
* Only datasets from instruments registered in the NAN portal are accepted.
* Successfully uploaded datasets are marked with a '''Transfer Mode''' of `arbitrary`.

This workflow ensures consistency, quality control, and traceability of data manually added to the NAN archive.


[[Category:Data Browser]]