Tools

In this central location, you can access and launch all tools maintained by the NIMH Data Archive. In all cases, a recent version of Java is required in order to use NDA tools. OpenJDK is not supported.

 

GUID Tool

Researchers throughout and beyond the field of mental health research use the NDA GUID Tool, originally developed for the use of the autism research community and the National Database for Autism Research (NDAR), to generate participant identifiers. The NDA GUID Tool takes personal information provided by research participants and uses it to securely create a unique identifier. Using this tool, participant data can be linked across studies and laboratories while always maintaining the participant's privacy. In order to use this tool, you will need a user account with the appropriate permission assigned.

Launch the NDA GUID Tool


Validation and Upload Tool

Contributors harmonizing and uploading their data to the NIMH Data Archive must use the Validation and Upload Tool to send their data and complete this process. This tool connects to the Data Dictionary and then allows you to load data templates and validate them against their definitions. This helps ensure that data in NDA is harmonized to a standard and serves as a "pre-upload" QA check on your data. After data is successfully harmonized, the same tool is used to package and upload the data to your NDA Collection. In addition to working with CSV data templates, the tool also supports direct uploads from a hosted AWS-RDS database.

Launch the HTML Validation and Upload Tool

 

Download Manager (Beta)

Download Manager User Guide

Last modified on Apr 30, 2021

The new Download Manager (Beta) tool is an electron client that was developed with the main philosophy of making the NDA Download Manager (Beta) extremely easy to use. It was completely recoded and even relies on a different set of backend tools (which have been documented on our GitHub page).

 

Download Manager (Beta) Installers

File Name

Distribution

Operating System

NDADownloadManager Setup 0.1.38.exe

Setup

Windows

NDADownloadManager-0.1.38.dmg

Setup

Mac OS X*

NDADownloadManager-0.1.38.AppImage

Standalone

Linux

NDADownloadManager 0.1.38.exe

Standalone

Windows

NDADownloadManager-0.1.38-mac.zip

Standalone

Mac OS X*

* On first install, you may need to right-click and open, rather than double clicking. If you are experiencing issues with the .zip file, you may need to use the .dmg.  Please contact NDA Help with any issues.

 

Using the Download Manager (Beta)

The first time the tool is launched, after sign in, the Download Manager (Beta) defaults your download location to the default provided by your operating system, you can go to the 'Settings' tab in the top navbar of the client and select 'Choose Download Directory' to change the download location. 

After a successful sign-in, the 'My Data Packages' tab will load with a list of all data packages you have made. From this page, you may select a data package to download. If you want to explore the data package first or select specific files/folders to download,  you can select the data package and then individually download the files or directories in that data package. Multiple files/folders can also be downloaded by selecting the checkmark next to all of the folders/files you wish to download, once all the desired files or directories are selected you can download them by selecting the top most download button.

Files and folders downloaded from a data package can be found in the chosen download location, in a directory named 'Package_<package-id>'. Individual files/folders can also be opened from Download Manager (Beta) by clicking the 'Show' button. If any errors are encountered during the download the Download Manager (Beta) tool will log them to a log file as well as notify you in the download progress console. If errors are encountered, you can press the 'Request Help' button in the footer to open a Help Desk ticket, which will attach the log file.

Depending on your internet connection and performance of storage device (internal hard drive, network-attached storage, etc.) downloading an entire data package may require several days to complete. Users can typically expect data packages in excess of 1 Terabyte to take several days to download.

Should you encounter an error you'd like to report you can use the 'Request Help' button located in the bottom left of the client, this will automatically email the helpdesk as well as attach your log file to the report. Should you no longer have access to the client but still want to report an error you can also email NDAHelp@nih.gov, be sure to include a copy of the log file from the error, if you are on Windows you can find this log file at '%APPDATA%/NDADownloadManager/', or on *nix systems at '~/.config/nda-download-manager/logs/'. After you get to your respective logs directory select the appropriate log, the latest log is named 'log.log', while previous log files are labeled by date.

 

Legacy Download Manager

Access to the Legacy Download Manager is being deprecated. Users are encouraged to read the announcement here: https://mailchi.mp/mail/new-nda-download-manager-tool-now-available

Users approved for access to NDA shared data can use the Legacy Download Manager to view and download data packages they have added to their account or created using the query tools. Access to download data to non-AWS internet addresses is limited to 20 Terabytes over 30 days. For more detail, including examples, please read about our user download threshold.

Note: Data packages containing omics data must be accessed through the cloud and cannot be downloaded directly using this tool. In addition to the Legacy Download Manager tool described below, a legacy python command line downloader is also available.

The Legacy Download Manager tool is provided as a Java Web Start application, which is launched using a Java Network Launch Protocol (JNLP) file downloaded from NDA. Opening the JNLP file requires the Java Runtime Environment (JRE) for Java 8, which will download resources from NDA necessary to launch the application. Note only Java 8 is compatible with NDA Legacy Download Manager.

Prerequisites

To launch (i.e., run) the tool, the following prerequisites must be satisfied:

  • Installation of Java Runtime Environment (JRE) for Java 8
  • Copy of JNLP file, or URL for the JNLP file

Verify you have the 1.8 version of the JRE installed by opening a command prompt (Windows) or terminal window (Mac and Linux), then enter the following command: java -version and press the ‘Enter’ key. You should expect to see output displaying either the installed version of Java Runtime Environment (JRE) or an error message indicating that the command is not recognized. The JRE version should begin with 1.8.0_ followed by the specific update version (i.e., java version "1.8.0_261").

To obtain a copy of the JNLP file, click Download NDA Legacy Download Manager JNLP. The browser should begin downloading a file named DownloadManager.jnlp.

 

Installing Java SE Runtime Environment (JRE) for Java 8

If you were not able to successfully verify the installation of JRE version 1.8, please download the installation file for your operating system and follow the instructions provided for installation. Note that other JRE versions are not compatible with this tool.

Windows: https://www.java.com/en/download/help/windows_manual_download.xml

Mac: https://www.java.com/en/download/help/mac_install.xml

Linux: https://www.java.com/en/download/help/linux_x64_install.xml

Further instructions for installing Linux JRE for Java 8 and switching between Java versions are provided here.

 

Launching the Legacy Download Manager

After successfully installing JRE 1.8, you should be able to double-click on the DownloadManager.jnlp file that was downloaded (see prerequisites). The installation should register the Java Web Launcher executable (javaws) when attempting to open or run a jnlp file. You may need to add an exception for the https://nda.nih.gov URL in the JRE configuration to allow the application to run.

If the system does not recognize which application to use for the jnlp file:

  1. Open a command prompt (Windows) or terminal (Mac and Linux),
  2. Change to the directory where you downloaded the DownloadManager.jnlp file, typically the Downloads folder for your user, and
  3. Enter the command: javaws DownloadManager.jnlp.

When the Java Web Launcher opens the DownloadManager.jnlp file, several files will be downloaded and a prompt will be displayed the text: Do you want to run this application?

The prompt will show the following information:

Name: Download Manager
Publisher: National Institutes of Health
Locations: https://nda.nih.gov Launched from downloaded JNLP file.

This application will run with unrestricted access which may put your computer and personal information at risk. Run this application only if you trust the locations and publisher above.

 

Click on the Run button to open the Legacy Download Manager, and you will be prompted to enter your NDA username and password. After entering a valid username and password, the tool will open.

Note: There may be some delay after entering your credentials, especially if your user has a large number of packages or packages that include a large number of files.

A new screen will open with a warning describing the terms of use for the Legacy Download Manager. After clicking the Accept button the tool will open a screen with a table listing your packages.

Using the Legacy Download Manager

Select the location for where packages should be saved by clicking the "Browse" button, and then select a directory with enough space for downloading your package. The size is provided in the table that displays each of your packages. The default location will be your user’s home directory.

Select the check box next to the package you wish to download and click the Start Downloads button. The package status must be 'Ready to Download' before you can begin downloading files. If the package status is Creating Package, press the Refresh Queue button to update the status. Depending on the size (number of subject records and files) of the package this can take up to 30 minutes after initiating the data package request in the NDA Web Application.

Legacy Download Manager will create a directory with the package name, in the location you selected. If any errors are encountered during the download, an errors.txt file will be created in this directory with detailed error messages, and the progress bar will not reach 100% completion.

Depending on your internet connection and performance of the storage device (internal hard drive, network-attached storage, etc.) downloading an entire package may require several days to complete. Users can typically expect packages in excess of 1 Terabyte to take several days to download.

If you encounter any errors or have additional questions, please open a Help Desk ticket by sending an email to NDAHelp@nih.gov. Please include the package id in your request, along with a brief description of the issue you are experiencing, and a copy of any errors.txt file if present.

 

Cloud Access Overview

The mission of the National Institute of Mental Health Data Archive (NDA) is to make research data available for reuse. Data collected across projects can be aggregated and made available using the GUID, including clinical data, and the results of imaging, genomics, and other experimental data collected from the same participants. In this way, separate experiments on genotypes and brain volumes can inform the research community on the over one hundred thousand subjects now contained in the NDA. The NDA’s cloud computation capability provides a framework in support of this infrastructure.

 

How does it work?

The NDA holds and protects rich datasets (fastq, brain imaging) in object-based storage (Amazon S3). To facilitate access, the NDA supports the deployment of data packages (created through the NDA Query tools) to an Amazon Web Service Oracle database. These databases contain a table for each data structure in a package. Associated data files are available via read-only access to NDA’s S3 objects. Addresses for those objects in the associated package are provided in the miNDAR table titled S3_LINKS. By providing this interface, the NDA envisions real-time computation against rich datasets that can be initiated without the need to download full data packages. Furthermore, a new category of data structure has been created called "evaluated data." Tables for these structures will be created for each miNDAR, allowing researchers using NDA cloud capabilities and computational pipelines to write any analyzed data directly back to the miNDAR database. This will enable the NDA to make this data available to the general research community when appropriate.

 

miNDARs can also be populated with your own data and uploaded directly back into the NDA for a streamlined data submission directly from a hosted database.

 

How do I get started?

The option to launch data packages to a cloud-hosted database will be available during package creation. You can deploy previously generated data packages as well as new ones.

 

Initiating miNDAR

To move data to Oracle, first create a data package in the NDA. Then, following registration, enter the package id and credentials requested on the miNDAR tab. This will start the miNDAR creation process, which takes approximately 10 minutes. Once created, the miNDAR connect details will be emailed to you, and can be used to establish a connection with your credentials.

 

Accessing Files

Access to download data to non-AWS internet addresses is limited, please read about our user download threshold. Access from AWS internet addresses is unlimited.

Files included in a data package are accessible from Amazon Web Services (AWS) S3 Object Storage. Each miNDAR package will have a table “S3_LINKS” table containing URIs for all objects in that package. Using direct web service calls to Amazon Web Service's S3 API, a third-party tool, or client libraries, data from these objects can be streamed or downloaded. 

Authorization to access S3 Objects requires authentication with AWS using temporary AWS credentials or presigned URL. Both forms of authorization are time-limited, and require individual users to authenticate with the package service, and request either credentials or presigned URL for one or more files within a specific package.

Users may access the web service using the swagger user interface, or by writing their own tool.

Examples are provided on the NDA GitHub Page.

 

Additional Help

Please see our Cloud Tutorials for a video demonstration of how to create a miNDAR.  Please contact the Help Desk with any questions.