NIMH Data Archive Tools
In this central location you can access and launch all tools maintained by the NIMH Data Archive. In all cases, a recent version of Java is required in order to use NDA tools. OpenJDK is not supported.
NDA GUID Tool
Last modified on Apr 29, 2021
The GUID, or Global Unique Identifier, is an alphanumeric code that is created by the NIMH Data Archive (NDA) GUID Tool and used as an identifier for a research participant. The GUID provides a secure mechanism to link research participants within and across research project datasets in NDA. The NDA GUID Tool and underlying algorithms were released in 2010 as the result of a collaboration between NIH, the Simons Foundation, and the informatics researcher community. Review the publication Using Global Unique Identifiers to Link Autism Collections (Johnson et al. 2010) for more information. Note that the GUID matching sensitivity has been reduced from the original design described in this paper.
The GUID itself is not personally identifiable information or protected health information.
Originally implemented to support the autism research community, the GUID is now available for other research communities needing a common subject identifier across research laboratories and repositories. Contact us at the NDA Help Desk for any questions regarding the GUID, the NDA GUID Tool, or to request the use of GUIDs in your research.
For more information regarding GUID functionality, please visit the Using the NDA GUID page . The above user manual includes information on the NDA GUID Tool Electron client. For in-line help with commands within the NDA GUID Tool command line client, use the -h option.
NDA GUID Tool Electron Installers
|
File Name |
Distribution |
Operating System |
|---|---|---|
|
Setup |
Windows |
|
|
Setup |
Mac OS X* |
|
|
Standalone |
Linux |
|
|
Standalone |
Windows |
|
|
Standalone |
Mac OS X* |
* On first install, you may need to right-click and open, rather than double clicking. If you are experiencing issues with the .zip file, you may need to use the .dmg. Please contact NDA Help with any issues.
NDA GUID Tool Command Line Installers
|
File Name |
Operating System |
|---|---|
|
Windows |
|
|
Mac OS X |
|
|
Linux |
Validation and Upload Tool
Contributors harmonizing and uploading their data to the NIMH Data Archive must use the Validation and Upload Tool to send their data and complete this process. This tool connects to the Data Dictionary and then allows you to load data templates and validate them against their definitions. This helps ensure that data in NDA is harmonized to a standard and serves as a "pre-upload" QA check on your data. After data is successfully harmonized, the same tool is used to package and upload the data to your NDA Collection. In addition to working with CSV data templates, the tool also supports direct uploads from a hosted AWS-RDS database. There are currently three different versions of the tool. Please review them below to determine the most appropriate one for your situation. If in doubt, please use the first option: the HTML version.
HTML Validation and Upload Tool
This version of the tool allows you to use it a webpage, validating the quality of your data and upload it directly through your web browser. Chrome, Firefox, Safari, and recent versions of Internet Explorer are supported
Launch the HTML Validation and Upload Tool
Python Validation and Upload Tool
A Python package and command line clients have been developed by the NDA to allow users to programmatically validate, package, submit, and/or download data without having to open their browser. It is both simple and convenient to use, and even more straightforward to setup. The package and clients interact with the Validation , Submission Package , and Data Submission web services.
If you have already installed python and pip, then the client only needs one line of code to be installed and ready to use. These tools offer users the ability to manage their submissions and downloads much faster than the HTML Validation and Upload Tool.
To learn how to access the tools, go to the nda-tools GitHub and follow the setup instructions.
Uploading data in BIDS format
To support the BIDS format for imaging data, NDA has added a new Data Element of type Manifest to the existing set of String, Integer, Float, Thumbnail, File, and GUID. The Manifest data element is like the File data element in that the data submission template specifies the location of an XML or JSON file containing a collection of files. This supports the capability to create NDA data structures that describe a collection of related file resources. This new element treats the files included in the Manifest as associated files for purposes of submission and these are ingested and stored as individual objects in AWS S3 Object storage, which also enables users to directly access specific files from the collection of files.
How to submit BIDS data using a manifest:
Manifest files can be submitted with three NDA data structures: image03, fmriresults01, and eeg_sub_files01. Instead of indicating a file in the
image_file or derived_files columns, list the JSON file in the Manifest data element column. As of now, data submission
for structures that use this new type will need to use the nda-tools vtcmd tool, which is distributed with the nda-tools
python package
For more information about Manifest data elements, examples, and helper scripts, please refer to our GitHub repository.
Download Manager
Last modified September 2022
The new Download Manager tool is an electron client that was developed with the main philosophy of making the NDA Download Manager extremely easy to use. It was completely recoded and even relies on a different set of backend tools (which have been documented on our GitHub page).
Download Manager Installers
|
File Name |
Distribution |
Operating System |
|---|---|---|
|
Setup |
Windows |
|
|
Setup |
Mac OS X* |
|
|
Standalone |
Linux |
|
|
Standalone |
Windows |
|
|
Standalone |
Mac OS X* |
* On first install, you may need to right-click and open, rather than double clicking. If you are experiencing issues with the .zip file, you may need to use the .dmg. Please contact NDA Help with any issues.
Using the Download Manager
The first time the tool is launched, after sign in, the Download Manager defaults your download location to the default provided by your operating system, you can go to the 'Settings' tab in the top navbar of the client and select 'Choose Download Directory' to change the download location.
After a successful sign in, the 'My Data Packages' tab will load with a list of all data packages you have made. From this page you may select a data package to download. If you want explore the data package first, or select specific files/folders to download, you can select the data package and then individually download the files or directories in that data package. Multiple files/folders can also be downloaded by selecting the check mark next to all of the folders/files you wish to download, once all the desired files or directories are selected you can download them by selecting the top most download button.
Files and folders downloaded from a data package can be found in the chosen download location, in a directory named 'Package_<package-id>'. Individual files/folders can also be opened from Download Manager by clicking the 'Show' button. If any errors are encountered during the download process, the Download Manager tool will log them to a log file as well as notify you in the download progress console. If errors are encountered, you can press the 'Request Help' button in the footer to open a Help Desk ticket, which will attach the log file.
Depending on your internet connection and performance of storage device (internal hard drive, network attached storage, etc.) downloading an entire data package may require several days to complete. Users can typically expect data packages in excess of 1 Terabyte to take several days to download.
Should you encounter an error you'd like to report you can use the 'Request Help' button located in the bottom left of the client, this will automatically email the helpdesk as well as attach your log file to the report. Should you no longer have access to the client but still want to report an error you can also email NDAHelp@nih.gov, be sure to include a copy of the log file from the error, if you are on Windows you can find this log file at '%APPDATA%/NDADownloadManager/', or on *nix systems at '~/.config/nda-download-manager/logs/'. After you get to your respective logs directory select the appropriate log, the latest log is named 'log.log', while previous log files are labeled by date.
Cloud Access Overview
The mission of the National Institute of Mental Health Data Archive (NDA) is to make research data available for reuse. Data collected across projects can be aggregated and made available using the GUID, including clinical data, and the results of imaging, genomic, and other experimental data collected from the same participants. In this way, separate experiments on genotypes and brain volumes can inform the research community on the over one hundred thousand subjects now contained in the NDA. The NDA’s cloud computation capability provides a framework in support of this infrastructure.
How does it work?
NDA holds and protects rich datasets (fastq, brain imaging) in object-based storage (Amazon S3). To facilitate access, the NDA supports the deployment of data packages (created through the NDA Query tools) to an Amazon Web Service Oracle database. These databases contain a table for each data structure in a package. Associated data files are available via read-only access to NDA’s S3 objects. Addresses for those objects in the associated package are provided in the miNDAR table titled S3_LINKS. By providing this interface, the NDA envisions real-time computation against rich datasets that can be initiated without the need to download full data packages. Furthermore, a new category of data structure has been created called " evaluated data." Tables for these structures will be created for each miNDAR, allowing researchers using NDA cloud capabilities and computational pipelines to write any data directly back to the miNDAR database. This will enable the NDA to make this data available to the general research community when appropriate.
miNDARs can also be populated with your own data and uploaded directly back into the NDA for a streamlined data submission directly from a hosted database.
How do I get started?
The option to launch data packages to a cloud-hosted database will be available during package creation. You can deploy previously generated data packages as well as new ones.
Initiating miNDAR
To move data to Oracle, first create a package in the NDA. Then, following registration, enter the package id and credentials requested on the miNDAR tab. This will start the miNDAR creation process, which takes approximately 10 minutes. Once created, the miNDAR connect details will be emailed to you, and can be used to establish a connection with your credentials.
Accessing Files
Access to download data to non-AWS internet addresses is limited, please read about our user download threshold. Access from AWS internet addresses is unlimited.
Files included in a data package are accessible from Amazon Web Services (AWS) S3 Object Storage. Each miNDAR package will have a table “S3_LINKS” table containing URIs for all objects in that package. Using direct web service calls to Amazon Web Service's S3 API, a third-party tool, or client libraries, data from these objects can be streamed or downloaded.
Authorization to access S3 Objects requires authentication with AWS using temporary AWS credentials or presigned URL. Both forms of authorization are time-limited, and require individual users to authenticate with the package service, and request either credentials or presigned URL for one or more files within a specific package.
Users may access the web service using the swagger user interface, or by writing their own tool.
Examples are provided on the NDA GitHub Page.
Additional Help
Please see our Cloud Tutorials for a video demonstration of how to create a miNDAR. Please contact the Help Desk with any questions.
