What can the Discovery Portal do for me? #

You can use the Discovery Portal to:

• Search across millions of datasets and other resources from numerous sources related to infectious and immune-mediated diseases
• Discover previously unknown resources to enrich biomedical research
• Download metadata or access via API to gather new insights about what’s available.
• Track research across NIH-funded programs or specific scientific areas.

Are there tutorials or demos for using the Discovery Portal? #

Yes. See the Knowledge Center for user guides, search tips, information for data providers, and more.

What data is available in the Discovery Portal? #

The Discovery Portal retrieves metadata related to allergic, infectious and immune-mediated disease. It aggregates metadata across numerous sources, including NIAID-supported repositories, general biomedical repositories, and other generalist sources. See our list of data sources.

What do the different resource types mean (datasets, tools, and resource catalogs)? #

• Datasets: Collections of data of a specific experimental type
• Resource Catalogs: Collections of scientific information or research outputs; these include databases, repositories, knowledge bases, citation indices, software indices, browser-based data portals, and more
• Computational Tools: Tools used to analyze or interpret scientific knowledge; these include software, workflows, computational tools, and code. Learn more about the types of resources in the NIAID Data Ecosystem here.

How do I suggest a data source that you don't have? #

If there are sources you’d like the Discovery Portal to include, you may suggest a source here.

Do I need to create an account or register to use the NIAID Data Ecosystem? #

No, the NIAID Data Ecosystem Discovery Portal is freely available for anyone to use. Some of the resources indexed by the Discovery Portal have access restrictions, however, and you may need to register with the source repository in order to access certain datasets. The Discovery Portal has a filter for ‘Conditions of Access’ so you can easily view resources that are open, closed, or restricted.

How do I download the data? #

Data is not stored in the Discovery Portal and cannot be downloaded directly from the Discovery Portal. The Discovery Portal can be used to search across repositories to find if/where data exists. All results are linked to the data source so you can access data from the data provider.

How do you handle controlled-access data? #

The Discovery Portal does not provide direct access to data. The Discovery Portal is a searchable interface that helps you find if/where data exists. When you find data you want to use, you must follow the link to the data provider's site to request access.

When I've found my dataset, can you help me get access to the data? #

The Discovery Portal can help you find datasets and link you to the data provider, but it cannot provide direct access to the data. Once you've found data you want to use, click the button "Access Data" and follow the data provider's guidelines to gain access from their site.

Can I preview the data before I access it? #

Currently, data cannot be previewed within the Discovery Portal. Some of the data providers offer data previews before accessing/downloading the dataset, but this will be offered within their site. Click the button "Access Data" to see if the data can be previewed.

Can I upload my own data? #

Data cannot be uploaded to the Discovery Portal. The Discovery Portal is not a repository and does not store data. There are ways to make your metadata available in the Discovery Portal, however - read more here.

How do I use the Advanced Search tool? #

The Advanced Search tool allows you to:

• Search across specific metadata fields, like disease, species, and funding source
• Build and edit custom queries using a simple user-friendly interface
• Preview result counts for your query

Learn all about the Advanced Search tool here.

Can I edit the string query I created using Advanced Search? #

Users can view their string query in the Advanced Search tool. Click the expandable button "view raw query" at the bottom of the Advanced Search page.

Can I write my own advanced string queries? #

Users can write their own fielded queries in the Discovery Portal. Learn more about writing your own fielded queries (including available fields you can search).

How do operators and syntax work in the Discovery Portal? #

Learn all about writing your own fielded queries, including the nitty-gritty of operators and syntax here.

Why can't I download data directly from the Discovery Portal? #

The Discovery Portal is not a repository. The Discovery Portal can be used to search across repositories to find if/where data exists. It helps support the FAIRness of data. However, users cannot analyze data in the Discovery Portal, nor download data directly from the Discovery Portal.

Why are some metadata fields empty? #

The Discovery Portal attempts to standardize metadata that is available; however, it cannot create information that does not exist. If metadata is missing at the source, it will also be absent within the Discovery Portal.

Why am I seeing some results that aren't related to immune-mediated or infectious disease? #

As the Discovery Portal aggregates data from some generalist repositories, search results may include datasets that are not related to allergic or infectious and immune-mediated disease.

Why am I seeing some duplicate results? #

The Discovery Portal searches across some sources that are also data aggregators. This means you may see two records for the same dataset from two different sources.

Why don't you include all data from all infectious disease repositories? #

The Discovery Portal does not aggregate every dataset from every source related to allergic or infectious and immune-mediated disease (IID). The Discovery Portal pulls from a list of data sources. If there are data sources you’d like included, you may suggest a source.

How does the Discovery Portal retrieve results? #

NIAID Data Ecosystem Discovery Crawlers harvest metadata from a variety of repositories and other sources. API infrastructure has also been created to access the metadata created by the Discovery Crawlers based on BioThings SDK. The Discovery Portal uses custom translators written in Python to transform metadata harvested by the Discovery Crawlers into a common schema, derived from schema.org. When you perform a basic search, the Discovery Portal looks for your terms anywhere within the metadata record. You can narrow your search to specific fields through advanced searching or you can use the filters.

How do you standardize the metadata? #

The Discovery Portal standardizes metadata to a common schema, so different data providers describe datasets in the same way. View the Data Ecosystem schemas on the Data Discovery Engine. This is based on schema.org. Read more about our Schemas. On the Sources page, you can read about how the metadata provided by the repositories are translated into the Data Ecosystem schema.

How are my search terms matched to results? #

When you perform a basic search, the Discovery Portal looks for your terms anywhere within the metadata record. You can narrow your search to specific fields through advanced searching or keyed searching.

How are results ordered/ranked? #

While a basic search will retrieve all results that contain your terms anywhere within the metadata record, the Discovery Portal gives the most weight to results that contain your terms in the resource's name.

How often are results updated? #

Currently, the Discovery Portal harvests metadata every quarter. The Sources page lists the last time the metadata was collected from the source.

Can I search on additional metadata fields? #

Yes. The Advanced Search tool allows users to build queries based on dozens of metadata fields. You can browse or search for fields, preview the number of records for each field, and enter your search terms - use the Advanced Search tool.

Can I filter on additional metadata fields? #

The Discovery Portal provides filters to help users narrow down their searches based on metadata that is available. To suggest enhancements to the filtering capabilities, make suggestions on GitHub.

Can I view additional metadata fields before I access the data? #

The Discovery Portal provides dataset details to make it easy for users to get a sense of what each resource contains when browsing search results. This is based on metadata that is available at the source. To suggest metadata enhancements, make suggestions on GitHub.

Why are some metadata fields empty? #

The Discovery Portal attempts to standardize metadata that is available; however, it cannot create information that does not exist. If metadata is missing at the source, it will also be absent within the Discovery Portal.

How do I download the metadata? #

To download metadata, go to the Search Results page and click the button "Download Metadata."

How do I access metadata via API? #

All the metadata we harvest can be accessed though the API at api.data.niaid.nih.gov/. You can also find API documentation there.

How can I search for COVID-19, influenza, or datasets for other viruses? #

Finding datasets for any disease of interest is as easy as typing the disease name in the NIAID Data Ecosystem’s search bar and pressing ‘Search.’ For further exploration, try using the Disease Page visualizations to view summaries of all resources related to certain high-priority diseases, like COVID-19, influenza, HIV/AIDs, asthma, tuberculosis, and malaria.

Can I use the NIAID Data Ecosystem to find global infectious disease data? #

Yes, resources ingested by the NIAID Data Ecosystem span numerous countries. Try using the Advanced Search tool to build a query that incorporates a location of interest.

Does the NIAID Data Ecosystem include clinical data? #

Yes, the NIAID Data Ecosystem indexes thousands of datasets from clinical studies, including research related to vaccines, diagnostics, therapeutics, and more. Try searching for a disease of interest and using the ‘Measurement Technique’ filter to narrow results to clinical studies.

Can I find time series data in the NIAID Data Ecosystem? #

Yes, the NIAID Data Ecosystem includes datasets from longitudinal studies related to many infectious and immune-mediated diseases. Try searching for a disease of interest and using the ‘Measurement Technique’ filter to narrow results to longitudinal studies.

Where do I ask questions or send feedback? #

For any questions, contact NIAIDDataEcosystem@mail.nih.gov. You can also submit issues or make suggestions on GitHub.