Citations from scientific articles are more than lines on a page. They can, when reading between those lines, shed some light on the development of scientific thought and on the progress of biomedical technology. We’ve previously posted some examples in blogs here, here, and here. But to better see the light, we all would benefit from more comprehensive data and easier access to them.
My colleagues within the NIH Office of Portfolio Analysis sought to answer this call. Drs. Ian Hutchins and George Santangelo embarked on a hefty bibliometric endeavor over the past several years to curate biomedical citation data. They aggregated over 420 million citation links from sources like Medline, PubMed Central, Entrez, CrossRef, and other unrestricted, open-access datasets. With this information in hand, we can now take a better glimpse into relationships between basic and applied research, into how a researchers’ works are cited, and into ways to make large-scale analyses of citation metrics easier and free.
As described in their recent PLOS Biology essay, the resulting resource, called the NIH Open Citation Collection (OCC), is now freely available and ready for the biomedical and behavioral research communities to use. You can access, visualize, and bulk download OCC data as part of the NIH’s webtool called iCite (Figure 1). iCite allows users to access bibliometric tools, look at productivity of research, and see how often references are cited.Figure 1
Figure 2 illustrates the new OCC web interface. Data from a group of publications are displayed on a summary table on the top. Various charts with visualizations lie beneath the summary table. They show publications over time (left), total citations per year by the publication year of the referenced article (center left) or the citing article (center right), and average citations per article in each publication year (right). These tables are customizable as publications are selected or deselected from the portfolio. You can also see information related to the article, such as links to the citing and referenced papers on PubMed, on the bottom of the screen.Figure 2
The new OCC resource collection within iCite aims to reduce the costs of large-scale analyses of structured citation data, a recognized impediment for the bibliometrics field. OCC goes further still. It enhances the quality, robustness, and reproducibility of analyses using citation data. Moreover, it allows those interested to freely access structured data and share it with others. And, it also provides for transparency, which improves understanding of how knowledge flows and applied technologies develop.
Let’s use OCC to see that knowledge flow in action (Figure 3). Here the team assessed citation networks associated with the development of cancer immunotherapy. Each dot represents a scientific paper. The color represents whether the paper describes basic (green), translational (yellow), or clinical (red) science. The most influential clinical trials are shown in the large red dots in the center. These trials formed part of the evidence base FDA required for approval as a clinical treatment.Figure 3
Information available in OCC will continue to grow. In addition to accumulating citations, the OCC will acquire data preprint servers and other materials currently not indexed in PubMed.
We invite you to take a look at and use the OCC. It will be exciting to see how the research community will use this new resource when conducting their own analyses. Data from these studies delving into citation dynamics may even provide additional insights that help all of us better understand how the scientific enterprise works and how we could make it even better.
As is now well known, black scientists are less successful than their white counterparts in obtaining support from NIH R01 awards as designated Principal Investigators (PIs) (see here and here). Though recent NIH efforts are showing promise to enhance diversity in the biomedical workforce (see this post), much work is still needed to address the funding gap.
In a paper recently published in Science Advances, we delved into the underlying factors associated with this funding gap. We identified three decision points where disparate outcomes arose between white and black researchers: 1) the decision to bring applications to discussion during peer review study section meetings; 2) impact score assignments for those applications brought to discussion; and (3) a previously unstudied factor, topic choice – that is what topic the investigators chose to study.
We analyzed 157,549 R01 applications, both new and renewals, from fiscal years 2011-2015. We confirmed previous findings that black researchers submit fewer applications as PIs than white researchers. Applications with black scientists as PIs were brought to discussion in peer review only 77% as frequently as applications from white researcher PI’s (Figure 1).Figure 1
When applications from black researchers were discussed in study section, they received worse impact scores— 38.4 + 13.4 vs 35.2 + 12.6. Combining lower submission rates, lower discussion rates, and worse impact, black scientists receive R01 funding only half as often as their white peers (Figure 1).
We found a number of differences in the characteristics of applications according to the race of the designated PI. Black scientists were more likely to propose research involving human subjects and less likely to propose work involving animal models.
We next used an informatics method called “word2vec” to bin the 157,549 applications into 150 topic clusters, which roughly aligns with the number of standing study sections at NIH. We can describe these clusters by word chains like “retina photoreceptor retinal cone MeSH_Photoreceptor_Cells_Vertebrate rod photoreceptor cells retinal degeneration” or “practice provider clinician care education evidence-based healthcare recommendation medical psychosocial.” Figure 2 shows word clouds associated with the topic clusters with the highest number of applications from black PIs (panel A) and with clusters in which there were no applications from black PIs (panel B). Note that the words in panels A and B are clearly qualitatively different.
A closer look at Figure 2 shows that black applicants were more likely to be associated with topics like health disparities, disease prevention and intervention, socioeconomic factors, healthcare, lifestyle, psychosocial, adolescent, and risk (panels A and C). Generally speaking, applications with these terms were less likely to be funded than topics linked like neuron, corneal, cell, and iron (panel B).Figure 2
Figure 3 summarizes the word2vec topic cluster data for all applications. Panel A shows the topics in descending order of proportion of applications designating black PIs. Panel B shows the number of applications for each topic (presented in the same order as in panel A). A third of the applications from black scientists mapped to only eight of the 150 topic clusters, which tended to experience lower success rates (panel C). These lower-success topic clusters tended to focus on community and population-level research. Of note, applications from white researchers in these lower-success topic clusters were also less likely to be funded, although not to the same degree as those from black researchers.Figure 3
We performed a series of multivariable analyses and found that after controlling for variables like applicants’ prior success, topic selection accounted for 21 percent of the funding gap observed between black and white researchers.
We briefly discussed potential implications of our findings – including the need to encourage a more diverse applicant pool, the potential value of mentoring systems to help investigators navigate the NIH system, and the possibility for NIH institutes and centers to consider discretionary funding for topics that may be under-appreciated by review but align with strategic priorities.
Whether you’re an awardee or an applicant interested in Career Development (K) awards, you probably have some questions. Use this post as a starting point to getting your answers with the following resources:
Still have questions about percent effort and K awards for your K award, or for a specific K award PI at your institution? Contact the grants specialist listed on the notice of award for guidance specific to you. General policy questions may be directed to the Division of Biomedical Research Workforce at NIHTrain@mail.nih.gov.
We make data on all funded NIH grants available to the public on the RePORT website. One of the ways we provide information is by school/department, which you can explore using the Awards by Location feature. Because of inconsistencies in the way information on department and school names are provided in grant applications, grantee officials may want to make changes in how that information is reflected in NIH systems.
NIH’s fiscal year ended on September 30, 2019, so now is the time for Signing Officials to verify the accuracy of their grant assignments to departments or components within institutions of higher education using the Grant Re-assign function in eRA Commons. Since the data in these files are “frozen” annually to ensure the reporting files produce consistent and meaningful results, any corrections must be made by 8:00 PM EDT on Friday, October 11, 2019 to be reflected in NIH annual reports.
Read the NIH Guide Notice for more information.
General registration rates for the NIH Regional Seminar on Program Funding and Grants Administration in Phoenix, AZ end on October 11. If you are new to the world of NIH funding then don’t miss this opportunity to register before the deadline!
The NIH Regional Seminar is a great introduction to NIH grants process and policies, offering flexible session tracks for administrators, new investigators, and all interests. Just some of the featured hot topics include:
Get to know the ins and outs of NIH funding by joining us on November 6-8 for the Fall 2019 NIH Regional Seminar in Phoenix, Arizona. See the tentative agenda, hotel/travel details, and more on the NIH Regional Seminar site.
The NIH Grants & Funding website has a wealth of information to help applicants and grant recipients navigate application submission and grant administration requirements. When you can’t find what you need online, don’t hesitate to reach out to NIH staff. Often, the best folks to talk to will be in one of the NIH institutes or Centers. Our Contacting Staff at the NIH Institutes and Centers page can help you understand the roles of NIH staff and help you contact the right person at each phase of the application and award process.
Having trouble navigating our eRA systems? Check out our general Help page for eRA Service Desk and other general support contacts.
Communicating with NIH staff is not only okay, it is encouraged. When in doubt, reach out – we’re here to help.
NIH is currently accepting public comments on the use of standards for capturing, integrating, and exchanging clinical data for research purposes (NOT-OD-19-150). This is a great opportunity to hear more from the community on ways to strengthen approaches that find, share, and access high-quality patient data, while also making it more interoperable and reusable. Such goals align with long-standing NIH data sharing policies and what was also called for in a related NIH strategic plan on data science.
The Request for Information focuses on the Fast Healthcare Interoperability Resources (FHIR®) standard (see also NOT-OD-19-122). Widely promoted and adopted for use in clinical care, FHIR is a standardized way to transmit structured clinical data between health information systems, while also protecting patient privacy and security, via an application programming interface.
The 21st Century Cures Act also called for such computational strategies to enhance the interoperability of electronic health records. The U.S. Department of Health and Human Services is coordinating this process and has called for the health care industry to adopt such strategies by using the FHIR standard to share patient data. Moreover, federal agencies and the private sector have used FHIR in various ways, from exchanging claims data, allowing individuals to import health records from providers, and integrating clinical trial management.
When putting the FHIR standard into practice, NIH can better ensure taxpayer resources are efficiently used to maximize their public value. For example, it could quicken the pace and lower costs for collecting, integrating, and using patient data available on trusted electronic health records for medical research. It is conceivable that when clinical, genomic, demographic, billing, claims, and socioeconomic information is easily accessible and interoperable, novel biomedical and behavioral research ideas could be tested, possibly leading to advances in science and public health. As a start towards this, we began soliciting ideas from the small business community to implement this standard in health information technology (NOT-OD-19-127).
Your thoughts on how NIH-funded researchers could adopt FHIR are welcomed electronically here through November 23, 2019. General topics of interest include researcher experiences with FHIR, one’s willingness to use it, any necessary tools, the need for research related to standards development, opportunities, and challenges.
We are pleased to announce that last month the National Institutes of Health (NIH), the Animal and Plant Health Inspection Service of the United States Department of Agriculture (USDA), and the Food and Drug Administration (FDA) published their final report on Reducing Administrative Burden for Researchers: Animal Care and Use in Research (NOT-OD-19-136). This report, called for in the 21st Century Cures Act, is the culmination of more than two years of diligent work to address inconsistent and overlapping policies governing oversight of research involving animals, while ensuring research findings remain credible and research institutions safeguard animal welfare.
Regulations and policies overseeing federally-sponsored research help ensure NIH remains a responsible steward of public funds. This extends to properly enforcing the public’s expectation on how laboratory animals will be used in biomedical research, teaching, and testing. Many stakeholders have pointed out, rightfully so in some cases, that with federal regulations comes increasing reporting requirements which adversely affect research productivity without necessarily improving animal welfare.
Experts from the USDA, the FDA, and the NIH undertook much collaborative, thoughtful, and rigorous deliberation to get us to this point. More on their process can be found here, here, and here. Their final report reflects, incorporates, and analyzes diverse perspectives of more than 1,300 researchers, animal welfare advocacy organizations, scientific professional societies, members of the public, and others who commented on an earlier draft version. Investigators, generally speaking, supported the report’s recommendations to reduce various administrative hurdles. But, importantly, they were also balanced with concerns raised by animal welfare organizations and the public to not lessen requirements in such a way that adversely effects the humane care and use of laboratory animals.
Well, what did they find? Below are some highlights. I encourage everybody to read the report for the full details and recommendations.
Opportunities exist to reduce duplicative regulations and policies. These focus generally around areas such as inspections, animal protocol review, and institutional reporting. For example, we will consider flexibilities in how and by whom semiannual inspections by Institutional Animal Care and Use Committees (IACUCs) are conducted. It may be possible to streamline some IACUC animal protocol reviews, such as by using Designated Member Review for low-risk activities. And, NIH and USDA will establish a process allowing annual reporting to both agencies on the same reporting schedule (such as through a streamlined data submission on a shared portal).
We recognize the importance and value of better coordination across federal funders. We will allow public comments for at least sixty days for all new significant policy guidances. Building on that, guidances from NIH, USDA, and other federal funders will be harmonized as appropriate to reduce duplicative and inconsistent requirements. We concur with the suggestion to enhance training and resources across federal agencies. One way to achieve this will be continuing to work with our USDA colleagues to develop new industry-led training and web resources to strengthen IACUC activities that focus on reducing burden on investigators.
Keep an eye out for our next steps. We are currently looking at all of the recommendations and thinking about possible ideas, guidance, and resources to address them. Any planned changes will be implemented over the next couple years, along with a strategy that considers how effective they were. This timeframe affords ample opportunity for your voice to still be heard along the way.
I would like to congratulate and thank my colleagues Dr. Patricia Brown and Ms. Lori Hampton (both of the NIH Office of Laboratory Animal Welfare) for their leadership in helping to prepare the report and for their help in writing this summary.
Letters of support are a valuable part of your grant application. They provide an opportunity for you to document the commitment and support of your institution and collaborators, the availability of required resources, and more.
In this next installment of the NIH’s All About Grants podcast series, Cathleen Cooper, Ph.D., who directs the NIH’s Center for Scientific Review’s Division of Receipt and Referral, joins us to talk all about letters of support (MP3 / Transcript). Hear what information should be included in these letters, what should not, how they differ from other letters submitted as part of an application, and more.
Applications are currently being accepted for the FY 2020 NIH Loan Repayment Program (LRP) until November 15, 2019. And, there are some important changes to the program we would like to spotlight for you.
For over three decades, the NIH LRPs have helped recruit and retain highly qualified health professionals into biomedical or biobehavioral research careers. LRP award funds repay a recipient’s qualified educational debt in return for a commitment to engage in NIH mission-relevant research at a domestic, nonprofit, or government entity. Priority research areas for the NIH LRPs include clinical, pediatrics, health disparities, contraception and infertility, as well as clinical research for individuals from disadvantaged backgrounds.
And, now here are some exciting new changes we would like to highlight about the LRPs going forward…
Starting earlier this month, NIH raised the loan repayment award amount from a maximum of $35,000 to a maximum of $50,000 per year (NOT-OD-19-117). This change, as part of implementing the 21st Century Cures Act, means that over a two-year new award, talented researchers can now receive a maximum of up to a total of $100,000. LRP awards are also competitively renewable.
Raising the LRP award maximum amount – a recommendation first presented in the 2014 NIH Physician Scientist Workforce Working Group report – is an exciting step in further strengthening and stabilizing the future of the biomedical research workforce. Staggering levels of educational debt – acquired during years of education and training – is a significant and often-reported barrier to beginning and sustaining a research career. We recognize that this is not only a financial barrier to early career scientists, but also an impediment to research innovation and discovery. Through this step forward, we aim to help early career scientists tackle educational debt in a meaningful way and continue to proactively shape the next phase of the biomedical research workforce
Next, all NIH Institutes and Centers will now fund LRP awards in FY 2020 in priority minority health and health disparities research areas. Previously, only one NIH institute supported health disparity LRP awards, which unfortunately led to many high-quality applications going unfunded each year. Expanding participation across NIH in this LRP will now enable a larger proportion and wider array of ideas in health disparities research be funded.
Finally, Emerging and Gap Areas of Research will be added as a new LRP category starting with next year’s application cycle, with awards made in FY 2021. As part of this new program, we will look for high quality, mission-relevant research ideas designed to pursue major opportunities and gaps in biomedical research and expand critical research in emerging areas of human health. Please stay tuned as more information is to come.
We encourage you to apply today. The LRP Information Center is available to help answer questions at 866-849-4047 or email@example.com. You can also stay connected with us on Twitter and Facebook for more updates.
What’s blue, gray, and read all over? Our refreshed NIH Grants and Funding site!
It’s all the content you rely on, but with an improved interface for mobile devices and updated colors and fonts and more to improve the site’s accessibility and streamline navigation. (We made no changes to the site structure or URLs.)
Have suggestions for content? We read every comment submitted through the in-page survey tool, so keep the comments coming so we can continue to improve.
Take a look and see what you think!
If your institution closes due to severe weather or other natural disasters, NIH has policies in place to help your research to continue. We recently published an NIH Guide Notice that reminds those impacted by Hurricane Dorian about the flexibilities for application and report submission provided by these policies.
For more resources, including guidance on animal welfare issues, check our Extramural Response to Natural Disasters page.
Reference letters and letters of support provide key information for reviewers and NIH staff. Check out the table below for an overview of when each letter is used, who writes them, and what should be included.Reference Letters Letters of Support When are they used? Used in Fellowships, mentored Career Development Awards, and other programs as requested Used to demonstrate:
We encourage everybody from graduate students to senior scientists to register for an ORCID account and link it to their eRA Commons personal profile. Starting October 1, 2019, ORCID identifiers will be required for individuals supported by institutional research training, career development, and other research education awards. xTrain appointments will not be accepted for agency review if potential appointees do not have an ORCID iD linked to their eRA personal profile. ORCID iDs will also be required for PD/PIs on individual fellowship and career development applications submitted for due dates on or after January 25, 2020.
Beginning with RPPRs due on or after October 1, 2019 (FY 2020), recipients must use the xTRACT system to create the required training tables for submission with NIH and AHRQ T15, T32, T90/R90, and TL1 progress reports. While it is not mandatory to use xTRACT for new and renewal applications for the specified types of training grants, it may be required in future years.
Check out our resources on xTRACT such as the user guide, instructional videos, and FAQs, available on the eRA website. For more details on its required use and implementation, see the full Guide Notice.
What better way to learn about NIH grants policy and processes than straight from the source? The NIH Regional Seminar on Program Funding and Grants Administration provides an array of pre-seminar workshops and sessions over the course of three days, all presented by 70 NIH & HHS review, program, grants and policy experts! Check out some of these topics designed to help you understand the NIH grants process, such as:
In addition to approximately 45 different session and workshop topics to choose from, you also have the opportunity to meet with our experts 1:1 to address your specific questions. Make plans to join your peers from all over the world and register today for the Fall 2019 NIH Regional Seminar in Phoenix, Arizona, November 6-8, 2019. See the tentative agenda, hotel/travel details, and more on the NIH Regional Seminar site.
The more you know, and the more that can be sent in a single email, the better. Applicant organizations will begin receiving centralized email notifications, listing applications that NIH does not intend to fund, approximately 14 months after the application’s council date. NIH eRA systems will automatically send these consolidated notifications to the Notice of Award email address listed in the organization’s eRA Commons Institutional Profile File (IPF) and the Authorized Organization Representatives (AORs)/Signing Officials (SOs) listed in the included grant applications. In addition, an “Unfunded notification sent on <mm/dd/yyyy>” message will be added to the eRA Commons Status Information screen for each application.
On May 22, I had the privilege of participating in a terrific national conference that focused on what institutions can do to foster a culture of research integrity (see the agenda here). The DHHS Office of Research Integrity (ORI), Northwestern University, and the Council of Graduate Schools hosted the conference, “The Role of Research Integrity in Promoting Excellence: Tools for Colleges and University Leaders.” The conference organizers’ goal was “to engage university and college leaders in lively discussions about strategies, resources, and tools for promoting research integrity for current and future scientists, and scholars at institutions nationwide.” That goal was met and then some. A number of institutional leaders described a number of concrete, practical, and intriguing efforts to promote integrity and excellence.
I was also given the opportunity to present my thoughts on promoting research integrity, something I have written about before. My May 22 talk dealt with approaches institutions may take to foster a culture of research integrity, and I wanted to share it here as a resource for others. By watching the video below, you will hear me discuss:
We hope you enjoy the talk.Re-recording of the presentation provided at the Northwestern University Research Integrity Conference
Sexual harassment is a serious and long-standing issue within the biomedical research enterprise, and NIH is striving to be part of the solution. On this episode of the “All About Grants” podcast, we sit down with Dr. Jodi Black, Deputy Director for the NIH’s Office of Extramural Research, to discuss what institutions, investigators, and others in the research community should know about NIH’s policies and expectations for assuring a safe and harassment-free work environment (MP3 / Transcript). She also outlines how to notify NIH with a potential issue and what NIH does to follow up on these notifications.
For more information on NIH’s anti-sexual harassment efforts, see our Anti-Sexual Harassment webpage.
You might be surprised to learn that we don’t have a generic set of forms posted on our website that you can use to submit to any NIH grant opportunity. Each funding opportunity announcement (FOA) includes the specific set of forms needed to apply to that program, so you need to find an FOA in order to access the application forms. Walk through the process of finding and accessing application forms with this grant application submission tips for success video.
Form-by-form, field-by-field instructions for completing your application may be found on the How to Apply – Application Guide page under the green header for Application Form Instructions. Use these instructions in conjunction with the guidance in the funding opportunity announcement (including the Related Notices section of the announcement) to develop your application. The annotated form sets are another handy tool to help you navigate the forms.