Creating a knowledge base for a custom GPT

Creating a knowledge base for a custom GPT#

We created a custom GPT (Generative Pretrained Transformer) model, designed specifically to assist in the analysis and synthesis of information pertaining to the 31 reproducible research insights. The knowledge base of this retrieval-augmented generation framework comprises GPT-4 summaries of RRInsights interviews coupled with the original abstract, title, and keywords of the respective articles. This specialized GPT, named RRInsights, is tailored to process and interpret the provided data in the context of reproducibility. Through its advanced natural language processing capabilities, RRInsights can efficiently analyze the scoped literature, extract key insights, and generate comprehensive overviews of the research papers focusing on MRI technology.

Warning

Setting up OpenAI authorization To enable API calls, you need to add your OpenAI token to the .env file.

Warning

Starting a session with OpenAI API The following code requires GPT tokens to feed insight files one by one and to return a summary. Estimated cost ~6-8 USD.

API key not found. Available dataset will be used. Example: 

[{'role': 'system', 'content': 'You are a helpful assistant designed to understand and summarize the reproducibility aspects of neuroimaging research articles.'}, {'role': 'user', 'content': 'The following content is from ARTICLE 1. In relavance to its title and abstract, create 4 keywords, then summarize what did the study achieve and why it was reproducible in 4 sentences: Title: In vivo magnetic resonance 31P‐Spectral Analysis With Neural Networks: 31P‐SPAWNN \nAbstract: We have introduced an artificial intelligence framework, 31P‐SPAWNN, in order to fully analyze phosphorus‐31 ( 31$$ {}^{31} $$ P) magnetic resonance spectra. The flexibility and speed of the technique rival traditional least‐square fitting methods, with the performance of the two approaches, are compared in this work. \nReproducibility Insights:\nGeneral questions\nQuestions about the specific reproducible research habit\n\nThis MRM Reproducible Research Insights interview is with Julien Songeon and Antoine Klauser from the University of Geneva in Switzerland, respectively first and last author of a paper entitled “In vivo magnetic resonance 31 P-Spectral Analysis With Neural Networks: 31P-SPAWNN”. This work was singled out because it demonstrated exemplary reproducible research practices; specifically, the authorsshared the source code for their proposed method (simulations, their deep learning network, and sample datasets).\nTo learn more, check outour recent MRM Highlights Q&A interview with Julien Songeon and Antoine Klauser.\n1. Why did you choose to share your code/data?\nThis approach naturally facilitates the demonstration presented in the manuscript, allowing interested readers to test the method themselves. Also, we enable others to validate and reproduce our results effectively.\nAlso, rather than adopting a competitive approach, we prefer, in this way, to promote collaboration in MR research, because a collaborative mindset can foster advancements and progress in the field. Ultimately, code sharing can assist other researchers who wish to pursue similar research directions. They can utilize our work as a foundation, potentially saving significant implementation and coding efforts. This accelerates the pace of research and encourages researchers to build upon existing knowledge rather than wasting resources on replication efforts.\n2. What is your lab or institutional policy on sharing research code and data?\nOur institution has a highly flexible policy regarding the sharing of code and data. While open science practices are strongly encouraged, there are no clear advantages for us nor mandatory constraints in place that require us to share .\n3. How do you think we might encourage researchers in the MRI community to contribute more open-source content along with their research papers?\nI believe that if code and data sharing were recognized to add clear value to a submitted manuscript, then authors would have a positive incentive to provide open-source content. In addition to evaluating factors such as innovation, soundness of methodology, quality of results, and clarity of demonstration, editors should acknowledge the inclusion of code and data as a valuable contribution to the scientific community.\n1. Your code repository is really well documented. What advice would you give to people preparing to share their first repository?\nWhen preparing to share a code repository, you need to think about the future users of your code and what information they will need to understand and use it effectively. The README file should explain the purpose of the code, and provide installation instructions, dependencies, and examples. The files could have comments within the code to clarify specific sections or functions. To help users navigate the repository and locate relevant files easily, I suggest organizing your code and files in a logical manner, using explicit naming. Finally, I would recommend including examples and/or tutorials, providing sample input data and scripts that demonstrate how to use your code. This can really help users to understand the expected workflow and apply the code to their own research.\n2. What questions did you ask yourselves while you were developing the code that would eventually be shared?\nIn the development phase, we aimed to create code that could be easily adapted and reused for different experiments or in different research scenarios.We also considered the computational complexity and performance of our code to ensure that it could handle large training datasets or complex computations effectively.Finally, we paid attention to the clarity and comprehensibility of our code. We aimed to write code that would be accessible to other researchers.\n3. How do you recommend that people use the project repository you shared?\nStart with the README file. Read the README file and try to rerun the command shown. it will provide an overview of the capability and usage of the code and model.Once you have a good understanding of the code, try to run it on your own data, adapting the number of points and the bandwidth and retraining a new model.Then, further adaptation could be implemented: you could add other metabolites, apply the method to other nuclei, explore other model architectures.\n4. Are there any other reproducible research habits that you didn’t use for this paper but might be interested in trying in the future?\nUsing tools likeDockerorSingularityto create reproducible and portable computing environments. This ensures that the code can be executed consistently across different systems and configurations without cumbersome installation or setup. In addition, keeping track of and archiving the successive versions of the code in a systematic manner allows researchers to easily track and manage changes to the code over time. This can be particularly useful when code is linked to a publication and extra data are required during the peer-review process. Also, we could try making automated pipelines that capture the entire workflow, from data acquisition to final results. Doing this simplifies the reproducibility process and reduces the potential for human error.\n'}, {'role': 'assistant', 'content': 'Keywords: 31P‐SPAWNN, neural networks, reproducibility, magnetic resonance spectroscopy\n\nStudy Achievements and Reproducibility Summary:\n\n1. The study introduced a new artificial intelligence framework, 31P‐SPAWNN, for analyzing 31P magnetic resonance spectra, demonstrating that it provides a fast and flexible alternative to traditional least-square fitting methods.\n2. Reproducibility was exemplary as the authors shared the source code, simulations, their deep learning network, and sample datasets, enabling other researchers to test, validate, and reproduce their results easily.\n3. Open sharing of the code and data was motivated by a commitment to collaborative advancement in MR research and the drive to facilitate incremental knowledge growth within the community without redundant effort.\n4. The authors suggest best practices for sharing research repositories, including comprehensive documentation, logical organization, and the use of tools like Docker or Singularity to improve the portability and consistency of the computational research environment, thereby enhancing reproducibility further.'}]

Synthesis

Create a high-level review of all the summarized content

API key not found. An example output will be displayed from the previous results. 

**Scoping Review: Reproducibility in Neuroimaging Research**

**Introduction**
Reproducibility remains a cornerstone of scientific inquiry, its importance amplified in the complex and rapidly evolving field of neuroimaging. Studies leveraging magnetic resonance imaging (MRI), magnetic resonance spectroscopy (MRS), and other neuroimaging modalities are inherently data-intensive and computationally demanding. Ensuring reproducibility thus poses a unique set of challenges and necessitates the diligent application of methods that foster transparency, verification, and extension of research findings. This scoping review examines the current landscape of reproducible research practices in neuroimaging across various studies, drawing attention to common strategies, tools, and repositories used to achieve reproducible outcomes.

**Methods and Techniques for Ensuring Reproducibility**
Reproducibility in neuroimaging research is predicated upon the open sharing of comprehensive data sets, the code underlying data processing and analysis, and the complete details of the experimental setup. A review of recent neuroimaging research articles reveals several key themes central to facilitating reproducibility in the field:

1. **Open-Source Software and Code Sharing:** The widespread adoption of open-source software such as FSL-MRS for MRS analysis, hMRI for MRI analysis, and Dockerfiles for environment replication reveals a community trend towards accessible and transparent computational tools. Papers detailing deep learning approaches, such as DeepCEST for chemical exchange saturation transfer (CEST) MRI analysis, typically accompany methodological descriptions with shared code repositories and pre-trained models. These resources enable other scientists to test, apply, and enhance the computational techniques introduced.

2. **Data Sharing and Standardization:** The growing endorsement of data sharing is reflected in the number of studies providing access to raw imaging data, pre-processing pipelines, and post-analysis results. Efforts to standardize data formats, such as using NIfTI for spectroscopy data or adhering to Brain Imaging Data Structure (BIDS) for organizing complex datasets, are of increasing importance. Repositories like Zenodo, XNAT, and the Open Science Framework (OSF) serve as vital resources for housing and curating neuroimaging data.

3. **Integration of Hardware and Software:** Several studies such as those integrating RF coils with commercial field cameras for ultrahigh-field MRI or developing vendor-neutral imaging sequences for quantitative MRI exemplify the coupling of hardware advancements with software solutions. Sharing design files (e.g., CAD) and performance data ensures that hardware innovations can be reproduced and utilized by other researchers.

4. **Quality Assurance and Validation:** Implementing quality assurance (QA) measures is essential for the validation of neuroimaging findings. The introduction of tools for automated QA processes, as seen in the development of platforms like PreQual for DWI analysis, signifies an emphasis on ensuring the integrity and reliability of neuroimaging data.

**Discussion and Future Directions**
The components of reproducibility in neuroimaging research are multifaceted, integrating not just data and code, but also the analytical pipelines and hardware configurations. The shift towards comprehensive sharing is motivated both by a scientific ethic of transparency and the practical need for rigorous validation of complex methodologies. As computational resources become more sophisticated, there are increasing calls for the incorporation of standardized descriptors, metadata, and data formats within the community.

Reproducibility is also bolstered by interactive documentation and tools such as Jupyter Notebooks, allowing for dynamic presentation and hands-on engagement with data and methods. Platforms incorporating such interactive elements are being utilized with greater frequency, providing real-time demonstration of analysis techniques and enabling peer-led validation.

The rise of machine learning and artificial intelligence in neuroimaging demands meticulous scrutiny for reproducibility. Studies implementing these technologies are beginning to pair their methodological descriptions with the open-source code, trained models, and simulation tools that underpin their algorithms.

**Conclusion**
The progress towards reproducibility in neuroimaging research points to a distinct cultural shift in the scientific community. The move towards open-access publishing, code-sharing platforms, and data repositories reflects a concerted effort to uphold the reproducibility of complex imaging studies. The initiatives and tools identified in this review serve as a blueprint for future studies to replicate successful practices, safeguards against bias, and accelerate neuroscientific discovery. As neuroimaging research continues to advance, upholding the principles of reproducibility will be essential to maintaining the integrity and translational potential of its findings.

Save outputs

Save GPT-generated summaries with matching paper IDs

API key not found. 3 example outputs will be displayed. 

Keywords: 31P‐SPAWNN, neural networks, reproducibility, magnetic resonance spectroscopy

Study Achievements and Reproducibility Summary:

1. The study introduced a new artificial intelligence framework, 31P‐SPAWNN, for analyzing 31P magnetic resonance spectra, demonstrating that it provides a fast and flexible alternative to traditional least-square fitting methods.
2. Reproducibility was exemplary as the authors shared the source code, simulations, their deep learning network, and sample datasets, enabling other researchers to test, validate, and reproduce their results easily.
3. Open sharing of the code and data was motivated by a commitment to collaborative advancement in MR research and the drive to facilitate incremental knowledge growth within the community without redundant effort.
4. The authors suggest best practices for sharing research repositories, including comprehensive documentation, logical organization, and the use of tools like Docker or Singularity to improve the portability and consistency of the computational research environment, thereby enhancing reproducibility further.

Keywords: Cystic Fibrosis, Hyperpolarized 129Xe MRI, Dynamic 19F MRI, Ventilation Abnormalities

Study Achievements and Reproducibility Summary:

1. The study achieved a quantitative comparison between dynamic 19F MRI and single breath hyperpolarized 129Xe MRI for detecting ventilation abnormalities in subjects with mild CF lung disease, contributing valuable insights into the efficacy of different imaging modalities for CF assessment.
2. Reproducibility was enhanced by sharing the code and data that reproduce several figures, allowing others in the research community to validate and potentially extend the findings.
3. The decision to share came during the submission process, and recognizing the benefits, including feedback and collaboration, the team has been motivated to continue sharing for future publications.
4. The researchers advocate for culture change within the MRI community where data sharing becomes an expectation and plan to explore additional reproducibility tools, such as explanatory videos, to further disseminate their research methods and findings.

Keywords: Dynamic MRI, Non-gated Acquisition, Large-scale Volumetric Imaging, Pulmonary Imaging

Study Achievements and Reproducibility Summary:

1. The study achieved the development of a framework capable of reconstructing large-scale volumetric dynamic MRI images from continuous, non-gated acquisitions, with successful applications demonstrated in pulmonary and DCE imaging.
2. Reproducibility was prioritized through the sharing of code and an interactive Google Colab notebook demo, which allows for easy visualization and manipulation of the reconstruction process.
3. The culture of sharing within UC Berkeley and the research team directly influenced the early decision to share code/data, with the practice of version controlling, documentation, and unit tests facilitating the sharing process.
4. Encouraging open-source contributions in the MRI community via showcases, hands-on sessions, and interviews has proven effective, and the authors intend to continue these efforts while exploring additional platforms like Zenodo for data hosting.

Create GPT embeddings

Use text-embedding-ada-002 model to create embeddings for the summaries.

                                             summary  n_tokens  \
Keywords: 31P‐SPAWNN, neural networks, reprodu...       201   
Keywords: Cystic Fibrosis, Hyperpolarized 129X...       203   
Keywords: Dynamic MRI, Non-gated Acquisition, ...       190   
Keywords: Real-Time MRI, Speech Production, 3D...       215   
Keywords: Inhomogeneous Magnetization Transfer...       195   

                                           embedding  
[-0.012139617465436459, 0.015481614507734776, ...  
[-0.0038842952344566584, 0.019686615094542503,...  
[-0.025156814604997635, 0.017279978841543198, ...  
[-0.028323667123913765, 0.013008967973291874, ...  
[-0.038999006152153015, 0.013461689464747906, ...  

import os
import requests

DATA_ROOT = "../data/repro_mri_scoping/gpt_summary_nov23"

# Function to make Semantic Scholar API call and extract title and authors
def get_paper_info(paper_id):
    api_url = f"https://api.semanticscholar.org/graph/v1/paper/{paper_id}?fields=url,year,authors,title"
    response = requests.get(api_url)
    
    if response.status_code == 200:
        data = response.json()
        title = data.get('title', 'Title Not Available')
        year = data.get('year', 'Year Not Available')
        authors = ', '.join(author.get('name', 'Unknown Author') for author in data.get('authors', []))
        return title, authors, year
    else:
        print(f"Failed to fetch data for paper_id: {paper_id}")
        return None, None, None

# Directory containing txt files
directory = DATA_ROOT

# Initialize an empty list to store data
combined_data = []

# Loop through each txt file in the directory
it =1
for filename in os.listdir(directory):
    if filename.endswith(".txt"):
        paper_id = os.path.splitext(filename)[0]
        file_path = os.path.join(directory, filename)
        
        # Read content of the txt file
        with open(file_path, 'r', encoding='utf-8') as file:
            content = file.read().strip()
        
        # Make Semantic Scholar API call to get paper title and authors
        title, authors, year = get_paper_info(paper_id)
        
        # Append data to the list
        combined_data.append(f"\nPaper number: {it}\nYear: {year}\nTitle: {title}\nAuthors: {authors}\n\n{content}\n\n ---------------------------------------------------")
        it = it+1
# Write the combined data to a text file
with open('combined_papers_insight.txt', 'w', encoding='utf-8') as output_file:
    output_file.writelines(combined_data)

1: Azeez Adebimpe, Maxwell Bertolero, Sudipto Dolui, Matthew Cieslak, Kristin Murtha, Erica B Baller, Bradley Boeve, Adam Boxer, Ellyn R Butler, Phil Cook, Stan Colcombe, Sydney Covitz, Christos Davatzikos, Diego G Davila, Mark A Elliott, Matthew W Flounders, Alexandre R Franco, Raquel E Gur, Ruben C Gur, Basma Jaber, Corey McMillian, ALLFTD Consortium, Michael Milham, Henk J M M Mutsaerts, Desmond J Oathes, Christopher A Olm, Jeffrey S Phillips, Will Tackett, David R Roalf, Howard Rosen, Tinashe M Tapera, M Dylan Tisdall, Dale Zhou, Oscar Esteban, Russell A Poldrack, John A Detre, and Theodore D Satterthwaite. ASLPrep: a platform for processing of arterial spin labeled MRI and quantification of regional brain perfusion. Nat. Methods, 19(6):683–686, June 2022.
2: Hilary Arksey and Lisa O'Malley. Scoping studies: towards a methodological framework. Int. J. Soc. Res. Methodol., 8(1):19–32, February 2005.
3: William T Clarke, Charlotte J Stagg, and Saad Jbabdi. FSL-MRS: an end-to-end spectroscopy analysis package. Magn. Reson. Med., 85(6):2950–2964, June 2021.
4: missing journal in Cohan2020-tw
5: Suzanne Fricke. Semantic scholar. J. Med. Libr. Assoc., 106(1):145, January 2018.
6: Danielle Levac, Heather Colquhoun, and Kelly K O'Brien. Scoping studies: advancing the methodology. Implement. Sci., 5:69, September 2010.
7: missing journal in McInnes2018-sc
8: Micah D J Peters, Christina M Godfrey, Hanan Khalil, Patricia McInerney, Deborah Parker, and Cassia Baldini Soares. Guidance for conducting systematic scoping reviews. Int. J. Evid. Based Healthc., 13(3):141–146, September 2015.
9: Layla Tabea Riemann, Christoph Stefan Aigner, Ralf Mekle, Oliver Speck, Georg Rose, Bernd Ittermann, Sebastian Schmitter, and Ariane Fillmer. Fourier-based decomposition for simultaneous 2-voxel MRS acquisition with 2SPECIAL. Magn. Reson. Med., 88(5):1978–1993, November 2022.
10: Julien Songeon, Sébastien Courvoisier, Lijing Xin, Thomas Agius, Oscar Dabrowski, Alban Longchamp, François Lazeyras, and Antoine Klauser. In vivo magnetic resonance 31 P-Spectral analysis with neural networks: 31P-SPAWNN. Magn. Reson. Med., 89(1):40–53, January 2023.
11: Nikola Stikov, Joshua D Trzasko, and Matt A Bernstein. Reproducibility and the future of MRI research. Magn. Reson. Med., 82(6):1981–1983, December 2019.
12: B Tilea, C Alberti, C Adamsbaum, P Armoogum, J F Oury, D Cabrol, G Sebag, G Kalifa, and C Garel. Cerebral biometry in fetal magnetic resonance imaging: new reference data. Ultrasound Obstet. Gynecol., 33(2):173–181, February 2009.