The Use of ChatGPT for Generating Scientific Citations : An experiment
Bergman, Jussi (2023)
2023
Tietojenkäsittelyopin maisteriohjelma - Master's Programme in Computer Science
Informaatioteknologian ja viestinnän tiedekunta - Faculty of Information Technology and Communication Sciences
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Hyväksymispäivämäärä
2023-08-21
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-202307197237
https://urn.fi/URN:NBN:fi:tuni-202307197237
Tiivistelmä
This research examined the capabilities of ChatGPT in producing a numbered list of references for a range of topics and the accuracy of each reference through manual evaluation.
Results suggest moderate levels of precision in generating reference lists, with 55% accuracy in titles, 43% in authors, 44% in sources, and 54% in overall relevance.
Based on the relatively low accuracy of the generated references, this study introduced and applied a novel "Reverse Order Method". This method involves generating a list of references, manually validating each, and then instructing ChatGPT to compose a theoretical introduction based on the validated references alone.
It's implied that the model's precise reproduction of a reference demonstrates repeated exposure and understanding of its content, enabling reliable citation in the final text. All final texts for all topics were evaluated as convincing and good quality scientific text with citations in place.
Though the assessment of the final texts was purely subjective, the study suggest the promising utility of the Reverse Order Method in crafting scientific texts using ChatGPT 3.5. The study underscores the potential of AI tools like ChatGPT in scientific writing, emphasising the role of manual validation in improving precision and the careful use of AI-generated references.
To enhance the understanding of the results, the study further explored the intricate inner workings of ChatGPT, concentrating on its 'transformer' architecture and the pursued 'learning objectives'. The study offered intuition by exploring the essential principles of language comprehension. A graphical representation, built upon existing research, was employed to illuminate this complex procedure.
Results suggest moderate levels of precision in generating reference lists, with 55% accuracy in titles, 43% in authors, 44% in sources, and 54% in overall relevance.
Based on the relatively low accuracy of the generated references, this study introduced and applied a novel "Reverse Order Method". This method involves generating a list of references, manually validating each, and then instructing ChatGPT to compose a theoretical introduction based on the validated references alone.
It's implied that the model's precise reproduction of a reference demonstrates repeated exposure and understanding of its content, enabling reliable citation in the final text. All final texts for all topics were evaluated as convincing and good quality scientific text with citations in place.
Though the assessment of the final texts was purely subjective, the study suggest the promising utility of the Reverse Order Method in crafting scientific texts using ChatGPT 3.5. The study underscores the potential of AI tools like ChatGPT in scientific writing, emphasising the role of manual validation in improving precision and the careful use of AI-generated references.
To enhance the understanding of the results, the study further explored the intricate inner workings of ChatGPT, concentrating on its 'transformer' architecture and the pursued 'learning objectives'. The study offered intuition by exploring the essential principles of language comprehension. A graphical representation, built upon existing research, was employed to illuminate this complex procedure.