UMIN-ICDS Clinical Trial
| Unique ID issued by UMIN | UMIN000053582 |
|---|---|
| Receipt number | R000061116 |
| Scientific Title | Performance of the ChatGPT (Generative Pretrained Transformer-4) on the Certification Test for Mental Health Management: Factorial design |
| Date of disclosure of the study information | 2024/02/09 |
| Last modified on | 2024/03/13 09:59:23 |
* This page includes information on clinical trials registered in UMIN clinical trial registed system.
* We don't aim to advertise certain products or treatments
Basic information
Public title
Performance of the ChatGPT on the Certification Test for Mental Health Management: Factorial design
Acronym
Performance of the ChatGPT on the Certification Test for Mental Health Management
Scientific Title
Performance of the ChatGPT (Generative Pretrained Transformer-4) on the Certification Test for Mental Health Management: Factorial design
Scientific Title:Acronym
Performance of the ChatGPT (Generative Pretrained Transformer-4) on the Certification Test for Mental Health Management
Region
| Japan |
Condition
Condition
NA
Classification by specialty
| Not applicable |
Classification by malignancy
Others
Genomic information
NO
Objectives
Narrative objectives1
The present study aims to examine the performance of large language models (LLMs) on an examination that requires knowledge of occupational mental health. In addition, the study investigates whether tuned prompts can improve LLM performance. The study adopts ChatGPT (Generative Pretrained Transformer-4, GPT-4) as the LLM, and adopts the Certification Test for Mental Health Management organized by the Osaka Chamber of Commerce and Industry as the examination. We set two prompt conditions as factorial design: simple texts of questions (simple prompt condition), and texts with techniques to get better results (tuned prompt condition). The scores of four sets of examinations are compared between these two conditions.
Basic objectives2
Efficacy
Basic objectives -Others
Trial characteristics_1
Exploratory
Trial characteristics_2
Pragmatic
Developmental phase
Not applicable
Assessment
Primary outcomes
Scores for multiple-choice and essay questions of the Certification Test for Mental Health Management
Key secondary outcomes
Base
Study type
Interventional
Study design
Basic design
Factorial
Randomization
Non-randomized
Randomization unit
Blinding
Single blind -investigator(s) and assessor(s) are blinded
Control
Active
Stratification
Dynamic allocation
Institution consideration
Blocking
Concealment
Intervention
No. of arms
2
Purpose of intervention
Educational,Counseling,Training
Type of intervention
| Other |
Interventions/Control_1
Tuned prompt condition
In this condition, prompts to the LLM are developed by using techniques to get better answers to the examination, which were reported in prompt engineering. The techniques include constraint, indicating the format of the output, few-shot prompting, and chain of thought.
Interventions/Control_2
Simple prompt condition
In this condition, prompts consist simply of the plain texts of questions in the examination.
Interventions/Control_3
Interventions/Control_4
Interventions/Control_5
Interventions/Control_6
Interventions/Control_7
Interventions/Control_8
Interventions/Control_9
Interventions/Control_10
Eligibility
Age-lower limit
| Not applicable |
Age-upper limit
| Not applicable |
Gender
Male and Female
Key inclusion criteria
Multiple-choice questions for the Type I, Type II, and Type III exams and essay questions for the Type I exam of the Certification Test for Mental Health Management, by the Osaka Chamber of Commerce and Industry, conducted from 2018 to 2023.
A total of four exams from each type will be included in the study: Type I exams from the 27th, 29th, 31st, and 33rd; Type II and Type III exams from the 31st to 34th. Each exam has 50 multiple-choice questions. A scoring committee grades essay questions.
Key exclusion criteria
N/A
Target sample size
204
Research contact person
Name of lead principal investigator
| 1st name | Norito |
| Middle name | |
| Last name | Kawakami |
Organization
Graduate School of Medicine, The University of Tokyo
Division name
Department of Digital Mental Health
Zip code
113-8655
Address
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan
TEL
+81-3-5800-9621
norito@m.u-tokyo.ac.jp
Public contact
Name of contact person
| 1st name | Mako |
| Middle name | |
| Last name | Iida |
Organization
Graduate School of Medicine, The University of Tokyo
Division name
Department of Mental Health
Zip code
113-8655
Address
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan
TEL
+81-3-5800-9621
Homepage URL
idamako@m.u-tokyo.ac.jp
Sponsor or person
Institute
Department of Digital Mental Health, Graduate School of Medicine, The University of Tokyo
Institute
Department
Personal name
Funding Source
Organization
Department of Digital Mental Health, Graduate School of Medicine, The University of Tokyo
Organization
Division
Category of Funding Organization
Self funding
Nationality of Funding Organization
Other related organizations
Co-sponsor
Name of secondary funder(s)
IRB Contact (For public release)
Organization
Office for Human Research Studies(OHRS) Graduate School of Medicine and Faculty of Medicine, The University of Tokyo Faculty of Medicine
Address
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033
Tel
03-5841-0818
ethics@m.u-tokyo.ac.jp
Secondary IDs
Secondary IDs
NO
Study ID_1
Org. issuing International ID_1
Study ID_2
Org. issuing International ID_2
IND to MHLW
Institutions
Institutions
Other administrative information
Date of disclosure of the study information
| 2024 | Year | 02 | Month | 09 | Day |
Related information
URL releasing protocol
Publication of results
Unpublished
Result
URL related to results and publications
Number of participants that the trial has enrolled
Results
Results date posted
Results Delayed
Results Delay Reason
Date of the first journal publication of results
Baseline Characteristics
Participant flow
Adverse events
Outcome measures
Plan to share IPD
IPD sharing Plan description
Progress
Recruitment status
No longer recruiting
Date of protocol fixation
| 2024 | Year | 01 | Month | 29 | Day |
Date of IRB
| 2024 | Year | 01 | Month | 29 | Day |
Anticipated trial start date
| 2024 | Year | 01 | Month | 29 | Day |
Last follow-up date
| 2024 | Year | 03 | Month | 31 | Day |
Date of closure to data entry
Date trial data considered complete
Date analysis concluded
Other
Other related information
Approval by the ethics committee is not required because this study will be conducted using generative AI.
I entered the date of protocol fixation as the date of the IRB.
Management information
Registered date
| 2024 | Year | 02 | Month | 09 | Day |
Last modified on
| 2024 | Year | 03 | Month | 13 | Day |
Link to view the page
Value
https://center6.umin.ac.jp/cgi-open-bin/icdr_e/ctr_view.cgi?recptno=R000061116
| Research Plan | |
|---|---|
| Registered date | File name |
| Research case data specifications | |
|---|---|
| Registered date | File name |
| Research case data | |
|---|---|
| Registered date | File name |
