Exploring Students’ Perceptions of ChatGPT: Thematic Analysis and Follow-Up Survey

ChatGPT has sparked both excitement and skepticism in education. To analyze its impact on teaching and learning it is crucial to understand how students perceive ChatGPT and assess its potential and challenges. Toward this, we conducted a two-stage study with senior students in a computer engineering program ( n=56 ). In the first stage, we asked the students to evaluate ChatGPT using their own words after they used it to complete one learning activity. The returned responses (3136 words) were analyzed by coding and theme building (36 codes and 15 themes). In the second stage, we used the derived codes and themes to create a 27-item questionnaire. The students responded to this questionnaire three weeks later after completing other activities with the help of ChatGPT. The results show that the students admire the capabilities of ChatGPT and find it interesting, motivating, and helpful for study and work. They find it easy to use and appreciate its human-like interface that provides well-structured responses and good explanations. However, many students feel that ChatGPT’s answers are not always accurate and most of them believe that it requires good background knowledge to work with since it does not replace human intelligence. So, most students think that ChatGPT needs to be improved but are optimistic that this will happen soon. When it comes to the negative impact of ChatGPT on learning, academic integrity, jobs, and life, the students are divided. We conclude that ChatGPT can and should be used for learning. However, students should be aware of its limitations. Educators should try using ChatGPT and guide students on effective prompting techniques and how to assess generated responses. The developers should improve their models to enhance the accuracy of given answers. The study provides insights into the capabilities and limitations of ChatGPT in education and informs future research and development.

View this article on IEEE Xplore

Published in the IEEE Education Society Section

Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models

The field of data visualisation has long aimed to devise solutions for generating visualisations directly from natural language text. Research in Natural Language Interfaces (NLIs) has contributed towards the development of such techniques. However, the implementation of workable NLIs has always been challenging due to the inherent ambiguity of natural language, as well as in consequence of unclear and poorly written user queries which pose problems for existing language models in discerning user intent. Instead of pursuing the usual path of developing new iterations of language models, this study uniquely proposes leveraging the advancements in pre-trained large language models (LLMs) such as ChatGPT and GPT-3 to convert free-form natural language directly into code for appropriate visualisations. This paper presents a novel system, Chat2VIS, which takes advantage of the capabilities of LLMs and demonstrates how, with effective prompt engineering, the complex problem of language understanding can be solved more efficiently, resulting in simpler and more accurate end-to-end solutions than prior approaches. Chat2VIS shows that LLMs together with the proposed prompts offer a reliable approach to rendering visualisations from natural language queries, even when queries are highly misspecified and underspecified. This solution also presents a significant reduction in costs for the development of NLI systems, while attaining greater visualisation inference abilities compared to traditional NLP approaches that use hand-crafted grammar rules and tailored models. This study also presents how LLM prompts can be constructed in a way that preserves data security and privacy while being generalisable to different datasets. This work compares the performance of GPT-3, Codex and ChatGPT across several case studies and contrasts the performances with prior studies.

View this article on IEEE Xplore

Code Generation Using Machine Learning: A Systematic Review

Recently, machine learning (ML) methods have been used to create powerful language models for a broad range of natural language processing tasks. An important subset of this field is that of generating code of programming languages for automatic software development. This review provides a broad and detailed overview of studies for code generation using ML. We selected 37 publications indexed in arXiv and IEEE Xplore databases that train ML models on programming language data to generate code. The three paradigms of code generation we identified in these studies are description-to-code, code-to-description, and code-to-code. The most popular applications that work in these paradigms were found to be code generation from natural language descriptions, documentation generation, and automatic program repair, respectively. The most frequently used ML models in these studies include recurrent neural networks, transformers, and convolutional neural networks. Other neural network architectures, as well as non-neural techniques, were also observed. In this review, we have summarized the applications, models, datasets, results, limitations, and future work of 37 publications. Additionally, we include discussions on topics general to the literature reviewed. This includes comparing different model types, comparing tokenizers, the volume and quality of data used, and methods for evaluating synthesized code. Furthermore, we provide three suggestions for future work for code generation using ML.

View this article on IEEE Xplore