How Multimodal Is Generative AI Research in Education?

Generative AI research in education is growing rapidly.

Students use conversational AI systems such as ChatGPT to write essays, check problem-solving processes and generate programming code. Teachers use AI to create lesson materials, design assessment questions and provide personalized feedback.

But generative AI does not simply mean large language models.

Generative AI can create or transform many types of information: text, images, speech, music, video, data, tables, graphs and more. Text-to-speech, speech-to-text, text-to-image and text-to-video systems can all be used in education.

The question raised by Ville Heilala, Roberto Araya and Raija Hämäläinen is simple:

How evenly does educational research on generative AI address these different modalities?

Their paper, Beyond Text-to-Text: An Overview of Multimodal and Generative Artificial Intelligence for Education Using Topic Modeling, presented at the 40th ACM/SIGAPP Symposium on Applied Computing, does not test the effectiveness of a single AI tool. Instead, it maps the broader research landscape.

The study asks which technologies and educational themes dominate the field, and which remain at the margins.

Its answer is clear.

Generative AI in education may be technologically multimodal.

But the research literature is still overwhelmingly text-centered.

Stepping Back From the Product Name “ChatGPT”

One important feature of the study is that the researchers did not use specific product names such as ChatGPT, Gemini or Claude in their literature search.

They were concerned that the market influence of particular companies and services might distort the results. Instead, the search strategy included technical concepts and modality relationships such as “generative AI,” “large language model,” “text-to-text,” “text-to-video” and “text-to-audio.”

The search also reflected combinations of different information forms, including text, video, audio, speech, music, data, image, graphs, tables, emotion, language and gesture.

This method was designed to avoid reducing educational generative AI research to “studies about ChatGPT.” The goal was broader: to understand how AI systems that generate and transform different forms of information are being studied in education.

That decision matters.

It allows the study to examine whether the field is truly multimodal, or whether the term “generative AI” is mostly being used as a proxy for text-based interaction with large language models.

From 4,475 Records to 4,175 Research Materials

The researchers collected literature from Dimensions.ai in August 2024 and supplemented the search using Scopus and Web of Science.

The initial search returned 4,475 records. After removing items without DOIs or abstracts, the final dataset included 4,175 research materials.

The dataset included journal articles, conference papers and book chapters. About 95% of the materials were published after 2014. However, the oldest study dated back to 1983 and focused on a text-to-speech reading aid for blind students.

That historical detail is important.

It shows that multimodal educational technology did not suddenly appear with the recent generative AI boom. Technologies such as text-to-speech and speech recognition have long been studied in special education and assistive technology.

Recent generative AI should therefore be understood not as a completely new field, but as an acceleration and recombination of older modality-conversion technologies with large-scale models and generative systems.

Letting the Data Form the Topics

The researchers analyzed the abstracts of 4,175 papers using BERTopic.

BERTopic is a topic-modeling method that does not merely count word frequency. It converts text into semantic embeddings, groups similar documents and then identifies representative terms for each topic.

The researchers first used the all-mpnet-base-v2 sentence-transformer model to convert each abstract into a 768-dimensional vector. They then used UMAP for dimensionality reduction and HDBSCAN for clustering similar studies. Finally, c-TF-IDF was used to extract representative words for each cluster, which the researchers interpreted as topics.

A notable methodological choice was that the researchers did not predefine the number of topics.

Instead, they ran 5,000 analyses using random combinations of key UMAP and HDBSCAN settings. They evaluated each result based on DBCV, the number of topics and the number of papers that were not assigned to any topic.

This process produced 54 initial topics, covering about 85% of the papers. The remaining 15% were not forced into existing categories.

The researchers then merged eight similar topics and excluded eight topics that were difficult to interpret. The final result was 38 topics, covering about 74% of the original dataset.

The aim was not to classify every paper at all costs.

It was to produce interpretable and coherent topic clusters.

LLMs 3,645 Times, ChatGPT 1,471 Times: The Text Bias Is Overwhelming

The first major finding was the overwhelming influence of large language models.

When the researchers analyzed whether major keywords appeared in paper titles or abstracts, expressions related to “large language model” appeared 3,645 times. “Generative artificial intelligence” appeared 2,177 times. “ChatGPT” appeared 1,471 times.

This is striking because ChatGPT was not included as a search term.

Nevertheless, ChatGPT appeared much more frequently in the collected abstracts than Gemini, Bard, Copilot, Claude or Perplexity.ai. This suggests that, in practice, ChatGPT functions as the representative research object in educational generative AI.

By contrast, the most common non-text modality, text-to-speech, appeared 461 times.

That is a significant gap.

The result suggests that what is often called “generative AI research in education” is, in practice, very close to “large language model research in education.”

Generative AI may be capable of producing many forms of information.

But educational research has mostly focused on interactions where users type questions and receive written responses.

Speech and Images Are Possible, but Still Peripheral

After large language models, one of the most studied technologies was text-to-speech.

Text-to-speech can help students by reading written text aloud. It can support reading comprehension and spelling, and it can improve accessibility for blind students or learners with reading difficulties.

Speech-to-text also has educational value. It can help students who struggle with writing express ideas orally and convert them into text. This is especially relevant for special education and learning support.

Text-to-image systems have been used for generating design ideas, visualizing structural engineering concepts and creating images for medical education. Image-to-image systems have been used to visualize complex molecular structures more realistically. Text-to-video research has explored educational content creation and personalized learning materials.

However, these studies remain much fewer than large language model studies and often focus on specific use cases.

Although generative AI can connect multiple senses and modes of expression, educational research often treats each technology as a separate tool.

The researchers found very limited work in which multiple modalities were integrated to automatically generate classroom materials.

The term “multimodal AI” is used often.

But studies in which text, speech, image and video are organically connected inside one learning experience remain at an early stage.

What the 38 Topics Reveal About the Field

The final 38 topics were organized into 14 broader thematic areas:

educational fields and subjects;

theoretical and philosophical foundations;

personalized learning support;

problem solving;

technology acceptance;

teacher professional development;

creativity;

serious games;

learning tools and content;

assessment;

ethics and security;

academic integrity;

chatbots;

and language learning.

This map shows that generative AI is not limited to one educational activity. Research is appearing across lesson-material creation, personalized feedback, problem solving, assessment, teacher education, creativity, foreign-language learning, game creation and research support.

The largest topic was large language models in medical education, with 654 papers.

The second largest topic was text-to-speech for learning support, with 546 papers.

Other major topics included generative AI in computing education, writing support and feedback, ethical and security issues, and educational chatbots.

In medical education, many studies examined whether models could simplify patient education materials or answer medical questions accurately. In computing education, the focus was on code generation, programming-assignment feedback and support for novice learners.

In mathematics and physics, researchers studied problem-solving ability, explanation accuracy and simulation use. In chemistry education, prompt-writing strategies and molecular or chemical concept learning were major areas of interest.

Generative AI is spreading across education.

But it is not spreading evenly across all subjects and school levels.

Higher Education Moved Fast, but K-12 Is Less Visible

At the topic level, higher education was the only school level that emerged as an independent area.

There was a separate topic for generative AI in higher education. By contrast, primary and secondary education did not form large, independent topic clusters.

This suggests that research on generative AI has expanded quickly around universities, while K-12 research remains comparatively limited.

The difference is not only about the number of studies.

In higher education, it is relatively easy to observe students using generative AI to write papers, generate code or search for research material. In K-12 education, researchers must consider developmental stages, curricula, privacy, teacher control, parental consent and the reliability of AI outputs.

Therefore, K-12 generative AI adoption cannot simply copy university models at a smaller scale.

It requires age-appropriate instructional design and stronger safety structures.

Personalized Learning Is Strong, but the Meaning of “Personalized” Is Still Narrow

Personalized learning support was a relatively large area in the research landscape.

It included text-to-speech, emotion analysis, writing support, feedback generation and intelligent tutoring systems.

Generative AI can adjust explanations based on student responses, rephrase wrong answers and provide feedback matched to learning pace. Its natural-language interaction is more flexible than many earlier adaptive learning systems.

But personalization should not mean merely generating different sentences for different students.

If a model does not understand a learner’s prior knowledge, misconceptions, goals, motivation and cognitive load, its responses may be linguistically different but educationally shallow.

Multimodal AI can expand the meaning of personalization.

A student with reading difficulties may receive spoken explanations.

A student struggling with abstract concepts may receive images or simulations.

A student with language-expression difficulties may respond through speech, gestures or drawings.

The core of personalization is not changing the wording of an answer.

It is allowing students to access and express knowledge through the forms of information they can understand and use.

Does Generative AI Support Creativity or Replace It?

Creativity, art and design also appeared as independent research topics.

Text-to-image systems can help students explore ideas, compare design alternatives and quickly see visual possibilities that would otherwise be difficult to imagine. In fashion and textile design education, researchers have examined idea generation and sustainable design exploration using generative AI. In art education, studies have analyzed how students and AI collaborate to produce digital artworks.

But as generation becomes faster, the boundary between creation and selection becomes less clear.

Students may not draw an image by hand, but they may design prompts, compare outputs, choose results and guide revisions. In that case, creativity shifts from manual execution to exploration, comparison and judgment.

At the same time, if students submit generated images without showing their process, the learning becomes invisible.

It becomes difficult to know whether AI supported creativity or replaced the student’s thinking.

That is why creative education with multimodal generative AI should evaluate more than final products.

Teachers should examine changes in prompts, comparison among generated outputs, selection criteria, revision processes and ethical judgment.

The important question is not only what the AI generated.

It is how the student collaborated with the AI.

The Tension Between Learning Support and Cheating

Assessment, academic integrity, ethics and security were closely connected in the research landscape.

Generative AI can evaluate student responses, generate multiple-choice questions and provide personalized feedback. It can reduce teachers’ workload and make immediate feedback possible.

At the same time, generative AI can write assignments, essays and programming code on behalf of students. As a result, research on detecting AI-generated writing and code also appeared as an independent topic.

The same technology can act as a learning tutor on one side and a tool for substituting student performance on the other.

A student who writes a draft and revises it after AI feedback is doing something very different from a student who submits an AI-generated final answer unchanged.

The key question is not simply whether AI was used.

It is how much the student participated in knowledge construction and judgment.

For that reason, assessment in the age of generative AI needs to move from final-product evaluation toward process evaluation.

Teachers may need to evaluate what questions students asked, how they reviewed AI responses, what they revised and what they rejected.

Education Needs Multimodal AI Trained for Learning

The researchers argue that education should move beyond simply importing general-purpose generative AI.

Models built for general users do not necessarily provide answers aligned with curricula, learning goals or students’ developmental levels.

An educational AI system should do more than generate the correct answer quickly.

It should diagnose misconceptions, give students time to think, ask questions aligned with learning goals and clearly indicate when evidence is weak.

The paper raises the need for models intentionally trained for educational requirements, such as EdGPT.

If such models are combined with multimodal functions — speech, image, video and simulation — students may not merely receive answers. They can compare, manipulate and reconstruct knowledge through different representations.

For example, a math system could explain a problem through text, equations, graphs, geometric transformations and spoken narration. In science, an AI system could show an experiment through video and simulation, understand a student’s spoken question and re-visualize the relevant part.

The educational value of multimodal AI is not that it shows more media.

It is that it helps learners understand and move between representations.

Teachers Need Pedagogical AI Imagination

The researchers suggest that teachers should directly explore technologies such as text-to-speech, speech-to-text and text-to-image.

They describe this as the formation of pedagogical AI imagination.

This does not simply mean learning how to use new features. It means imagining how a specific AI technology can redesign an existing learning activity.

Text-to-speech can be more than a function that reads a textbook aloud. Students can listen to their own writing and revise sentence flow.

Speech-to-text can be more than a dictation tool. It can connect speaking and writing.

Text-to-image can be more than a way to create decorative images. It can help students externalize and compare their conceptual understanding visually.

The educational effect of AI is not built into the technology itself.

It depends on when the tool is used, what choices students are given and how the generated output is reviewed.

Teacher education should therefore go beyond prompt-writing tips or service tutorials. It should include instructional design with AI, evaluation of reliability and bias, and protection of students’ thinking processes.

A Map of 4,175 Studies Still Does Not Show Everything

Topic modeling on a large literature set can show the broad shape of a research field.

But it cannot reveal every detail of instructional design or actual learning effects in individual studies.

The researchers acknowledge this limitation. Topic modeling provides a high-level landscape, but small or emerging themes may be hidden. Because the search did not include specific product names, some studies centered on particular services may have been missed.

Terminology also creates problems.

“Copilot” may refer to a general AI assistant, Microsoft Copilot or GitHub Copilot. “Perplexity” may refer to an AI search service or to a statistical measure in language modeling.

Another limitation is that the original search strategy did not include “teacher” and “teaching” as independent core search terms. Although teacher education and professional development appeared in the results, the overall search design centered more on students and schools. Future research may need separate analysis focused on how teachers understand and integrate AI into instruction.

Some modalities were also underrepresented, including text-to-sign language.

As multimodal AI continues to develop, research categories and search terms must also be updated.

Educational AI Must Move Beyond the Chat Window

The picture presented by this study is paradoxical.

Generative AI has advanced rapidly enough to create images, speech, video and music. Yet educational research remains strongly focused on text dialogue and writing generation.

Large language models clearly offer value in writing, feedback, problem solving, programming and assessment. But students do not learn only through text. They see, hear, speak, move and connect different representations as they form understanding.

Researching multimodal AI in education therefore does not mean simply adding an image generator or speech synthesizer.

It means deciding which form of information a learner needs and organizing text, speech, image, video and simulation into one coherent learning flow.

The next stage of generative AI in education may not depend only on introducing larger language models.

It may depend on connecting existing AI modalities to the characteristics of learners and subjects, while ensuring that students compare, verify and reconstruct generated information rather than passively consume it.

When educational AI moves beyond the chat window, it can move beyond writing answers for students.

It can become a learning environment that expands how students understand, express and build knowledge.