Jump to

ATLAS

Voice control of generative artificial intelligences in an augmented reality environment

Fast facts

Organizational unit
Information technology
Contact person
Prof. Dr.-Ing. Karsten Lehn
Category
- Teaching project
Project purpose
Student project work

About the project

As part of their project work 1, students of the Bachelor's study program Digital Technologies have developed a voice control system for artificial intelligence in a...

What is it about?

As part of the first project, two students on the Bachelor's degree course in Digital Technologies developed the ATLAS application. This name stands for an "augmentation technology for linguistic analysis and control" and consists of an application for the HoloLens 2, an augmented reality headset from Microsoft. ATLAS enables its users to interact with the OpenAI interface for Generative Pretrained Transformer - and thus with generative artificial intelligence - by means of voice-based interaction.

In addition to general information output, the application can be used to launch various functions of the HoloLens 2 through spoken conversation. These functions include taking photos, generating images, visually displaying longer outputs and displaying the environment recognized by the HoloLens in the user's area.

How was it realized?

The application is based on the Unity game engine and the Mixed Reality Toolkit 2 (MRTK2) from Microsoft. With the help of MRTK, the output generated by OpenAI was visualized in an interactive augmented reality display. The transcription of the user's spoken input (conversion from speech to text) and the synthesizing of the output (conversion from text to speech) are also carried out by the language libraries within the MRTK. The C# programming language was used to implement the application.