SeGen
IT Security of Generative AI
Whether for vacation planning, drafting emails, or more complex work-related organizational issues and tasks, applications of generative artificial intelligence have become part of many people’s daily and professional lives. However, as AI becomes more widespread, new attack vectors are emerging – specifically targeting the AI models themselves. How secure is generative AI, and how can it be attacked? The SeGen (Security of Generative AI) project is getting to the bottom of these questions.
Fraunhofer SIT researchers are investigating various attack vectors targeting AI models and developing attack scenarios as well as countermeasures to better protect generative AI and thereby make its use in both private and corporate contexts safer.
In SeGen, Fraunhofer researchers are working with open-weight models such as Deepseek, Teuken, Mistral, Qwen, and Llama. These models offer many advantages that are of interest to both businesses and academia. These include, for example, digital sovereignty and transparency in the reproduction of results. The insights gained from open-weight models can later be applied to proprietary models.
What does “Open Weight” mean?
The weights of a language model are the weightings of the connections in the neural network that are learned during training. By adjusting these weightings during training, the model learns to provide increasingly better responses. “Open Weight” means that these trained weights are made publicly available.
How does LLM training work?
A large language model (LLM) consists of billions of artificial neurons. There are connections between these neurons. Each of these connections has a numerical value called a weight. These numbers determine how strongly information is passed on. At the beginning, the weights are random. The training algorithm then calculates errors and adjusts the weights to minimize these errors. This process repeats until the error can no longer be minimized. Once the weights have been adjusted, the model is fully trained. The weights contain all the learned knowledge and the patterns recognized in the training data.
At SeGen, researchers are investigating the following attack scenarios, among others:
Backdoor attacks
Attackers could take an open-weight model, alter the weights – for example, through manipulated adapters – and thus secretly introduce an unintended functionality (backdoor) into the model that is activated by a trigger.
For example, they could …
… intentionally train the model with false facts,
… favor certain products, or
… program a bias for or against certain groups of people.
People are unlikely to detect this malicious functionality, which makes this type of attack particularly dangerous.
Manipulated Adapters
Open-weight models can be deployed for specialized purposes with relatively little effort using so-called parameter-efficient adapters. For example, a company wants to adapt a Llama model for technical documentation. To do this, it leaves the original Llama model unchanged. An adapter is trained on technical texts. When the model is used, the base model and the adapter are loaded together. The result behaves like a specialized model, even though only a small add-on was trained. This offers the major advantage that little training hardware is required and adapters can be easily shared.
However, these adapters can be misused. If an adapter has been maliciously manipulated, the entire base model plus adapter can produce problematic results. In SeGen, researchers are investigating how to check adapters for manipulation. They have already developed a method to classify adapters. This allows users to easily determine whether an adapter has been manipulated.
Jailbreaks
Another type of attack involves disabling security mechanisms in AI models (jailbreaks). Through a process known as alignment, protective mechanisms are trained into the models to refuse to answer questions that involve illegal activities: For example, building a bomb, manufacturing drugs, and the like. Some neurons in the model’s network become safety neurons that are activated when such questions are asked and then refuse to provide an answer. As part of their research into the alignment process of AI models, the SeGen researchers were able to show that the underlying safety mechanisms are surprisingly fragile. They demonstrated that existing safeguards can be circumvented without compromising model performance. Based on these findings, they are now developing robust countermeasures to protect against such attacks.
About the Project
SeGen is a joint project of Fraunhofer SIT and the Dieter Schwarz Foundation based in Heilbronn. The project began in early June 2026 and will run through the end of June 2027.
