Artificial Intelligence & Security

Analysis of Disinformation, Manipulation, and Deepfakes

Today, misinformation spreads at breakneck speed via social networks, messaging services, and online platforms. Fraunhofer SIT is therefore developing methods to better understand and combat disinformation. We develop tools that reveal underlying connections and investigate the patterns of disinformation spread as well as the roles of the actors involved in social networks. In addition, we develop prototypes and R&D solutions for detecting manipulation in image, audio, and video files, as well as for the automated detection of deepfakes. 

Through our research in this area, we support government agencies, media outlets, and platform operators in identifying and containing disinformation campaigns at an early stage. At the same time, we strengthen the integrity of digital information and protect public order and a free and democratic society. 

Security analyses of AI-generated source code and the use of AI for code analysis

More and more developers are using AI assistants to write software; in essence, coding with AI has become the new norm. While this speeds up the work, it also creates new security vulnerabilities: Automatically generated code often contains weaknesses that remain undetected without targeted testing (technical debt).

The software experts at Fraunhofer SIT are investigating how such vulnerabilities can be systematically detected and how AI-supported analysis tools can improve code security testing. The goal is, for example, to enhance traditional scanners, enable detection across various platforms, and make security testing more efficient and precise—so that the speed advantage of AI-generated code does not come at the expense of IT security.

Securing AI Systems Against Manipulation and Misuse

Language models and other AI systems are not only used as tools but are also targets of attack themselves. For example, anyone who uses a large language model in a corporate product must expect that attackers will attempt to manipulate the system through targeted inputs: They can inject hidden instructions into documents, bypass built-in security mechanisms, or trick the model into revealing confidential information. Such attacks are difficult to detect and even harder to systematically defend against. 

Researchers at Fraunhofer SIT are analyzing the vulnerabilities in AI systems and exploring how effective countermeasures can be developed without limiting the capabilities of the language model. With these findings, we support companies and government agencies in using AI systems securely and in compliance with the law.

Using AI for Security Engineering

Traditional security scanners deliver verifiable results with a deliberately controllable level of accuracy and completeness within a precisely definable framework. At the same time, they quickly reach their limits when dealing with complex, context-dependent vulnerabilities such as logical errors. Artificial intelligence, on the other hand, can capture semantic relationships in the code and detect even subtle or novel vulnerability patterns—though without the reliability of traditional methods. At Fraunhofer SIT, we are therefore researching how both approaches can be combined: the precision of traditional scanners is combined with the semantic analysis capabilities of AI models to detect vulnerabilities more reliably and comprehensively.

Another focus is the perspective of offensive security: AI systems can also be specifically deployed as attack tools, for example, to automatically search for vulnerabilities or to generate exploits. Understanding these capabilities is essential for developing effective defense strategies. Researchers at Fraunhofer SIT are therefore investigating both the defensive and offensive use of AI in security engineering to provide companies and government agencies with a realistic picture of the current threat landscape as well as practical countermeasures.

Automated CSAM Detection

Our researchers are developing methods for detecting depictions of sexual violence against children (Child Sexual Abuse Material, CSAM) in real-world scenarios. A particular challenge lies in developing high-performance AI models without training them on illegal CSAM. Instead, multiple models are trained separately using legal image data and then merged into a single comprehensive model. In collaboration with government agencies, our models are evaluated in multiple iterations using real-world case data, without this data (or any other CSAM) being incorporated into the training process. This makes it relatively straightforward to share and deploy the models, since—unlike many AI solutions currently used by government agencies—they were never trained on case data at any point. Even if training data could be reconstructed from a model, there would be no risk of exposing illegal material. Another advantage of the models is that they make it possible to understand how their decisions are reached. This increases their transparency and makes it easier to verify the results.

Data Protection-Compliant AI Use

Anyone who trains or operates AI systems with sensitive data faces a fundamental dilemma: How can insights be gained without revealing the underlying data? In close collaboration with TU Darmstadt, we are conducting research on cryptographic methods that enable computations to be performed on encrypted data—so that an AI model can learn without ever gaining access to the raw data (Fully Homomorphic Encryption). These techniques are crucial for medical applications, financial services, and anywhere else where confidentiality and analysis are both required.

A particular challenge arises when the training data contains personally identifiable information, which means its processing is subject to the provisions of the General Data Protection Regulation (GDPR). However, due to the way deep learning models work, it is difficult in practice to uphold the rights of data subjects. For this reason, researchers at SIT are working on the anonymization of texts—that is, the removal of personally identifiable information—so that individual persons can no longer be identified. This will enable, for example, text data from government sources to be used for AI training in a manner that complies with the GDPR.

Real-World Lab for High-Security Applications

Fraunhofer SIT also collaborates with high-performance computing centers such as the Green IT Cube at the Society for Heavy Ion Research (GSI). This infrastructure enables the analysis of highly sensitive data in a secure real-world laboratory and supports projects in both research and industry. The combination of computing power, AI expertise, and real-world testing environments creates a unique ecosystem for innovation and security.