KIKu

The illegal trade in ancient cultural artifacts and archaeological objects is a global problem. Criminals loot archaeological sites or steal exhibits from museums and sell them. Police and customs officials rely on tools that allow them to quickly and reliably verify the authenticity of archaeological objects. Researchers and scientists at Fraunhofer SIT are therefore developing an IT application as part of the KIKu project that helps identify illegally traded cultural artifacts and can provide indications as to whether an object originates from an illegal excavation. The app also provides an initial assessment of the country of origin from which the cultural artifact might come.

How the KIKu App Works

In KIKu, Fraunhofer experts in IT forensics are collaborating with cosee, a Darmstadt-based company specializing in digital product development, to create a prototype app for identifying cultural artifacts. Investigators can use the app on their smartphones to photograph the cultural artifact under examination from multiple angles. The KIKu app sends the images to a deep learning network, which analyzes the photos. The network then sends an initial assessment back to the KIKu app on the investigator’s smartphone to assist in evaluating the cultural artifact. A key factor in the app’s success will be supporting users in taking photos so that they are usable by the deep learning network (sufficiently lit, appropriate perspective, etc.). In this context, the company cosee contributes its experience in developing user-centered (mobile) applications.

How is the deep learning network trained?

The experts and scientists at Fraunhofer SIT are working with data from the Prussian Cultural Heritage Foundation (SPK), which comprises several thousand data sets on ancient cultural artifacts and archaeological objects, along with corresponding assessments by archaeologists and art history experts. This dataset is relatively small for training a neural network. The scientists and experts at Fraunhofer SIT are therefore using transfer learning in the KIKu project. With this method, they can take an already trained network (such as one for general image recognition or a similarly suitable network) and train it more specifically using the available data, optimizing it for the image analysis of cultural heritage objects. This enables the network to recognize 3D cultural artifacts such as vases, statues, or similar objects, and not just two-dimensional paintings. According to the experts, even a few hundred data sets can achieve significant improvements here.

Technical Support for Police and Customs

The deep learning network, trained in this specific way, is now sent photos of an object from various perspectives via the KIKu app. The deep learning network then determines whether the photos resemble an object in the database or not. If it finds similar objects, it selects the corresponding data sets and sends them to the KIKu app, along with information compiled by art experts and archaeologists, such as the object’s region of origin and the era from which it dates. This gives investigators an initial indication of whether the object they wish to examine was acquired legally or comes from an illegal excavation and may not be traded.

In other words: The network is designed to apply the expert knowledge—which has been incorporated into the Prussian Cultural Heritage Foundation’s database during the assessment and description of cultural objects—to current cases currently under investigation. In this way, the network can assist investigative authorities in identifying even as-yet-unknown objects that originate from illegal excavations (and for which, naturally, no images are yet available).

In a second step, the neural network compares the photos from the KIKu app with a police database containing looted art treasures and stolen cultural property. If there is a match, the app immediately sounds the alarm.

Follow-up project to ILLICID

KIKu is funded by the Federal Government Commissioner for Culture and the Media (BKM) with 250,000 euros from the federal government’s national AI strategy and will run until the end of 2021. It is a follow-up project to the BMBF-funded project "ILLICID – Illegal Trade in Cultural Property in Germany." ILLICID was launched in 2015 as part of the “Research for Civil Security II” program of the Federal Ministry of Education and Research (BMBF). It is a transdisciplinary research project on the illegal trade in cultural property.