Multimodal Knowledge Graphs for Industrial AI Assistance
Researchers at the Information Processing and Telecommunications Center (IPTC), Universidad Politécnica de Madrid (UPM), together with partners from Sopra Steria and D-Cube, have developed the Semantic Processing Engine (SPE), a modular AI-powered system that transforms complex technical manuals into structured, queryable Knowledge Graphs (KGs). Developed within the EU-funded Motivate XR project, SPE combines two synergistic components: the Semantic Textual Engine (STE), which extracts entities, procedures, and relationships from unstructured text using a quantized Mistral-7B model, and the Semantic Graphical Engine (SGE), which applies the vision-language model Pixtral-12B to interpret diagrams, tables, and figures, grounding them in their surrounding textual context.
The system was validated across three industrial case studies—aeronautics, home-appliance maintenance, and aluminum assembly—demonstrating near-linear scalability with document length, a low hallucination rate (3.33%), and consistently high graph connectivity. While the SGE’s visual processing accounts for roughly two-thirds of total processing time, this cost is incurred only once during offline document ingestion, enabling near-instantaneous query responses afterward. Compared to text-only and traditional NLP baselines, the multimodal approach achieved a 66% improvement in knowledge extraction density, confirming that bridging text and imagery is essential for capturing complete procedural knowledge.
Practical applications of this technology center on driving intelligent assistance and generating dynamic content for Extended Reality (XR) training frameworks. Specifically, the system automates the production of AR/VR work instructions for assembly and maintenance lines, thereby minimizing the manual effort traditional translation of static documentation demands. Ultimately, by establishing explainable and fully auditable knowledge graphs, the SPE offers a robust foundation for high-stakes industrial sectors looking to digitalize technical training and safeguard critical operational knowledge against a shrinking pool of senior experts
Bibliographic reference:
Del-Río, A., Ruiz, V., Petit, A., Colomines, L. , Choropanitis, P. & Serrano. J. A multimodal GraphRAG engine for semantic knowledge extraction in industrial environments In Expert Systems with Applications,art. 13344, 332, Part A. https://doi.org/10.1016/j.eswa.2026.133441
Alberto del Río Ponce: GS / ORCID / LinkedIn
Verónica Ruiz Bejarano: ORCID / LinkedIn
Javier Serrano Romero: ORCID / LinkedIn
Share this:
