A Framework of Cross-Modal Learning for Solving Geometry Problems

This paper presents a framework of deep cross-modal learning networks for solving geometry problems. Existing geometry solvers either focus on single-modal problems or multi-modal problems, and they cannot fit each other. In this paper, we propose a cross-modal learning architecture for solving geom...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of IEEE International Conference on Teaching, Assessment, and Learning for Engineering (Online) pp. 506 - 512
Main Authors: Guo, Fucheng, Jian, Pengpeng, Wang, Yanli, Wang, Qingjiang
Format: Conference Proceeding
Language:English
Published: IEEE 05.12.2021
Subjects:
Automatic solver
Cognition
cross-modal learning
Education
Feature extraction
Geometry
geometry problems
Head
shared encoder-decoder
Transformers
Transforms
ISSN:2470-6698
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper presents a framework of deep cross-modal learning networks for solving geometry problems. Existing geometry solvers either focus on single-modal problems or multi-modal problems, and they cannot fit each other. In this paper, we propose a cross-modal learning architecture for solving geometry problems, which can effectively solve both text and image-text geometry problems. To attack the representation of cross-modal features is a key challenge in understanding geometry problems that this paper adopts a shared encoder, in which the text and (or) image features are masked by using self-attention units and a multi-layer transformer is used to realize the interaction between cross-modal features. Further, we adopt a shared decoder to decode the single-modal features or the series sequence of multi-modal features according to the input of the encoder is single-modal or multi-modal problems. The representation of the decoder is transferred to task-specific heads for geometry relations extraction, theorems reasoning and geometry problem solving. The proposed algorithm can produce the solutions of geometry problems in a readable way. The experimental results shown that the proposed algorithm has good property and performance in solving geometry problems.
ISSN:2470-6698
DOI:10.1109/TALE52509.2021.9678945