Automatic Dialect Detection for Low Resource Santali Language

Among different natural language processing (NLP) tasks, language detection is considered a difficult task especially for similar languages, varieties, and dialects. To build any tools or technologies in NLP, it is required reliable and robust language detection tool. Although such language detectio...

Full description

Saved in:
Bibliographic Details
Published in:2021 19th OITS International Conference on Information Technology (OCIT) pp. 234 - 238
Main Authors: Sahoo, Sunil Kumar, Mishra, Brojo Kishore, Parida, Shantipriya, Dash, Satya Ranjan, Besra, Jatindra Nath, Tello, Esau Villatoro
Format: Conference Proceeding
Language:English
Published: IEEE 01.12.2021
Subjects:
Buildings
Dialect Detection
Information technology
Language Detection
Natural language processing
Performance evaluation
Reliability
Supervised Autoencoder
Task analysis
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Among different natural language processing (NLP) tasks, language detection is considered a difficult task especially for similar languages, varieties, and dialects. To build any tools or technologies in NLP, it is required reliable and robust language detection tool. Although such language detection tools and technologies are available for many high-resource languages, it is difficult to find them for low-resource languages due to the lack of training datasets. In this work, we compiled a language detection dataset for the low resource language Santali. We considered the Sanatali written in Odia script and standard Odia script. We used a deep supervised autoencoder for language detection. The deep supervised autoencoder can detect both Odia and Santali languages although written using the same Odia script. We obtained an accuracy of 100% on the test set. Our proposed dataset will be enriched Santali language and help researchers to build tools and technologies for this low resource language.
DOI:10.1109/OCIT53463.2021.00055