Cross-Domain Zero-Shot Performance of Small Language Models for Knowledge Extraction Tasks

Abstract

This paper presents a comprehensive evaluation of small language
models (up to 500M parameters) fine-tuned for Named Entity Recog-
nition (NER) on the CoNLL dataset. We examine the performance
of publicly available models obtained from Hugging Face in a cross-
domain, zero-shot setting across three tasks: NER, Named Entity
Linking (NEL), and Relation Extraction (RE). Our experiments en-
able an assessment of their suitability as ready-to-use components
in natural language processing (NLP) pipelines and provide insights
into their robustness and generalization capabilities across tasks
and domains.
All models were integrated into a standardized entity linking and
extraction pipeline that employs a consistent evaluation algorithm.
Our experiments reveal substantial variations in cross-domain NER
performance. For NEL, linking accuracy was highly sensitive to
domain shifts, while for RE, the choice of integration algorithm
significantly affected overall performance, resulting in comparable
outcomes across models. These findings highlight the continued
usefulness and relevance of smaller Transformer models for special-
ized knowledge extraction tasks and emphasize the importance of
advances in representation learning to enhance their generalization
and robustness.