SEMANTIC NOVELTY DETECTION IN SCIENTIFIC DOCUMENTS USING TRANSFORMER EMBEDDINGS

The rapid growth of digital document collections has increased the need for effective methods to identify novel information. This paper presents a semantic novelty detection framework based on text preprocessing, transformer-based embeddings, and cosine similarity. Documents are converted into dense vector representations, and sentence-level semantic comparison is performed to identify novel information within semantically similar documents. The proposed approach enables the automatic detection of both repeated and newly emerging content in large-scale document collections while preserving semantic relationships beyond lexical matching. Experimental results demonstrate that the framework effectively distinguishes novel information from semantically similar documents, highlighting its potential for applications in information retrieval, document management, and intelligent text analysis.

Hali tarjima qilinmagan