Increasing the Reliability of Information Based on the Statistical Information Redundancy of Texts
Аннотация
The problem has been formulated and the methodology for increasing the reliability of information in automated office work systems has been developed based on the use of statistical information redundancy, properties and characteristics of elements of electronic document texts. The rules have been formalized and the existing algorithms for checking distorted information have been modified based on a set of standards, keywords, phrases, a dictionary of word forms, frequency dictionaries, as well as the use of morphological analysis and n-grams of structured natural language models. A technique for estimating conditional probabilities of one-, two-, three-, -fold errors in text information has been developed. Tools for checking the reliability of texts by clustering words, as well as checking by operators of one- and two-way line viewing, have been implemented. A technique for increasing reliability based on -grams of text spelling control with adaptable boundaries, variables and intervals of the membership function has been proposed. A software package for increasing the reliability of information in C++ has been implemented and tested, synthesizing the tools of the SPHINX-4 Framework and CUDA parallel computing in its structure.