The document describes a project called OPATCH that aims to create an advanced online search infrastructure for a historical newspaper archive. OPATCH will use computational linguistic methods like parsing, tagging, and named entity recognition to correct errors from optical character recognition (OCR) processing on the newspapers, which are from 1910-1920 and in difficult-to-read Fraktur font. The project will start with error-prone OCR text that cannot be manually corrected at scale. It will develop and test a method to generate and select candidates for correcting OCR errors using edit distances and ngram frequencies.

1. Correcting OCR results with
computational linguistic
methods
Michel Généreux, EURAC
Bozen / Bolzano
in collaboration with Egon W. Stemle, Lionel Nicolas, Verena Lyding
and Katalin Szabò
1
Institute for Specialised Communication and Multilingualism
Michel Généreux 27.10.2014

2. The OPATCH project (Open Platform for Access to and Analysis of Textual Documents of
Cultural Heritage) aims at creating an advanced on-line search infrastructure for research
in an historical newspapers archive.
• Duration: 24 months (Jan 2014 – Dec 2015)
• Fundings: Autonome Provinz Bozen-Südtirol, Landesgesetz Nr. 14, „Forschung und Innovation“
• Partners:
• Landesbibliothek Dr. Friedrich Teßmann, Bozen
• Institut für Corpuslinguistik und Texttechnologie (ICLTT), Wien
For implementing this, OPATCH builds on computational linguistic (CL) methods for structural
parsing, word class tagging and named entity recognition.
Dating between 1910 and 1920, the newspapers are typed in the blackletter Fraktur font and
paper quality is derogated due to age.
Hence, in OPATCH we are starting from majorly error-prone OCR-ed text, in quantities that
cannot realistically be corrected manually.
2
Institute for Specialised Communication and Multilingualism
Introduction
Michel Généreux 27.10.2014

3. A Glance at the Teßmann collection
626,287 pages for a total of 819,310,354 tokens;
616,751,127 tokens are in the reference dictionary, so a degree of cleanness
Our post-OCR correction system is based on 10 OCR-ed pages with cleanness
Given that the dictionary covers on average 91% of all words, 0.90 cleanness
Focusing on the best pages from the Teßmann collection ...
3
Institute for Specialised Communication and Multilingualism
of 0.75
of 0.70
is almost perfect OCR.
Michel Généreux 27.10.2014

4. Cumul.
number of
PM_1900 1032 1137449 0.901 1032 1137449 1024804 90.1%
AM_1890 8 9457 0.899 1040 1146906 1033302 90.1%
FA_1900 8 8652 0.887 1048 1155558 1040978 90.1%
PM_1910 737 750131 0.886 1785 1905689 1705286 89.5%
IS_1900 2970 2817010 0.879 4755 4722699 4180623 88.5%
WB_1900 72 45481 0.872 4827 4768180 4220278 88.5%
SVB_1890 9749 14306473 0.809 70546 92484068 75861319 82.0%
17545 26314923 0.807 88091 118798991 97103539 81.7%
16015 12153784 0.806 104106 130952775 106894635 81.6%
4
Institute for Specialised Communication and Multilingualism
OPATCH 100k pages
Newsp_Decade Number
of pages
Number of
tokens
Number of
tokens in
dictionary
pages
Cumul.
number of
tokens
Cumul.
number of
tokens in dict.
SVB_1900
Volksblatt
BZN_1890
Bozner
Nachrichten
Michel Généreux 27.10.2014

5. OPATCH unnanotated 5k pages
We select the 5000 cleanest pages from the Teßmann collection above in the years 1910-1920. The average
cleanness for this corpus is 91%, so no need for cleaning. This results in the following un-annotated example
corpus:
5
Institute for Specialised Communication and Multilingualism
Michel Généreux 27.10.2014

6. OPATCH annotated 5k pages
Automated annotations for Part-of-Speech (POS), Lemmas and Named Entities (NE). We also have a list of
roughly 31k locations and names for South Tyrol compiled by Teßmann.
POS LEMMA NE
Bozen NE Bozen I-LOC
hat VAFIN haben
ein ART eine
Museum NN Museum
für APPR für
Ötzi NE Ötzi I-PER
6
Institute for Specialised Communication and Multilingualism
Bozen hat ein Museum für Ötzi.
Michel Généreux 27.10.2014

7. Corpora
Ten OCR-ed pages with their manually corrected versions.
• 10,468 tokens and 3,621 types. Eight pages (8,324/2,487) are
used as training data and two pages (2,144/1,134) for testing.
• More than one out of two tokens is misrecognized, among
which almost half (48%) need a minimum of three edit
operations for correction.
Reference corpus: 5M words and 5M bigrams
• From the WEB and also http://www.gutenberg.org/ Romane
• The dictionary covers 91% of all words in the ten OCR-ed pages
7
Institute for Specialised Communication and Multilingualism
und Erzählungen (1910-20)
Michel Généreux 27.10.2014

8. Approach for OCR-ed correction
1. Probability models. Collate and tally all edit-operations (delete, insert and
replace) needed to transform all unrecognized tokens from the training
OCR-ed texts to its corrected form in the Gold Standard: n|u 98
* we obtain two probability models: constrained and unconstrained
2. Candidate generation is achieved by finding the closest entry in the
dictionary by applying the minimum number of edit-operations to an
unrecognized OCR-ed token. The number of candidates is function of the
maximum number of edit-operations allowed and the model used.
* wundestc → wundesten: ’c’ → ’e’ and inserting a ’n’ after the ’e’.
3. Selection of the most suitable candidate, given relative frequency and
... word word word word word WORD word word word word ...
8
Institute for Specialised Communication and Multilingualism
context:
Michel Généreux 27.10.2014

9. Experiment 1
Artificially created errors
●To achieve this we extracted random trigrams from the GS (left context, target, right context) and
applied, in reverse, the edit error model.
●Errors were introduced up to a limit of two per target and contexts.
●At the end of this process, we have two context words and five candidates, including the target.
●En is the maximum edit-operations performed to generate candidates.
9
Institute for Specialised Communication and Multilingualism
Michel Généreux 27.10.2014

10. Experiment 2: Real errors
10
Institute for Specialised Communication and Multilingualism
Michel Généreux 27.10.2014

11. The approach we presented to correct OCR errors considered four features of two types: edit-distance
Results showed that a simple scoring system can correct with very high accuracy OCR-ed texts
under idealized conditions: no more than two edit operations and a wide-coverage dictionary.
Obviously, these conditions do not always hold in practice, thus an observed accu-racy
drops to 10%. Wrong substitions by the OCR process have also been neglected.
Nevertheless, we can expect to improve our dictionary coverage so that very noisy OCR-ed texts
(i.e. 48% error with distance of at least three to target) can be corrected with accuracies up to
20%.
OCR-ed texts with less challenging error patterns can be corrected with accuracies up to 61%
(distance two) and 86% (distance one).
Reference: Michel Généreux, Egon W. Stemle, Verena Lyding and Lionel Nicolas. Correcting OCR
errors for German in Fraktur font. Pisa, 9-10 dicembre 2014. La prima Conferenza di Linguistica
Computazionale Italiana.
11
Institute for Specialised Communication and Multilingualism
Discussion
and n-grams frequencies.
Michel Généreux 27.10.2014