Post-Editing of Machine Translation: Developing Requirements and Compensation Schemes

Post-Editing of Machine
Translation (PEMT)
Developing Requirements and
Compensation Schemes

Overview
 MT 101
 Estimation of MT quality
 Definition of post-editing
 PEMT Requirements
 PEMT skills
 Pricing and compensation
© 2013 Luigi Muzii Developing PEMT Requirements and Compensation Schemes 2

Caveat emptor
 Your presenter is an MT enthusiast
 Using MT since 1990
 No bias
 MT is here to stay
 No special knowledge
 Could sound trivial
• Common sense

Methods
 Rule-based Machine Translation (RbMT)
 Transfer
 Interlingua
 Data-driven (stochastic) machine
translation
 Example-based Machine
Translation (EbMT)
 Statistical Machine Translation (SMT)
 Both methods can work for projects where
MT is suitable
 Most commercial systems are now all hybrids of some sort
 Post processing and cleanup

Rules-based
Machine Translation (RbMT)
 Analytical approach
 Grammatical representation of language
• Morphological analysis
– Inflection and conjugation of
words
• Syntactic analysis
– Sequence of words and sentence
structure
• Semantic analysis
– Meaning of words in context
 Heavy dependence on bilingual dictionaries

RbMT Issues
 Disambiguation
 Plain, correct, and consistent source
 Constant refinement of rules
 Accurate and comprehensive dictionaries

Example-based Machine
Translation (EbMT)
 Bilingual corpus
 Body of reference for similarities
• Combination of segments
– Best approximation
 Fuzzy matching algorithms

Statistical
Machine Translation (SMT)
 Empirical strategy
 Statistical probability
• Statistical assessment of words and phrase positioning
within segments from corpus
– Brute force computing

Two Models for Learning Data
 Translation model
 Words and word sequences in SL to find the most
likely corresponding words in TL
 Target-language model
 The most likely way in which corresponding TL
words will be combined

EbMT & SMT Issues
 Parallel corpora
 Analysis is challenging
• Problems with large corpora
 Translation memories instead
of corpora
 Segmentation for accuracy and
matching
• Problems with alignment

Ambiguity in SMT
 Translation models handle word sequences
 Likeliness of reproducing a wrong interpretation if
in model
 Collocations (dependencies between words)
could be hard to capture
 Target-language models

SWOT Analysis for MT
• Speed
• Volumes
• Consistency
• Complexity
• Error incidence
• Amount of skills, expertise and
understanding needed
• Not commonplace
• Least engaging, highly rewarding, non-
binding content
• Areas for improvement
• Training and customization
• Language data and rules optimization
• Writing
• Controlled languages
• Reliability
• Problematic ROI

Modes of Use
Unrestricted
texts
High
quality
Restricted input
Low quality
Impractical
Interactive
Fully
automatic

Estimating MT quality
 Biased baseline: MT is always bad
 Most translators do not know much about MT
• Same old jokes about silly mistakes
• A mixture of ignorance and fear
 Automatic metrics
 Hard interpretation
 PEMT effort
 Annotation guidelines
• Assigning 1-5 scores

Automatic Metrics
 BLEU
 No indication of accuracy
 1 ≤ P ≤ 0
• 1 = professional human translation
• .65 = human quality
• A score increase does not necessarily mean improved translation quality
 NIST
 Based on BLEU, with some alterations
 METEOR (Metric for Evaluation of Translation with Explicit ORdering)
 Based on BLEU
• Harmonic mean of unigram precision and recall
 F-Measure (F1 Score or F-Score)
 A measure of a test’s accuracy used in machine learning
 Based on BLEU, with some alterations
 WER (Word Error Rate)
 Most often used in speech recognition
• 1 ≤ P ≤ 0

Prerequisites
 Post-editing throughput must be faster than
translation
 Post-editing must be less keyboard intensive
than translation
 Post-editing must be less cognitively
demanding than translation

PEMT Effort
 System
 RbMT
• Dictionary
• Rules
• Customizability
 Data‐driven
• Suitability of input
• Training data
– Volume
– Domain
 Product captivity
• Different technologies that can or cannot be used within more than just one tool
 Language pair
 Outcome in one language combination cannot be compared with that in
another
 Text type
 Domain

PEMT Issues
 System
 RbMT
• Incorrect word/term
• Incorrect attachment
• Meaning not
disambiguated
 Data‐driven
• Missing words
• Capitalization
• Punctuation
• Fluency inconsistency

Post-editing
 Gist
 Raw MT
• Disposable (volatile UGC)
• Validation of automatic evaluation
 Light
 Making the translation understandable
• Ignoring all stylistic issues
• Adjusting mechanical errors (capitalization, and
punctuation)
• Replacing unknown words (misspelled in ST)
• Removing redundant words
 Heavy
 Making the translation stylistically appropriate
• Fixing machine‐induced meaning distortion
• Making grammatical and syntactic adjustments
• Checking terminology (new terms)
• Partially or completely rewriting sentences
– Adjusting for target language fluency

Degrees of Post-editing
 User requirements
 Quality expectations
 Perishability
 Volume
 Text function
 Turn‐around time

PEMT Specs
 Type of MT
 In house vs. outsourced MT
 Type of MT output
 Generic, untrained MT output
 Trained MT output
 Quality guidelines and index for raw translation
 ≥ 40% reusable
• BLEU
– Acceptable 0.3 to 0.5
– Good: 0.5 to 1
 Request a sample

PEMT Specs
 Rationale for MT
 Increased throughput
• More languages
• More content
 Faster turnaround time
 Reduced cost
 Accuracy and consistency
 Target consumer
 Quality of the finished product
 Reprocessing
 Publication
 Amount and type of PEMT
 Gist
 Light
 Heavy
 Participation in ongoing training of engine

Warning
 MT engines are not all equal
 Raw output quality is not consistent from
system to system and language to language
 MT error patterns are not consistent from
segment to segment

PEMT Instructions
 Clear and concise
 Tools
 Stick to style guide
• Language specific conventions
• Country/region standards
• Grammar, syntax and orthographic conventions
 Retain as much raw translation as possible
 Don’t hesitate too long over a problem
 Don’t worry about style
 Don’t embark on time‐consuming research
 Make changes only where absolutely necessary
• Non sense
• Wrong words (possibly misspelled in source text)
• Missing words
• Punctuation, capitalization
• Inflection
• Gender
• Word order
• Formatting

PEMT Instructions
 Example for heavy PEMT project
 The quality expected is publishable quality, this means no
deletions or omissions in the text, full accuracy and no
mistranslations with regards to the source text,
compliance to language rules of grammar and spelling for
the target language, and compliance to the terminology
following the glossary provided.
 Make as less edits as possible
• Just correct errors
– Follow the glossary
– If different terminology is found, replace it with the one in the glossary
• Do not introduce preferential changes
• Do not re-write text, unless to correct nonsenses
• Do not try to ‘improve’ the text

PEMT skills
 Working knowledge of SL
 Excellent command of TL
 Specialized domain knowledge
 Ability to comply with guidelines
 Unbiased attitude towards MT
Establish your own PEMT academyEstablish your own PEMT academy

Pricing and Compensation
 It will take a year or two more to build out a
widely accepted and dominant compensation
model
 The final model will be tied to productivity

Pricing and Compensation
 Common approaches
 Paying as for high fuzzy matches
• 85%-94%
• PEMT and post-editing of fuzzy matches are deeply
different
– Fuzzy matches are inherently correct segments
 Minor changes (possibly a term or two)
– MT is not necessarily inherently correct
 Even ‘Light PEMT’ could eventually result heavy
 Paying a time-based fee

Always
 Provide candidate post-editors with MT
samples before contracting
 Agree on throughput rates
• 450 to 750 words/hour
 Run a pilot project
 For every domain
 For any new combination

Compensation Grid
 Generals
 Method
 Type of output
• Generic, trained or untrained
 Quality of output
 Number of references
 Quality expectations
• Threshold
 File formats
• Tagging
 Time-based fee
 Productivity rate
• Productivity differ by post-editor
 Time for filling in QA forms
• For ongoing training of engine
 Use a spreadsheet to track time

Post-Editing of Machine Translation: Developing Requirements and Compensation Schemes

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Post-Editing of Machine Translation: Developing Requirements and Compensation Schemes

Semelhante a Post-Editing of Machine Translation: Developing Requirements and Compensation Schemes (20)

Mais de Luigi Muzii

Mais de Luigi Muzii (20)

Último

Último (20)

Post-Editing of Machine Translation: Developing Requirements and Compensation Schemes

Notas do Editor