Humans vs Machines: An MT cost benefit analysis
•Transferir como PPTX, PDF•
1 gostou•1,206 visualizações
In this presentation we compare machine translation with human translation. We look at a side by side comparison of workflows and a cost benefit analisys.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais de RDC
Último
Último (20)
Humans vs Machines: An MT cost benefit analysis
- 1. Humans vs. Machines Translations for a working world
- 2. Welcome • • • • Translations for a working world
- 3. MT and the current landscape • • • • • • • Translations for a working world
- 4. The Project • • • • • • • Translations for a working world
- 5. Quick Break! Translations for a working world
- 6. Human Translation Workflow Translations for a working world
- 7. Human Translation Translations for a working world
- 8. Human Translation Translations for a working world
- 9. MT + Post-Editing Workflow Translations for a working world
- 10. MT + Post-Editing Workflow Translations for a working world
- 11. Machine Translation + Post-Editing Translations for a working world
- 12. Quick Break! Translations for a working world
- 13. Comparison Translations for a working world
- 14. Where is MT applicable? • • • • • • • • • Translations for a working world
- 15. What are the requirements? • • • • • • Translations for a working world
- 16. What’s the potential? • • • • • Translations for a working world
- 17. Questions? Translations for a working world
- 18. Thank you! Translations for a working world
Notas do Editor
- Hello everyone and welcome to todays webinar, Humans vs Machine: an MT cost benefit analisys. My name is Rob Davies, I’m the Marketing Manager here at Milengo and I‘ll be presenting today. I‘m also joined by Deepan Patel who‘s our MT Lead Project Manager, he‘ll be on hand for any technical questions, and also Valarie Badame, our Marketing Assistant who will be keeping collating questions for the end of the webinar.Just so you know, this webinar is being recorded so you can access it again at a later date, and we’ll also be making the slides from the webinar available too. Details of how to access the recording and the slides will be emailed to you after the webinar.We’ll be running polls at intervals throughout the webinar and you should see the poll questions pop up on screen. We’ll go through the responses during the webinar.We’ll also have a Q&A session at the end of the webinar, within the gotowebinar window there’s an area where you can submit questions. If you have a question during the presentation feel free to go ahead and submit them. I’ll keep an eye on questions during the presentation and we’ll try and answer all of them at the end.Lastly a data sheet that covers our MT post editing services will be made available to everyone after the webinar.
- So, first of all I’d like to start by looking at MT and the current landscape. Now for those of you who’ve been involved with the language industry for a while, you’ll have heard all the talk surrounding Machine Translation, and the fact that for some it is has been set to revolutionize the industry now, for the last 10 years or so! So if MT is so revolutionary, why isn’t human translation dead yet?Well, first reality hasn’t matched up with expectations, the ability of a machine to recognize the intricacies of language the way a human being does, just simply hasn’t been achieved yet. We’re talking about things like humor, metaphor, meaning and style. Remember, even the most sophisticated MT systems are just an algorithm, a mathematical model used to express the relationship between 2 languages. In a nutshell, computers aren't as smart as human translators, they can provide very good literal translations, but can’t deal with the cultural nuances that make each language unique.So, what’s available right now in terms of MT? Well everyone knows about free tools like Google Translate. But people use these primarily for gist translations, to get an understanding or an overview of a text, and they’re great for that, but not great for business. Why? Because they can’t handle large volumes of text, they can’t handle industry specific language like engineering or medical terminology and most importantly, any text you translate becomes part of their system and this raises concerns about confidentiality. I’m sure Microsoft wouldn’t want Google knowing about the product manuals for their new smart phones before they’re released, for example.So what’s available for business right now? Well, the big boys like Microsoft built their own systems. But that costs a huge amount in development and resources. At the other end of the spectrum you’ve got free open source solutions like Moses. But anyone who’s had experience with programming knows that a command prompt isn’t particularly user friendly and can’t be implemented as a production ready tool.So, what did we decide to do? Well at Milengo we’ve spent a couple of years testing a few systems and we currently use infrastructure built around a Moses core which is a lot more straight forward to use. But the key here is that we use human beings to post edit the output. This ensures that and the end of the project you can’t tell the difference between an MT and Human translated output, since it’s always been past the eyes of a human linguist. Quality is not an issue.So how does it work? And more importantly how does MT + Post editing differ from a human only approach?
- Well, we decided to do a little analysis and compare the 2 workflows on identical or near identical projects. Since we were already working with one of our clients on moving to an MT/PE workflow we decided to compare both, before and after. We were lucky here for a number of reasons.First of all both projects involved the translation of a User Interface and Technical Documentation for a piece of software designed to run customer service portals. The language was very similar across both projects, and all files were delivered to us ready for translation in ITD format, which was easy for us to deal with. Lastly the volume was also very similar, approximately 180k words for the human translation project and 189k for the MT project.The TM we used for both projects was identical as we’d kept a backup of the client TM before we began the first project. So we used an identical used an identical TM on both.On the technology side we used a platform called Smartmate for the MT engine. For anyone who’d like more information on the technical details of exactly how we set up and train an MT engine using Smartmate, I’d recommend watching our last MT webinar which is available on the resources section of our website at Milengo.com.Ok. So before we get into the workflow comparison, I want to take a little break and run a quick poll amongst the audience.
- I’m interested to know who is currently using MT in their translation workflows?You should see the question on screen now so go ahead and pick your answer.Ok I’ll leave it for a few more seconds, see if we get any more answers.So great, it looks like x amount of people are already using MT so the next part of the webinar you’ll see how one of our workflows compares with a human approach.
- Ok. So, lets look at the 2 translation workflows in detail. We’ll start by looking at the human only translation process we use when taking on a client for the first time. This usually involves a small pilot project to establish best working practices. This allows us to set up tools, prepare translation memories, really get a full understanding of how the client wants to work. Most importantly it also gives our translation teams the feedback they need to make sure that once we go into production on bigger projects, the style and accuracy of the translations are exactly as our clients want them.Once we received the ITD source files, TM and glossary from the client we assembled our language team. This consisted of 3 translators with technical translation experience and a senior post editor, also with subject matter expertise. While the team was assembled we prepared the TM and the glossaries. Once the resources and tools were set up, we ran a fuzzy match analysis of the source files against the TM, to leverage any previously translated content, and then handed them off for translation and editing. Once the first days worth of translation was finished our senior linguist got to work with second level editing and QA, and we continued like this until all translation and editing was complete. Once the files were handed back from the translators we ran a round of QA to ensure there were no errors and then delivered the translated ITD files over to the client. After feedback and review had been received our translators and editors made amends where necessary, updated the TM and glossary, carried out a final round of QA, and then we delivered the files back to the client. Once this pilot project was complete we were ready to move on to a production scenario with the full 180k word project.
- So here’s the production workflow. As you can see it's a little different to the pilot project. Again, the source files were sent to us in ITD format but this time there was no need to set up tools or assemble teams. Sometimes we will update TM’s and glossaries at this point if other projects have been carried out in the mean time, however in this case we didn't need to.As before we carried out a fuzzy match analysis of the source files against the TM and then handed off the files to the translators. Our translators set to work and our senior linguist began editing the day after. Once all translation and editing was complete we went through a round of QA, and handed the translated ITD files off to the client. Now since we’d already had the chance to train our teams on the client content in the pilot project linguistic queries were lower. We’d also identified stylistic and terminology preferences in the pilot, and so very little review and amendment was required for the final translations.So that’s the process, now lets look at costs and timescales for a production project with a human only translation workflow.
- I’ve split this into 2 here. Project management and Translation & Review.Now project management covers everything pre translation and post QA, so as you can see that’s team selection, tool set up, file prep and source file analysis. For a project size of 180k words this took us 3 days. The cost for this was 10% of the final project cost which is our standard project management fee.Translation was split between 3 translators who each averaged around 2500 words per day, or 7500 words per day total. Translation of the 180k word text took 24 days. As I mentioned before, our senior linguist began editing in parallel with the translators one day after they began, and averaged around 5000 words per day. As the second level editing covered the entire 180k words it determined the overall timescale. The entire translation/editing stage took 37 days in total.Total time for the entire 180k word project including translation/editing and project management was 40 days. Costs were based on our per word rate for English into German and our standard project management rate of 10% of the total project cost.Now lets look at our MT workflow, beginning with the pilot project which includes engine setup and training.
- So, this is a standard process we use when building a new engine for the first time, as was the case for our client here. As before we received the source files as ITD files, along with the client TM and glossary and set everything up for the translation and editing process. File preparation required an extra step as we needed to convert the ITD source files into XLIF to process through the MT engine. Our post editing team was built while the technical set up was being completed and we chose 3 post editors and a senior language lead for second level review.During engine training we first extracted a test set from the TM of 1000 segments, these would then be excluded from the training data used to build the engine. Once the engine was ready we processed the test set through the engine and generated a BLEU score by comparing MT output against the reference translations of the test set. The BLUE score generated was 54 which indicated a level of quality and fluency good enough to see definite productivity benefits during post editing.Next we took the source files (approximately 15k words for the pilot project) and ran an analysis against the TM, everything over 85% match was pre-translated using the TM. The full source text was run back through the engine which ignored everything over an 85% match and translated the rest. Once this was complete it was post edited and put through a round of QA. Before we could deliver the files they went through post processing, converting them back to ITD format for deliver. A final round of feedback and amends was then implemented before a last round of QA and delivery back to the client.Now as you can see this is quite a different process, I’ve highlighted the 5 extra steps with stars. However as you’ll see shortly, more steps in the process doesn’t necessarily translate into a longer project timescale.
- So lets look at the production workflow.Well, you’ll notice its a lot more straightforward since all the hard work building and training the engine has already been done. As always, the files were received in ITD format and as with the MT pilot converted to XLIF.At the same time we ran a fuzzy match analysis on the files and pre-translated everything over an 85% match using the TM. All 189k words were put through the MT engine, ignoring the pre-translated matches, and then handed on to our post editors. With post editing completed we went through QA, conversion from XLIF back to ITD and delivery to the client. A final a round of review and amends was implemented, all of which was also used to retrain the engine and improve accuracy. Final QA was completed and the files were handed back to the client.So that’s the production workflow, lets look at times and costs.
- Well, project management is the same as with a human only approach since we’re carrying out the same project management tasks. The cost doesn’t change either since project management is always 10% of the final project cost. As before this took us 3 days.Engine setup and training is a significant additional step in the process compared to human only translation and therefore requires an extra day. However since MT can process 2000 words per minute, the actual time taken translating is cut right down to 66 minutes since only words under an 85% match are translated. Now as post-editing is a very different discipline to translation it takes less time, in fact post editors can usually handle twice the daily word volume of translators. This means the entire 189k project was edited in just 13 days. As before, second level editing from our senior linguist began after the first round of translation and this was also slightly faster at an average of 7000 words per day for 24 days.So the total time for the project including project management, but not the extra day engine set up as this had been completed, was 28 days giving a time saving of 12 days.In terms of cost, project management was the same, but post-editing depending on the level you require, is up to 50% cheaper. So with an MT/Post editing workflow, not only did the client finish the project in 30% less time, the overall cost was almost half what they would usually expect to pay.It’s also worth noting here that once a production workflow is set up, and the post edited, amended output is used to retrain the engine, accuracy grows exponentially. So essentially, the more you use the engine, the smarter it gest and the less post editing is required. The leads to even less post editing, and shorter project times.
- Ok, before we go on to our conclusion here I just want to take another quick break since that’s quite a bit of information to take in. I also want to run another quick poll so fingers at the ready!Now we’ve been through the workflows I’m interested to know if you think it is significantly more complicated to implement MT than traditional translation?Is this one of the sticking points that’s so far caused low adoption?Ok great so it looks like yes/no it is/isn’t more complicated. So lets move on to the comparison between the 2.
- Now it’s time to sum everything up. Lets look at how the 2 approaches compare. Humans vs Machines (humans vs a human/machine team which is how I prefer to look at it).Project management and set up. This is the part of the pilot project at the beginning which is directly comparable. How long does it take and how much does it cost to get up and running with MT and post editing as opposed to straight forward human translation.Well, in this case, the human only approach is still quicker. While the project management tasks were the same between projects with MT you always have to add extra time to set up and train an engine, here we added an extra day. So that’s 3 days project management/setup for a human only approach vs 4 days for MT. However, here’s the good news, the cost is exactly the same. Project management is always 10% and the cost of setting up and training an engine, we’re offering that free of charge.So lets have a look at Translation Time. Now this is the area we see a BIG difference. Our team of translators took 37 days to translate and edit 180, 000words. Our MT engine took 66 minutes! We had to post edit these which took us 24 days but as you can see, 13 days is a big difference.Now onto the cost. As I mentioned before, our post editors don't need the same skill set as a professional linguist and therefore our per word rates can be cheaper, which depending on the level of post editing you need, can be up to 50% cheaper!So. Looking at how they stack up. MT takes longer to set up, but, once it’s in production MT+Human Post editing delivers on average a 33% productivity gain and works out to be up to 50% cheaper than a human only approach. However, while all this is great news, there are some things to take into account!
- Here’s what you need to bear in mind. MT doesn’t work for everything, for it to be successful you need source text with controlled language. That’s why technical translation, which tends to be very linguistically consistent, works well. Advertising, sales or marketing copy are much better handled by professional human translators.If you author your files and documents in a controlled environment, even better. In tests we ran a couple of years back we saw a dramatic impact on accuracy from MT engines before and after being fed controlled language.Also the larger the project the better, we wouldn’t recommend anything lower than 10,000 words as the cost an time benefits from using MT+ post editing will be negligible. Some example uses are for:Product manualsInstruction manualsData sheetsTechnical specsKnowledge bases and help files
- So if you’re ready to start a pilot project for your company, here are some requirements.Firstly, with the engines we use we’ve found they’re best applied to technical language, and therefore IT and technology companies stand to benefit most. That doesn’t mean however that if you’re not in IT or technology that MT is not suitable. Any industry that produces large volumes of standardized content and has the available existing language assets, should benefit from an MT/PE workflow.To achieve the cost savings and make the investment worthwhile an annual spend of over $50,000 per language pair is recommended. As I mentioned earlier, project sizes of over 10,000 words are required to get cost savings of between 30% and 50%.Now, there are also some specific requirements for training an MT engine to a quality level that is useable by post editors. To train an engine to this kind of level we would recommend a TM with a minimum of 500,000 sentence pairs or translated units. We would also recommend a minimum of 300MB of monolingual data, although we can be flexible with this in certain cases.So those are the requirements, and I’ll leave you with my final slide which really covers the potential of the technology.
- Well, according to industry figures in 2011 the total global spend on language services was $30 billion. Around 55% of that, or $16.5 billion was spent on translation. Less than 3%, interestingly, was spent on MT post editing. Now if we were to take a conservative estimate and say 20% of that was technical translation, where MT is best suited, that leaves us $3.3 billion spent on technical translation alone. With a saving of 30% MT with post editing could have a potential cost benefit of nearly $1.1 billion dollars.Looking at an average translation project value, also from the 2011 numbers, the average figure across all industries polled was just over $13,000 per project. So here MT has the potential to not only save time, but also push the average cost per project under $10,000 dollars. Again another big impact that this technology has the potential to make.So, these are really just some figures to illustrate the kind of savings that MT can make, both in time and cost, vs a traditional human only approach, and as I said before, this doesn’t mean human translation is going to be made obsolete any time soon. There are still plenty of applications where MT can’t compete. Marketing, sales, advertising and for that fact any other creative copy are still best handle by skilled linguists. But for large volumes of standardized text, I think it’s clear that MT and post editing can really make a big impact.
- So, that brings us to the end of this presentation. I’d like to open things up and go through any questions we’ve had during the webinar and also give you time to ask any more.So lets have a look at the questions.Ok so that’s all the questions for today
- Finally I’d like to say thank you to everyone who attended today, as I mentioned at the beginning we will make the recording available to everyone who attended. And for more information on Machine Translation and post editing, visit our website www.milengo.com and check out our resources section. There you’ll find our previous MT webinar along with a case study and MT data covering our MT and post editing service.If you have any questions at all, feel free to email them to sales@milengo.com and we’ll get back to you as soon as possible.And that concludes the webinar for today.Thank you.