“Post-editing” – what does it really mean?

Note: This is a revised version of a text previously published at the eMpTy Pages blog. This version is more up-to-date (and slightly enlarged), but the blog post is followed by several interesting comments not included here.

I have read many articles and presentations and even dissertations on post-editing of machine translation (PEMT), and strangely, very few of them have made a clear distinction between the editing of a complete, pre-translated document and the editing of machine-translated text segments during interactive translation in a CAT tool. In fact, in many of them it seems as if the authors are primarily thinking of the latter. Furthermore, most descriptions or definitions of “post-editing” do not even seem to take into account any such distinction. All the more reason, then, to welcome the following definition in ISO 17100, Translation services – Requirements for translation services:


edit and correct machine translation output

Note: This definition means that the post-editor will edit output automatically generated by a machine translation engine. It does not refer to a situation where a translator sees and uses a suggestion from a machine translation engine within a CAT (computer-aided translation) tool.

And yet… in ISO 18587, Translation services – Post-editing of machine translation output – Requirements, we are once again back in the uncertain state: the above note has been removed, and there are no clues as to whether the standard makes any difference between the two ways of producing the target text to be edited.

This may be reasonable in view of the fact that the requirements on the “post-editor” arguably are the same in both cases (and it seems that was the rationale for the decision to delete the note). Still, that does not mean that the situation and conditions for the translator are the same, nor that the client – in most cases a translation agency, or language service provider (LSP) – see them as the same. In fact, when I asked some translation agencies whether they see the work done during interactive translation using MT as being post-editing, they told me that it is not.

But why not? There are definitions of different levels of post-editing, so theoretically all types of such jobs should be possible to include under that heading. But there are problems here. There is “light” and “full” post-editing, and briefly, the former means that the resulting text is comprehensible and accurate, but the editor need not – in fact, should not – strive for a much “better” text than that, and should use as much of the raw MT version as possible. The purpose is to produce a reasonably adequate text with relatively little effort, and it is not meant for publishing. Producing such a text is sometimes called ”gisting” (you provide the gist of the contents). The latter means that the result should be ”indistinguishable from human output” (in the words of ISO 18587), or ”publishable”. And yet there are conflicting views on this: some sources say that stylistic perfection is not expected and that clients actually do not expect the result to be comparable to “human” translation. ”Do not worry too much about style, standards of textuality” and ”Quality expectations: medium”.[1]  And: ”Texts that are post-edited should not strive for linguistic perfection; instead, the goal should be linguistic adequacy.”[2]

Looking past the definition in the standard and instead using definitions of what I perceive as practice, I would say that there are in fact three levels of utilizing MT output: At the top there is the result ”indistinguishable from human output”; i.e. it is impossible to tell whether MT has been used or not. Slightly below that, there is the ”full” post-editing: correct in all aspects but perhaps not top-level stylistically. And then there is the ”ligth” level: useable for understanding but not more (not fun to read). Of course these categories are only points on a continuous scale; it is difficult to objectively test that a PEMT text fulfils the criteria of one or the other (is the light version really not above the target level? is the full version really up to the requirements?), even if such criteria are defined in ISO 18587 (and other places).

But quality levels are one thing – whether the job is performed on a complete, pre-translated text or interactively in a CAT tool is another. So why should that matter? And it really may not, as witness the point of view taken by the authors of ISO 18587 – that is, it may not matter to the quality of the work performed or the results achieved. But it matters a great deal to the translator doing the work. Basically, there are three possible job scenarios:

A: The job consists of editing (“post-editing”) a complete document which has been machine-translated; the source document is attached, and the client defines the desired level of quality. The editor (usually an experienced translator) can reasonably assess the quality of the translation and based on that make an offer. The assessment includes the time s/he believes the job will take, including any necessary adaptation of the source and target texts for handling in a CAT tool.

B: The job is very much like a normal translation in a CAT tool except that in addition to, or instead of, an accompanying TM the translator is assigned an MT engine by the client (usually a translation agency). Here, too, a level of quality is defined. Usually a pre-analysis showing the possible MT (and TM) matches is also provided. The translator is furthermore told that the compensation will be based on a post-analysis[3] of the edited file and depend on how much use has been made of the MT (and, as the case may be, the TM) suggestions. Still, it is not possible for the translator either to assess the time required (partly because there is still no method for judging in advance the quality of an MT engine) or the final payment. Also, s/he does not know how the post-analysis is made, so the final compensation will be based on trust.

(And, of course, if this method of basing payment on a post-assessment of the job done becomes accepted, one can easily foresee it being applied as well to traditional jobs using CAT tools in combination with TMs, without machine translation.)

C: The job is completely like a normal translation in a CAT tool, and the compensation is based on the translator’s offer (word price or packet price); a TM and a customary TM matches analysis may be involved (with the common adjustment of segment prices). However, the translator can also – on his or her own accord – use MT; depending on the need for confidentiality it may be an in-house engine using only the translator’s own TMs; or it may be online engines with confidentiality guaranteed. Whatever the case, the translator stands to win some time thanks to the MT resources without having to lower his or her pricing.

In addition to this, there are differences between scenarios A and B in how the work is done. For instance, in A you can use Find & replace to make changes in all target segments; not so in B (unless you start by pre-translating the whole text using MT) – but there you may have some assistance by various other functions offered by the CAT tool and also by using Regular expressions (regex). And if it’s a big job, it might be worthwile, in scenario A, to create a TM based on the texts and then redo the translation using that TM plus any suitable CAT tool features (and regex).

Furthermore, there are some interesting research results as to the efforts involved, insights which may be of help to the would-be editor. It seems that editing medium quality MT (in all scenarios) takes more effort than editing poor ones – it is cognitively more demanding than discarding and rewriting the text. Also the amount of effort needed to detect an error and decide how to correct it may be greater than the rewriting itself; and reordering words and correcting mistranslated words takes the longest time of all. An additional aspect is that all jobs involving “light” quality is likely to be avoided by most translators. Not only does it go against the grain of everything a translator finds joy in doing, i.e. the best job possible; experience also shows that all the many decisions that have to be made regarding which changes need to be made and which not often take so much time that the total effort with “light” quality editing is not much less than that with “full” (or even ”best”) quality.

In addition, it seems that post-editors differ more in terms of actual PE time than in the number of edits they make. Interestingly, it also seems that translators leave more errors in TM-matched segments than in MT-matched ones. And the mistakes are of different kinds.

These facts, plus the fact that MT quality today is taking great steps forward (not least thanks to the fast development of neural MT, even taking into account the hype factor), are likely to speed up the current trend. According to Arle Lommel, senior analyst at CSA Research and an expert in the field, it can be described thus:

A major shift right now is that post-editing is being replaced by “augmented translation.” In this view, language professionals don’t correct MT, but instead use it as a resource alongside TM and terminology. This means that buyers will increasingly just look for translation, rather than distinguishing between machine and human translation. They will just buy “translation” and the expectation will be that MT will be used if it makes sense. The MT component of this approach is already visible in tools from Lilt, SDL, and others, but we’re still in the early days of this change.

This will probably mean that we can partly do away with the “post-editing” misnomer – editing is editing, regardless of whether the suggestion presented in the CAT tool interface comes from a TM or an MT engine. (This erasing of boundaries is well described by Sharon O’Brien.[4]) Therefore, the term “post-editing” should be reserved only for the very specific case in scenario A, otherwise the concept will be meaningless. This view is taken in, for instance, the contributions by a post-editor educator and an experienced post-editor in the recently published Machine Translation – What Language Professionals Need to Know[5],[6].

And since mr Lommel’s prophesy means that scenario A stands to vanish, it also seems that eventually we will be left with mainly scenarios B and C – which leaves the matter, for translators, of how to come to grips with B. This is a new situation which is likely to take time and discussions to arrive at a solution (or solutions) palatable to everyone involved. Meanwhile, us translators should aim to make the best possible use of scenario C. MT is here and will not go away even if some people would wish it to.

[1] O’Brien, Sharon, Roturier, Johann, and de Almeida, Giselle (2009): Post-Editing MT Output. CNGL. http://www.mt-archive.info/MTS-2009-OBrien-ppt.pdf

[2] Hansen-Schirra, Silvia, Schaeffer, Moritz, and Nitke, Jean (2017). Post-editing: strategies, quality, efficiency. In: Porsiel (ed.): Machine Translation. Berlin: BDÜ Fachverlag.

[3] See for instance Memsource’s ”Post-editing Analysis”, https://help.memsource.com/hc/en-us/articles/115003942912-Post-editing-Analysis

[4] O’Brien, Sharon (2016): Post-Editing and CAT. In: 2016 48 EST Newsletter. https://issuu.com/est.newsletter/docs/2016_48-est

[5] Hansen-Schirra, Schaeffer, Nitke, ibid.

[6] Grizzo, Sara (2017): Working as a post-editor: a field report. In: Porsiel (ed.): Machine Translation. Berlin: BDÜ Fachverlag.

You can leave a response, or trackback from your own site.

Leave a comment

Powered by WordPress | Designed by: backlink indexing | Thanks to Mens Wallets, warcraft gold and buy backlinks