O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

How Can We Make Algorithmic News More Transparent?

554 visualizações

Publicada em

A proposed framework for making news algorithms more transparent, presented at the http://www.algorithmic.news/ conference #AlgorithmicNews

Publicada em: Tecnologia
  • Seja o primeiro a comentar

How Can We Make Algorithmic News More Transparent?

  1. 1. How Can We Make Algorithmic News More Transparent? Stuart Myles Director of Information Management, Associated Press www.ap.org @smyles Algorithms, Automation and News, 22nd May 2018
  2. 2. News Algorithm Transparency • Automation of production, distribution and consumption of news • Such as performing analyses, ranking search results, generating news reports • A framework for making news algorithms more transparent • Four levels of transparency - from simple “disclosure” to full “reproduction” • Three sets of stakeholders - technicians, journalists and readers • Some transparency examples from the Associated Press • Including where we could do better • Suggestions for further transparency topics to explore www.ap.org @smyles
  3. 3. Transparency Stakeholders (1/3): Technicians • Technical transparency is the main focus of algorithmic transparency research • Is the algorithm working correctly? • Accuracy, bias, semantic drift • Automatic decisions • Ranking, classification, filtering • Synthetic news • Fill-in-the-blank templates, summaries, video-from-text, text-from- images www.ap.org @smyles
  4. 4. Transparency Stakeholders (2/3): Readers • Transparency often cited as a means of (re)building trust in journalism • It seems likely that lack of transparency undermines trust • How can algorithmic news be more transparent for consumers of news? • Readers • Viewers • Listeners • Why am I seeing / not seeing this news item? www.ap.org @smyles
  5. 5. Transparency Stakeholders (3/3): Journalists • Most algorithms used in news originate outside journalism • News algorithms should reflect an organization’s editorial voice • (Do platforms which deliver news have an editorial voice?) • Journalists should be involved in crafting the explanations for readers • Readers need understandable explanations • Journalists themselves should have some understanding of the algorithms which power and mediate their work • Journalists and editors • Lawyers, archivists, other professionals within a news organization www.ap.org @smyles
  6. 6. Levels of Transparency (1/4): Disclosure • Reveal algorithms were used in creating or making decisions about news items • Could be a general statement about a set of items • Might be attached to individual items • Could identify which algorithms were used and in what ways • Disclosure is the minimal level of transparency • May be the only form of transparency available if the algorithm is handled by a 3rd party? • Disclosures are most useful for readers www.ap.org @smyles
  7. 7. Disclosures by Associated Press • AP includes a disclosure on automatically synthesized text stories AP created this story using Automated Insights' Wordsmith Platform (http://automatedinsights.com/ap) and data from Zacks Investment Research. • Indicates that AP journalists created the story • Journalists designed template • Template is automatically populated per our rules • Identify 3rd party data sources and tools • Included on every automatically created story www.ap.org @smyles
  8. 8. Levels of Transparency (2/4): Justification • Justifications aim to show that the results of the algorithmic news are reasonable in a particular instance • A step up from disclosure - provides a degree of transparency • Offers some reasons for the algorithmic result in a particular instance • Not a comprehensive set of reasons • Discusses a particular decision, rather than the general use of an algorithm • A complete set of reasons may not be appropriate • To keep proprietary information confidential • Recipient cannot reasonably be expected to understand the full scope of the workings of the news algorithm www.ap.org @smyles
  9. 9. Facebook Advert Justification • Facebook’s “Why am I seeing this ad?” One reason you’re seeing this is [specific campaign criteria]. This is based on [tracking techniques]. There may be other reasons you’re seeing this ad, including [broad targeting criteria]. This is based on [other types of tracking techniques]. • Offers a potential model for news algorithm justifications • Justifications would be most helpful for readers and journalists • Avoids the excuse of “it is too complex to fully explain” www.ap.org @smyles
  10. 10. Levels of Transparency (3/4): Explanation • Why was a particular decision, categorization or arrangement of news selected and not some other? • Explain the outcome of a specific instance of a news algorithm • I haven’t found algorithmic explanations “in the wild” • There is active research into how to generate algorithmic explanations • Algorithmic approximations, generate counterfactuals, generate rules • AP’s rule based system provides explanations suitable for technicians • What makes an explanation useful? www.ap.org @smyles
  11. 11. Useful Explanations (1/2) • A useful explanation allow you to take an action in response • Alter an algorithm which has made an incorrect decision • Alter a news item to conform to an algorithm’s criteria • Such as adding missing metadata • Alter other metadata, not on the item, to get a different result • Such as user preferences • Useful explanation can include confidence scores, as well as narrative • Is editorial review required, due to low confidence? www.ap.org @smyles
  12. 12. Useful Explanations (2/2) • Useful explanations of multiple decisions can reveal systematic issues • Biased decisions which favour or penalize particular groups • Where an algorithm is suitable and where it will not be applicable • Are the training data or assumptions out-of-date? www.ap.org @smyles
  13. 13. Levels of Transparency (4/4): Reproduction • Sufficient information to allow the news algorithm to be independently replicated • Provide underlying data and code directly • Describe in a “nerd box” • It may not be possible to provide all the data • Time-sensitive like trending topics or there’s just too much of it • Algorithms may be proprietary • Running algorithms and handling data can require a lot of technical wherewithal • Most useful for technicians www.ap.org @smyles
  14. 14. Rule based classification at AP: Explanation and Reproduction • AP auto-categorizes all English-language content • Tags for people, places, companies, organizations and subjects • AP’s automated rules engine has about 200,000 classification rules • Hand-crafted by a team of specialists • Sophisticated strategies for disambiguation and precision • “Classification Admin” application to develop and test rules • Evaluate a news item against one or more rules • Highlights why a text matches the rules • Both reproducibility and a form of explanation • Explanations are not designed to be shared with journalists or readers www.ap.org @smyles
  15. 15. News Algorithm Transparency at AP • In writing the paper, I felt AP has a good transparency story • But I see there is room for improvement • We use disclosure on machine-generated stories • We use rules-based classification • Transparency for internal technicians • We make algorithms and data for data journalism stories available to AP members • Requires a certain amount of know how • We could do better at transparency for journalists and news consumers • With justifications and explanations www.ap.org @smyles
  16. 16. Other Areas to Explore • Difficulty of effectively conveying explanations of news algorithms • Visual symbols and icons for disclosure of the use of algorithms? • Narratives for justification or explanation of algorithmic outcomes, rather than statistical readouts, using Natural Language Generation? • Are narratives helpful for photo, video or audio? • Transparency for algorithmic errors and corrections? • Transparency for time-dependent algorithms? • Trending stories or collaborative filtering • All transparency all the time? Or only on demand? • In the pre-algorithm days, we didn’t have full transparency • Equivalent to responding to letters to the Editor • Consumers don’t require transparency from other algorithmic systems – e.g. cars www.ap.org @smyles
  17. 17. News Algorithm Transparency • A framework for making news algorithms more transparent • Three sets of stakeholders • Technicians, journalists and readers • Four levels of transparency • Disclosure, justifications, explanations, reproduction • Active research into how to generate justifications and explanations • Journalists should be involved in crafting useful justifications and explanations www.ap.org @smyles