A proposed framework for making news algorithms more transparent, presented at the http://www.algorithmic.news/ conference #AlgorithmicNews

1. How Can We Make Algorithmic News
More Transparent?
Stuart Myles
Director of Information Management, Associated Press
www.ap.org @smyles
Algorithms, Automation and News, 22nd May 2018

2. News Algorithm Transparency
• Automation of production, distribution and consumption of
news
• Such as performing analyses, ranking search results, generating
news reports
• A framework for making news algorithms more transparent
• Four levels of transparency - from simple “disclosure” to full
“reproduction”
• Three sets of stakeholders - technicians, journalists and readers
• Some transparency examples from the Associated Press
• Including where we could do better
• Suggestions for further transparency topics to explore
www.ap.org @smyles

3. Transparency Stakeholders (1/3):
Technicians
• Technical transparency is the main focus of algorithmic
transparency research
• Is the algorithm working correctly?
• Accuracy, bias, semantic drift
• Automatic decisions
• Ranking, classification, filtering
• Synthetic news
• Fill-in-the-blank templates, summaries, video-from-text, text-from-
images
www.ap.org @smyles

4. Transparency Stakeholders (2/3):
Readers
• Transparency often cited as a means of (re)building trust in
journalism
• It seems likely that lack of transparency undermines trust
• How can algorithmic news be more transparent for
consumers of news?
• Readers
• Viewers
• Listeners
• Why am I seeing / not seeing this news item?
www.ap.org @smyles

5. Transparency Stakeholders (3/3):
Journalists
• Most algorithms used in news originate outside journalism
• News algorithms should reflect an organization’s editorial voice
• (Do platforms which deliver news have an editorial voice?)
• Journalists should be involved in crafting the explanations for
readers
• Readers need understandable explanations
• Journalists themselves should have some understanding of the
algorithms which power and mediate their work
• Journalists and editors
• Lawyers, archivists, other professionals within a news
organization
www.ap.org @smyles

6. Levels of Transparency (1/4):
Disclosure
• Reveal algorithms were used in creating or making decisions
about news items
• Could be a general statement about a set of items
• Might be attached to individual items
• Could identify which algorithms were used and in what ways
• Disclosure is the minimal level of transparency
• May be the only form of transparency available if the algorithm is
handled by a 3rd party?
• Disclosures are most useful for readers
www.ap.org @smyles

7. Disclosures by Associated Press
• AP includes a disclosure on automatically synthesized text
stories
AP created this story using Automated Insights' Wordsmith Platform
(http://automatedinsights.com/ap) and data from Zacks Investment Research.
• Indicates that AP journalists created the story
• Journalists designed template
• Template is automatically populated per our rules
• Identify 3rd party data sources and tools
• Included on every automatically created story
www.ap.org @smyles

8. Levels of Transparency (2/4):
Justification
• Justifications aim to show that the results of the algorithmic
news are reasonable in a particular instance
• A step up from disclosure - provides a degree of transparency
• Offers some reasons for the algorithmic result in a particular
instance
• Not a comprehensive set of reasons
• Discusses a particular decision, rather than the general use of an
algorithm
• A complete set of reasons may not be appropriate
• To keep proprietary information confidential
• Recipient cannot reasonably be expected to understand the full scope of
the workings of the news algorithm
www.ap.org @smyles

9. Facebook Advert Justification
• Facebook’s “Why am I seeing this ad?”
One reason you’re seeing this is [specific campaign criteria]. This is based on
[tracking techniques]. There may be other reasons you’re seeing this ad,
including [broad targeting criteria]. This is based on [other types of tracking
techniques].
• Offers a potential model for news algorithm justifications
• Justifications would be most helpful for readers and journalists
• Avoids the excuse of “it is too complex to fully explain”
www.ap.org @smyles

10. Levels of Transparency (3/4):
Explanation
• Why was a particular decision, categorization or arrangement of
news selected and not some other?
• Explain the outcome of a specific instance of a news algorithm
• I haven’t found algorithmic explanations “in the wild”
• There is active research into how to generate algorithmic explanations
• Algorithmic approximations, generate counterfactuals, generate rules
• AP’s rule based system provides explanations suitable for technicians
• What makes an explanation useful?
www.ap.org @smyles

11. Useful Explanations (1/2)
• A useful explanation allow you to take an action in response
• Alter an algorithm which has made an incorrect decision
• Alter a news item to conform to an algorithm’s criteria
• Such as adding missing metadata
• Alter other metadata, not on the item, to get a different
result
• Such as user preferences
• Useful explanation can include confidence scores, as well as
narrative
• Is editorial review required, due to low confidence?
www.ap.org @smyles

12. Useful Explanations (2/2)
• Useful explanations of multiple decisions can reveal
systematic issues
• Biased decisions which favour or penalize particular groups
• Where an algorithm is suitable and where it will not be
applicable
• Are the training data or assumptions out-of-date?
www.ap.org @smyles

13. Levels of Transparency (4/4):
Reproduction
• Sufficient information to allow the news algorithm to be
independently replicated
• Provide underlying data and code directly
• Describe in a “nerd box”
• It may not be possible to provide all the data
• Time-sensitive like trending topics or there’s just too much of it
• Algorithms may be proprietary
• Running algorithms and handling data can require a lot of
technical wherewithal
• Most useful for technicians
www.ap.org @smyles

14. Rule based classification at AP:
Explanation and Reproduction
• AP auto-categorizes all English-language content
• Tags for people, places, companies, organizations and subjects
• AP’s automated rules engine has about 200,000 classification
rules
• Hand-crafted by a team of specialists
• Sophisticated strategies for disambiguation and precision
• “Classification Admin” application to develop and test rules
• Evaluate a news item against one or more rules
• Highlights why a text matches the rules
• Both reproducibility and a form of explanation
• Explanations are not designed to be shared with journalists or readers
www.ap.org @smyles

15. News Algorithm Transparency at AP
• In writing the paper, I felt AP has a good transparency story
• But I see there is room for improvement
• We use disclosure on machine-generated stories
• We use rules-based classification
• Transparency for internal technicians
• We make algorithms and data for data journalism stories
available to AP members
• Requires a certain amount of know how
• We could do better at transparency for journalists and news
consumers
• With justifications and explanations
www.ap.org @smyles

16. Other Areas to Explore
• Difficulty of effectively conveying explanations of news algorithms
• Visual symbols and icons for disclosure of the use of algorithms?
• Narratives for justification or explanation of algorithmic outcomes, rather than
statistical readouts, using Natural Language Generation?
• Are narratives helpful for photo, video or audio?
• Transparency for algorithmic errors and corrections?
• Transparency for time-dependent algorithms?
• Trending stories or collaborative filtering
• All transparency all the time? Or only on demand?
• In the pre-algorithm days, we didn’t have full transparency
• Equivalent to responding to letters to the Editor
• Consumers don’t require transparency from other algorithmic systems – e.g. cars
www.ap.org @smyles

17. News Algorithm Transparency
• A framework for making news algorithms more transparent
• Three sets of stakeholders
• Technicians, journalists and readers
• Four levels of transparency
• Disclosure, justifications, explanations, reproduction
• Active research into how to generate justifications and
explanations
• Journalists should be involved in crafting useful justifications and
explanations
www.ap.org @smyles