An In-depth Analysis of the Effect of Text Normalization in Social Media

•Download as PPT, PDF•

0 likes•190 views

This document proposes a taxonomy for characterizing different types of text normalization edits based on their level of granularity and impact on downstream natural language processing applications like syntactic parsing, named-entity recognition, and text-to-speech synthesis. It examines the effects of coarse-grained edit types like addition, replacement, and removal as well as more fine-grained types like corrections to verb forms, determiners, capitalization, contractions, slang, and Twitter-specific terms. The analysis found that parsing is impacted by most normalization operations, entity recognition depends strongly on replacements, and speech synthesis benefits broadly from normalization but requires handling domain-specific terms.

Technology

Methodology
Taxonomy of Normalization Edits
Tyler Baldwin and Yunyao Li
IBM Research - Almaden
An In-depth Analysis of the Effect of Text Normalization
in Social Media
Normalization For Parsing
Normalization maps sentences or their tokens to their standard form. It can help downstream applications that expect
the data in “clean” formats. Early approaches were very application-centric.
Social media normalization has been characterized in the literature as a mapping from non-standard words to their
standard form, ignoring many operations.
Perfect normalization is difficult but the typical approach may not be sufficient for all applications.
Should we return to an application-centric approach?
Previous taxonomies do not agree on the appropriate level of granularity to examine.
Several levels of granularity allow for examination of big picture and in-depth.
Three applications: Syntactic parsing, named-entity recognition, text-to-speech synthesis
-600 tweets, TREC Twitter data
-Gold standard annotated to ideal form by human annotators
-Edit types examined via ablation
Normalization For NER Normalization For TTS
ADDITION
REPLACEMENT
REMOVAL
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
Coarse Granularity - Parsing
F-Measure Difference
BEVERB
DETERMINER
OTHER
SUBJECT
CAPITALIZATION
CONTRACTION
OTHER
SLANG
OTHER
TWITTER
ADDITION
REPLACEMENT
REMOVAL
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09
Fine Granularity - NER
F-Measure Difference
BEVERB
DETERMINER
OTHER
SUBJECT
CAPITALIZATION
CONTRACTION
OTHER
SLANG
OTHER
TWITTER
ADDITION
REPLACEMENT
REMOVAL
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14
Fine Granularity - Parsing
F-Measure Difference
ADDITION
REPLACEMENT
REMOVAL
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
Coarse Granularity - NER
F-Measure Difference
BEVERB
DETERMINER
OTHER
SUBJECT
CAPITALIZATION
CONTRACTION
OTHER
SLANG
OTHER
TWITTER
ADDITION
REPLACEMENT
REMOVAL
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
Fine Granularity - TTS
F-Measure Difference
ADDITION
REPLACEMENT
REMOVAL
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
Coarse Granularity - TTS
F-Measure Difference
Edit
Insertion Replacement Removal
Punctuation Word Word Word
Subj. Be Det. Other Slang Cont. Cap. Other Twitter Other
Punctuation Punctuation
Parsing is heavily impacted by most normalization operations. Entity recognition is mostly dependent on replacement edits.
Capitalization correction is important as well.
Speech synthesis is heavily impacted by normalization generally, but
handling domain-specific terms is critical.
@someGuy idk kinda wanna get NEW ipad
@someGuy I don't know kind of want to get NEW ipad
I don't know, I kind of want to get a new iPad.
Typical
Ideal

More from Yunyao Li

The ability to build large-scale domain-specific knowledge bases that capture and extend the implicit knowledge of human experts is the foundation for many AI systems. We use an ontology-driven approach for the creation, representation and consumption of such domain-specific knowledge bases. This approach relies on several well-known building blocks: natural language processing, entity resolution, data transformation and fusion. I will present several human-in-the-loop work that target domain experts (rather than programmers) to extract the domain knowledge from the human expert and map it into the "right" models or algorithms. I will also share successful use cases in several domains, including Compliance, Finance, and Healthcare: by using these tools we can match the level of accuracy achieved by manual efforts, but at a significantly lower cost and much higher scale and automation.

Human in the Loop AI for Building Knowledge Bases

Yunyao Li

Slides for talk given at Women in Engineering on March 20, 2021. Abstract: Natural language understanding is a fundamental task in artificial intelligence. English understanding has reached a mature state and successfully deployed in multiple IBM AI products and services, such as Watson Natural Language Understanding and Watson Discovery. However, scaling existing products/services to support additional languages remain an open challenge. In this talk, we will discuss the open challenges in supporting universal natural language understanding. We will share our work in the past few years in addressing these challenges. We will also showcase how universal semantic representation of natural languages can enable cross-lingual information extraction in concrete domains (e.g. compliance) and show ongoing efforts towards seamless scaling existing NLP capabilities across languages with minimal efforts.

Towards Universal Language Understanding

Yunyao Li

Tutorial at AACL'2020 (http://www.aacl2020.org/program/tutorials/#t4-explainability-for-natural-language-processing). More recent version: https://www.slideshare.net/YunyaoLi/explainability-for-natural-language-processing-249912819 Title: Explainability for Natural Language Processing @article{aacl2020xaitutorial, title={Explainability for Natural Language Processing}, author= {Dhanorkar, Shipi and Li, Yunyao and Popa, Lucian and Qian, Kun and Wolf, Christine T and Xu, Anbang}, journal={AACL-IJCNLP 2020}, year={2020} Presenter: Shipi Dhanorkar, Christine Wolf, Kun Qian, Anbang Xu, Lucian Popa and Yunyao Li Video: https://www.youtube.com/watch?v=3tnrGe_JA0s&feature=youtu.be Abstract: We propose a cutting-edge tutorial that investigates the issues of transparency and interpretability as they relate to NLP. Both the research community and industry have been developing new techniques to render black-box NLP models more transparent and interpretable. Reporting from an interdisciplinary team of social science, human-computer interaction (HCI), and NLP researchers, our tutorial has two components: an introduction to explainable AI (XAI) and a review of the state-of-the-art for explainability research in NLP; and findings from a qualitative interview study of individuals working on real-world NLP projects at a large, multinational technology and consulting corporation. The first component will introduce core concepts related to explainability in NLP. Then, we will discuss explainability for NLP tasks and report on a systematic literature review of the state-of-the-art literature in AI, NLP, and HCI conferences. The second component reports on our qualitative interview study which identifies practical challenges and concerns that arise in real-world development projects which include NLP.

Explainability for Natural Language Processing

Yunyao Li

Keynote talk given at Pacific Asia Conference on Language, Information and Computation (PACLIC 34) on Pacific Asia Conference on Language, Information and Computation (PACLIC 34) on October 24, 2020. Title: Towards Universal Natural Language Understanding Abstract: Understanding the semantics of the natural language is a fundamental task in artificial intelligence. English semantic understanding has reached a mature state and successfully deployed in multiple IBM AI products and services, such as Watson Natural Language Understanding and Watson Compare and Comply. However, scaling existing products/services to support additional languages remain an open challenge. In this talk, we will discuss the open challenges in supporting universal natural language understanding. We will share our work in addressing these challenges in the past few years to provide the same unified semantic representation across languages. We will also showcase how such universal semantic understanding of natural languages can enable cross-lingual information extraction in concrete domains (e.g. insurance and compliance) and show promise towards seamless scaling existing NLP capabilities across languages with minimal efforts.

Towards Universal Language Understanding (2020 version)

Yunyao Li

Keynote talk at TextXD 2019(https://www.textxd.org) Abstract: Understanding the semantics of the natural language is a fundamental task in artificial intelligence. English semantic understanding has reached a mature state and successfully deployed in multiple IBM AI products and services, such as Watson Natural Language Understanding and Watson Compare and Comply. However, scaling existing products/services to support additional languages remain an open challenge. In this demo, we will present Polyglot, a multilingual semantic parser capable of semantically parsing sentences in 9 different languages from 4 different language groups into the same unified semantic representation. We will also showcase how such universal semantic understanding of natural languages can enable cross-lingual information extraction in concrete domains (e.g. insurance and compliance) and show promise towards seamless scaling existing NLP capabilities across languages with minimal efforts.

Towards Universal Semantic Understanding of Natural Languages

Yunyao Li

Slides for our COLING'18 paper: http://aclweb.org/anthology/C18-1058 Fundamental to several knowledge-centric applications is the need to identify named entities from their textual mentions. However, entities lack a unique representation and their mentions can differ greatly. These variations arise in complex ways that cannot be captured using textual similarity metrics. However, entities have underlying structures, typically shared by entities of the same entity type, that can help reason over their name variations. Discovering, learning and manipulating these structures typically requires high manual effort in the form of large amounts of labeled training data and handwritten transformation programs. In this work, we propose an active-learning based framework that drastically reduces the labeled data required to learn the structures of entities. We show that programs for mapping entity mentions to their structures can be automatically generated using human-comprehensible labels. Our experiments show that our framework consistently outperforms both handwritten programs and supervised learning models. We also demonstrate the utility of our framework in relation extraction and entity resolution tasks.

Exploiting Structure in Representation of Named Entities using Active Learning

Yunyao Li

Slides for our COLING'16 paper http://aclweb.org/anthology/C/C16/C16-1058.pdf Abstract: Semantic role labeling (SRL) is the task of identifying and labeling predicate-argument structures in sentences with semantic frame and role labels. A known challenge in SRL is the large number of low-frequency exceptions in training data, which are highly context-specific and difficult to generalize. To overcome this challenge, we propose the use of instance-based learning that performs no explicit generalization, but rather extrapolates predictions from the most similar instances in the training data. We present a variant of k-nearest neighbors (kNN) classification with composite features to identify nearest neighbors for SRL. We show that high-quality predictions can be derived from a very small number of similar instances. In a comparative evaluation we experimentally demonstrate that our instance-based learning approach significantly outperforms current state-of-the-art systems on both in-domain and out-of-domain data, reaching F1-scores of 89,28% and 79.91% respectively

K-SRL: Instance-based Learning for Semantic Role Labeling

Yunyao Li

Coling poster

Yunyao Li

Coling demo

Yunyao Li

Slides deck for SIGMOD 2017 Tutorial. ABSTRACT: The volume of natural language text data has been rapidly increasing over the past two decades, due to factors such as the growth of the Web, the low cost associated to publishing and the progress on the digitization of printed texts. This growth combined with the proliferation of natural language systems for search and retrieving information provides tremendous opportunities for studying some of the areas where database systems and natural language processing systems overlap. This tutorial explores two more relevant areas of overlap to the database community: (1) managing natural language text data in a relational database, and (2) developing natural language interfaces to databases. The tutorial presents state-of-the-art methods, related systems, research opportunities and challenges covering both area.

Natural Language Data Management and Interfaces: Recent Development and Open ...

Yunyao Li

Poster for our ACL paper "Polyglot: Multilingual Semantic Role Labeling with Unified Labels". Abstract: We present POLYGLOT, a semantic role labeling system capable of semantically parsing sentences in 9 different languages from 4 different language groups. A core differentiator is that this system predicts English Proposition Bank labels for all supported languages. This means that for instance a Japanese sentence will be tagged with the same labels as an English sentence with similar semantics would be. This is made possible by training the system with target language data that was automatically labeled with English PropBank labels using an annotation projection approach. We give an overview of our system, the automatically produced training data, and discuss possible applications and limitations of this work. We present a demonstrator that accepts sentences in English, German, French, Spanish, Japanese, Chinese, Arabic, Russian and Hindi and outputs a visualization of its shallow semantics.

Polyglot: Multilingual Semantic Role Labeling with Unified Labels

Yunyao Li

Transparent Machine Learning for Information Extraction: State-of-the-Art and...

Yunyao Li

Invited Talk at Modern Data Management Systems Summit on August 29-30, 2014 at Tsinghua University in Beijing, China. http://ise.thss.tsinghua.edu.cn/MDMS/English/program.jsp Abstract: Modern enterprises are increasingly relying on complex analyses on large data sets to drive business decisions. Tasks such as root cause analysis from system logs and lead generation based on social media, customer retention and digital marketing are rapidly gaining importance. These applications generally consist of three major analytic phases: text analytics, semi-structured data processing (joins, group-by, aggregation), and statistical/predictive modeling. The size of the datasets in conjunction with the complexity of the analysis necessitates large-scale distributed processing of the analytical algorithms. At IBM we are building tools and technologies based on declarative languages to support each of these analytic phases. The declarative nature of the language abstracts away the need for programmer-optimization. Furthermore, the syntax of these languages is designed to appeal to the corresponding communities. As an example for statistical modeling, we expose a high-level language with syntax similar to R -- a very popular statistical processing language. In this talk I will give an overview of some real-world big data applications we are currently working on and use that to motivate the need for declarative analytics consisting of the three major phases discussed above. I will then describe, in some detail, declarative systems for text analytics along with a discussion on speeds, feeds and comparisons.

The Power of Declarative Analytics

Yunyao Li

This is the slides used in our 3-hour tutorial at VLDB'2014. Yunyao Li, Ziyang Liu, Huaiyu Zhu: Enterprise Search in the Big Data Era: Recent Developments and Open Challenges. PVLDB 7(13): 1717-1718 (2014) Abstract: Enterprise search allows users in an enterprise to retrieve desired information through a simple search interface. It is widely viewed as an important productivity tool within an enterprise. While Internet search engines have been highly successful, enterprise search remains notoriously challenging due to a variety of unique challenges, and is being made more so by the increasing heterogeneity and volume of enterprise data. On the other hand, enterprise search also presents opportunities to succeed in ways beyond current Internet search capabilities. This tutorial presents an organized overview of these challenges and opportunities, and reviews the state-of-the-art techniques for building a reliable and high quality enterprise search engine, in the context of the rise of big data.

Enterprise Search in the Big Data Era: Recent Developments and Open Challenges

Yunyao Li

SystemT: Declarative Information Extraction

Yunyao Li

Tyler Baldwin, Yunyao Li, Bogdan Alexe, Ioana Roxana Stanoi: Automatic Term Ambiguity Detection. ACL (2) 2013: 804-809 Abstract: While the resolution of term ambiguity is important for information extraction (IE) systems, the cost of resolving each instance of an entity can be prohibitively expensive on large datasets. To combat this, this work looks at ambiguity detection at the term, rather than the instance, level. By making a judgment about the general ambiguity of a term, a system is able to handle ambiguous and unambiguous cases differently, improving through-put and quality. To address the term ambiguity detection problem, we employ a model that combines data from language models, ontologies, and topic modeling. Results over a dataset of entities from four product domains show that the proposed approach achieves significantly above baseline F-measure of 0.96.

Automatic Term Ambiguity Detection

Yunyao Li

Information Extraction --- An one hour summary

Yunyao Li

Adaptive Parser-Centric Text Normalization

Yunyao Li

Enterprise information extraction: recent developments and open challenges

Yunyao Li

Automatic suggestion of query-rewrite rules for enterprise search

Yunyao Li

More from Yunyao Li (20)

Human in the Loop AI for Building Knowledge Bases

Towards Universal Language Understanding

Explainability for Natural Language Processing

Towards Universal Language Understanding (2020 version)

Towards Universal Semantic Understanding of Natural Languages

Exploiting Structure in Representation of Named Entities using Active Learning

K-SRL: Instance-based Learning for Semantic Role Labeling

Coling poster

Coling demo

Natural Language Data Management and Interfaces: Recent Development and Open ...

Polyglot: Multilingual Semantic Role Labeling with Unified Labels

Transparent Machine Learning for Information Extraction: State-of-the-Art and...

The Power of Declarative Analytics

Enterprise Search in the Big Data Era: Recent Developments and Open Challenges

SystemT: Declarative Information Extraction

Automatic Term Ambiguity Detection

Information Extraction --- An one hour summary

Adaptive Parser-Centric Text Normalization

Enterprise information extraction: recent developments and open challenges

Automatic suggestion of query-rewrite rules for enterprise search

Recently uploaded

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Histor y of HAM Radio presentation slide

vu2urc

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Real Time Object Detection Using Open CV

Khem

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

Recently uploaded (20)

Tech Trends Report 2024 Future Today Institute.pdf

🐬 The future of MySQL is Postgres 🐘

Histor y of HAM Radio presentation slide

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Driving Behavioral Change for Information Management through Data-Driven Gree...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Powerful Google developer tools for immediate impact! (2023-24 C)

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

AWS Community Day CPH - Three problems of Terraform

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Axa Assurance Maroc - Insurer Innovation Award 2024

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Exploring the Future Potential of AI-Enabled Smartphone Processors

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Real Time Object Detection Using Open CV

Strategies for Landing an Oracle DBA Job as a Fresher

[2024]Digital Global Overview Report 2024 Meltwater.pdf

Advantages of Hiring UIUX Design Service Providers for Your Business

HTML Injection Attacks: Impact and Mitigation Strategies

An In-depth Analysis of the Effect of Text Normalization in Social Media

1. Methodology Taxonomy of Normalization Edits Tyler Baldwin and Yunyao Li IBM Research - Almaden An In-depth Analysis of the Effect of Text Normalization in Social Media Normalization For Parsing Normalization maps sentences or their tokens to their standard form. It can help downstream applications that expect the data in “clean” formats. Early approaches were very application-centric. Social media normalization has been characterized in the literature as a mapping from non-standard words to their standard form, ignoring many operations. Perfect normalization is difficult but the typical approach may not be sufficient for all applications. Should we return to an application-centric approach? Previous taxonomies do not agree on the appropriate level of granularity to examine. Several levels of granularity allow for examination of big picture and in-depth. Three applications: Syntactic parsing, named-entity recognition, text-to-speech synthesis -600 tweets, TREC Twitter data -Gold standard annotated to ideal form by human annotators -Edit types examined via ablation Normalization For NER Normalization For TTS ADDITION REPLACEMENT REMOVAL 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Coarse Granularity - Parsing F-Measure Difference BEVERB DETERMINER OTHER SUBJECT CAPITALIZATION CONTRACTION OTHER SLANG OTHER TWITTER ADDITION REPLACEMENT REMOVAL 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 Fine Granularity - NER F-Measure Difference BEVERB DETERMINER OTHER SUBJECT CAPITALIZATION CONTRACTION OTHER SLANG OTHER TWITTER ADDITION REPLACEMENT REMOVAL 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 Fine Granularity - Parsing F-Measure Difference ADDITION REPLACEMENT REMOVAL 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 Coarse Granularity - NER F-Measure Difference BEVERB DETERMINER OTHER SUBJECT CAPITALIZATION CONTRACTION OTHER SLANG OTHER TWITTER ADDITION REPLACEMENT REMOVAL 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Fine Granularity - TTS F-Measure Difference ADDITION REPLACEMENT REMOVAL 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Coarse Granularity - TTS F-Measure Difference Edit Insertion Replacement Removal Punctuation Word Word Word Subj. Be Det. Other Slang Cont. Cap. Other Twitter Other Punctuation Punctuation Parsing is heavily impacted by most normalization operations. Entity recognition is mostly dependent on replacement edits. Capitalization correction is important as well. Speech synthesis is heavily impacted by normalization generally, but handling domain-specific terms is critical. @someGuy idk kinda wanna get NEW ipad @someGuy I don't know kind of want to get NEW ipad I don't know, I kind of want to get a new iPad. Typical Ideal

An In-depth Analysis of the Effect of Text Normalization in Social Media

Recommended

Recommended

More Related Content

More from Yunyao Li

More from Yunyao Li (20)

Recently uploaded

Recently uploaded (20)

An In-depth Analysis of the Effect of Text Normalization in Social Media