How to develop Natural language processing tools for a minoritized variety such as Galician-Portugues

1. An approach to open source NLP tools
for Galician as a minoritized variety of
Portuguese in Spain
José Ramom Pichel Campos
R&D Director
imaxin|software
www.imaxin.com

2. 1. Imaxin|software
2. Global Languages/Minority/Endangered/Minoritized
Languages
3. What are the most important challenges of
Minority/Endangered/Minoritized Languages in relation to
develop Natural Language Processing tools?
4. Galician-Portuguese Language as a sample of
Minority/Minoritized language to develop open
source/proprietary NLP tools
www.imaxin.com

3. Minority, Endangered and Minoritized are not global
languages, so ....
What should we take into account to develop Natural language
processing tools for Minority, Endangered and Minoritized
languages?
Galician as an example
www.imaxin.com

4. Generally, when Computational Scientists want to approach
Languages and Computers...They are thinking of....Computers
www.imaxin.com

5. ....But languages are spoken by people.....
www.imaxin.com

6. ..... in Societies....
www.imaxin.com

7. So, before developing NLP tools for Endangered Languages you
should think of Language in a Society
www.imaxin.com

8. Sociolinguistics is the descriptive study of the effect of any and all
aspects of society, including cultural norms, expectations, and
context, on the way language is used, and the effects of language
use on society.
www.imaxin.com

9. Basic issues of Sociolinguistics to focus on better
developments of NLP tools for languages
What about Global Languages?
What about Minority Languages?
What about Endangered Languages?
What about Minoritized Languages?
And finally, when we approach to a language, is it a different
languages or a variety of a language?
www.imaxin.com

10. Global Languages (by Wikipedia)
A world language is a language spoken
internationally and which is learned by many
people as a second language.
A world language is not only characterized by the
number of speakers (native or second language
speakers), but also by its geographical
distribution, international organizations and in
diplomatic relations.
www.imaxin.com

11. Global Languages
"A language is a dialect with an army and navy"
Sociolinguist and Yiddish scholar Max Weinreich
www.imaxin.com

12. www.imaxin.com
What about next Global Languages?

13. www.imaxin.com
Is Mandarin Chinese an easy language to learn?
é o chinês mandarim uma linguagem fácil de aprender?
是中国普通话的容易学的语言？

14. www.imaxin.com
The historical reason for this is the period of expansionist European imperialism
and colonialism (and the more powerful economies and armies in the world)
(English, French, Spanish, Portuguese, Dutch, etc.)

15. "A language is a dialect with an army and navy"
Sociolinguist and Yiddish scholar Max Weinreich
Language = dialect + army + navy
Dialect = Language – (army + navy)
Dialect: Minority > Minoritized > Endangered
www.imaxin.com

16. Minority Languages (by Wikipedia)
A minority language is a language spoken by a minority of the
population of a territory. Such people are termed linguistic minorities
or language minorities.
www.imaxin.com

17. Minority Languages
www.imaxin.com

18. Endangered Languages (by Wikipedia)
An endangered language is a language that is at risk of falling out of
use as its speakers die out or shift to speaking another language.
Language loss occurs when the language has no more native
speakers, and becomes a "dead language". If eventually no one
speaks the language at all, it becomes an "extinct language".
www.imaxin.com

19. Endangered Languages (by Wikipedia)
........While languages have always become extinct throughout
human history, they are currently disappearing at an accelerated rate
due to the processes of globalization and neo-colonialism, where the
economically powerful languages dominate other languages.
http://www.voanews.com/content/rosetta-project-preserves-key-to-
endangered-languages/1713317.html
www.imaxin.com

20. Minoritized language
Minoritized language is a term that refers to sociolinguistic languages
that have suffered marginalization, persecution or even banning at
some point in their history. It's therefore a concept that highlights the
presence of an enforcement action leading to a cut in use.
Minority and Minoritized Language are not synonymous
www.imaxin.com

21. Minoritized language
www.imaxin.com

22. Different languages or different varieties of the same language?
As we know, “A language is a dialect with an army and navy" is a
quip about the arbitrariness of the distinction between a dialect and a
language. It points out the influence that social and political
conditions can have over a community's perception of the status of a
language or dialect.
The adage was popularized by the sociolinguist and Yiddish scholar
Max Weinreich, who heard it from a member of the audience at one
of his lectures.
www.imaxin.com

23. Different languages or different varieties of the same language?
www.imaxin.com

24. www.imaxin.com
Natural Language Processing Tools for any kind of language
Spell-checkers, Grammar-Checkers, Machine Translation,
Lemmatizer, Morphological Analyzer, POSTagger, etc.

25. What are the most important challenges of
Minority/Endangered/Minoritized Languages in relation to
develop Natural Language Processing tools?
www.imaxin.com

26. 1. Is there a stable written standard?
2. Is there a prescriptive authority of written standard language?
3. What is our target? Kids? Old-people?
4. Kind of Language (Minoritized, Minority and Distance
between other Languages)
5. Is there a recognized grammar?
6. Are there enough monolingual and bilingual corpus ?
www.imaxin.com

27. 1. Is there a stable written standard?
A written language is the representation of a language by means of a
writing system. Written language is an invention in that it must be
taught to children; children will pick up spoken language (oral or sign)
by exposure without being specifically taught.
A standard language (also standard dialect or standardized dialect) is
a language variety used by a group of people in their public
discourse.
www.imaxin.com

28. 1. Is there a stable written standard?
Nynorsk and Bokmål
Nynorsk was developed by the linguist Ivar Aasen in the 1850s,
based on rural, spoken Norwegian, rather than the cultured,
Danish-influenced Norwegian spoken in cities. Its first official
codification was in 1901, was given the name Nynorsk in 1929, and
has been used officially (alongside Bokmål) since 1938.
www.imaxin.com

29. 1. Is there a stable written standard?
Nationalist and Reintegrationist Galician
The nationalist considers Galician and Portuguese to be two distinct
languages, despite the fact of the two being closely related.
Nationalist favour differentiated rules of writing and spelling between
Galician and Portuguese. In this fashion, Galician spelling follows the
model of Spanish orthography. This view is held by the majority of
public and Government organizations. Its standard norm, the
"NOMIGa", is elaborated by the Real Academia Galega (Royal
Galician Academy) and the Instituto da Língua Galega (Institute for
Galician Language).
www.imaxin.com

30. 2. Is there an official authority of written standard language?
www.imaxin.com

31. 3. What is our target? Children? Old-people?
www.imaxin.com

32. 3. Children
Technological skills
Educational skills in language
standard
Very often, they don't speak their
grandparents' language on a
daily basis
www.imaxin.com

33. 3. Old people
No technological skills
No educational skills in language
standard
They don't speak standard
language
They are ashamed of their own
language (it is useless, they
think)
www.imaxin.com

34. 4. Kind of Language (Minoritized, Minority and Distance
between other Languages, Endangered)
MT MN Example
0 1 Luxembourgish
1 0 Galician-Portuguese, Catalan-Valencian
1 0 Catalan
1 1 Galician-Spanish, Valencian, Aragonese, Friulan, Asturian
www.imaxin.com

35. 5. Is there a recognized grammar?
www.imaxin.com

36. 6. Is there a good enough monolingual and bilingual corpus ? Is
it open source?
The Europarl parallel corpus is extracted from the proceedings of the
European Parliament. It includes versions in 21 European languages:
Romanic (French, Italian, Spanish, Portuguese, Romanian),
Germanic (English, Dutch, German, Danish, Swedish), Slavik
(Bulgarian, Czech, Polish, Slovak, Slovene), Finni-Ugric (Finnish,
Hungarian, Estonian), Baltic (Latvian, Lithuanian), and Greek.
http://www.statmt.org/europarl/
www.imaxin.com

37. 6. Is there a good enough monolingual and bilingual corpus? Is
it open source?
Tesouro galego-português
http://ilg.usc.es/Tesouro/pt/
www.imaxin.com

38. GALIZA, as a sample to learn to develop nlp tools for
minority/endangered/minoritized languages
www.imaxin.com

39. www.imaxin.com
Political Map versus Languages Map

40. www.imaxin.com
Our language: Galician (Globally known as Portuguese)
https://www.youtube.com/watch?v=RPRxAcckmUA

41. Galiza is a sociolinguistics lab to develop NLP tools
www.imaxin.com

42. 1. Is there a stable written standard?
2. Is there a prescriptive authority of written standard language?
3. What is our target? Kids? Old-people?
4. Kind of Language (Minoritized, Minority and Distance
between other Languages)
5. Is there a recognized grammar?
6. Are there enough monolingual and bilingual corpus ?
www.imaxin.com

43. Galiza, as a sociolinguistics lab implies another point of view on
how to develop natural language processing tools
Galician-Spanish: Minoritized + Minority Language
“Vou facer o camiño de Santiago. Ao chegar a Galicia podes ver
polas montañas moitos carballos.”
Galician-Portuguese: Minoritized Language
“Vou fazer o caminho de Santiago. Ao chegar à Galiza podes ver
polas montanhas muitos carvalhos.”
www.imaxin.com

44. Galician-Spanish: Minoritized + Minority Language
http://www.xunta.es/linguagalega/ferramentas_informaticas
Features
You have to develop software from scratch (high investment).
In case of open source, galician is high-dependent on volunteers and
Public investment.
Huge diversity on terminology
Private software depends on strategy of big companies (Microsoft,
Apple, Sun, etc.)
Interferences with Spanish
www.imaxin.com

45. Spell-Checkers
Minoritized errors + Spell-checker errors
http://www.xunta.es/linguagalega/galgo
www.imaxin.com

46. Monolingual and bilingual corpora
http://sli.uvigo.es/RILG/
http://webs.uvigo.es/sli/recursos_en.html
www.imaxin.com

47. Monolingual dictionaries
http://www.realacademiagalega.org/dicionario
www.imaxin.com

48. Games and NLP
http://portaldaspalabras.org/
www.imaxin.com

49. GalNET: Ontology of Galician-Spanish
http://sli.uvigo.es/galnet/galnet_var.php?ili=ili-30-12090890-n
www.imaxin.com

50. Galician-Portuguese: Minoritized Language
http://gramatica.usc.es/~gamallo/
http://www.estraviz.org/
Features
You have just to customize software from Portuguese state-of-art, in
case of necessity
In case of open source, galician is less-dependent on volunteers and
Public investment.
Less diversity on terminology (based on Portuguese and Brazillian
choices)
Big companies (Microsoft, Apple, Sun, etc.) are more interested in
localize galician variety
Interferences with Spanish
Increase more open source because of you are using open source
from Portugal and Brazil www.imaxin.com

51. Open source Spell-checker
http://extensions.libreoffice.org/extension-center/corrector-orto
grafico-para-galego
www.imaxin.com

52. Open Source Non-sexist Grammar Checker
http://www.exeria.net/que.php
www.imaxin.com

53. Open Source Grammar Checker
http://wiki.mancomun.org/index.php/Golfi
%C3%B1o._Corrector_gramatical_para_OpenOffice.org
www.imaxin.com

54. Natural language processing tools (Galician-Portuguese)
Open source
http://www-nlp.stanford.edu/links/statnlp.html
http://gramatica.usc.es/~gamallo/
www.imaxin.com

55. FLIP Portuguese spell-checker and Galician
www.imaxin.com

56. Machine Translation
Open Source Machine Translation
(Apertium and Matxin)
http://www.opentrad.com
Google Translate
https://translate.google.com/
www.imaxin.com

57. Machine Translation
www.imaxin.com

58. Opentrad (Apertium)
RBMT
www.imaxin.com

59. Moses (SMT)
www.imaxin.com

60. ILP Paz Andrade
www.imaxin.com

61. Trip to Endangered Languages
This is a blog about a trip through Endangered Languages in Europe.
I'm convinced that Human Language Technologies can save them
from disappearance.
http://tripendangeredlanguages.wordpress.com/
www.imaxin.com

62. Thanks a lot!
Obrigado!
@jramompichel
jramompichel@imaxin.com
https://www.youtube.com/watch?v=l3zzn4k9dRE
www.imaxin.com

63. salgueiriños de abaixo nº11 L6
15703 Santiago de Compostela (A Coruña)
voz. +34 981 554 068 imaxin@imaxin.com
www.imaxin.com
Facebook: www.facebook.com/imaxinsoftware
Twitter: @imaxinsoftware
www.imaxin.com