Python – Preprocessing of Tamil Text
Last Updated :
27 Sep, 2021
Preprocessing is the major part of Natural Language Processing. In order to classify any text with high accuracy, cleaned data plays a major role. So, the first step in NLP before analyzing or classifying is preprocessing of data. Many python libraries support preprocessing for the English language. But for the Tamil Language, there are very few preprocessing libraries available. Here is an example of a few preprocessing techniques for Tamil text.
The preprocessing techniques involved in this article are
- Punctuation removal
- Tokenization
- Stop words removal
Punctuation removal:
Python3
import string
print (string.punctuation)
|
Output:
!”#$%&'()*+,-./:;<=>?@[\]^_`{|}~
If there are any of the above punctuations in the text, they will be removed after preprocessing. This can be removed by using the python string module.
Python3
def punctuation_remove(text_data):
punctuation = "".join([t for t in text_data if t not in string.punctuation])
return punctuation
punctuation_removed = punctuation_remove( "வெற்றி *பெற வேண்டும், என்ற பதற்றம் ^இல்லாமல் _இருப்பது தான் 'வெற்றி பெறுவதற்கான சிறந்த வழி." )
print (punctuation_removed)
|
Output:
வெற்றி பெற வேண்டும் என்ற பதற்றம் இல்லாமல் இருப்பது தான் வெற்றி பெறுவதற்கான சிறந்த வழி
Explanation: All the punctuations in the given text are removed.
Tokenization:
Tokenization is nothing but splitting each word in a sentence into a token which will be used for further classification. To convert a text into tokens, a regular expression module in python is used.
Python3
import re
def tokenization(text_data):
tokens_text = re.split( ' ' ,text_data)
return tokens_text
tokenized_text = tokenization(punctuation_removed)
print (tokenized_text)
|
Output:
[‘வெற்றி’, ‘பெற’, ‘வேண்டும்’, ‘என்ற’, ‘பதற்றம்’, ‘இல்லாமல்’, ‘இருப்பது’, ‘தான்’, ‘வெற்றி’, ‘பெறுவதற்கான’, ‘சிறந்த’, ‘வழி’]
Explanation: All the words in a sentence are split into tokens.
Stop words removal:
Stop words are frequently used words in a language. These words are unnecessary for the meaning of the sentence. Removal of stop words can be done by using the NLTK package in python. The NLTK package supports many languages like English, French, German, Finnish, Italian, etc but not the Tamil language. So, download the stop words for Tamil in the given link – Github Link and name the file as tamil and place it in the below location in your system – “ ….\AppData\Roaming\nltk_data\corpora\stopwords“
After this process, the NLTK package supports Tamil stop words also
Python3
import nltk
stopwords = nltk.corpus.stopwords.words( 'tamil' )
def stopwords_remove(text_data):
removed = [s for s in text_data if s not in stopwords]
return removed
stopwords_removed = stopwords_remove(tokenized_text)
print (stopwords_removed)
|
Output:
[‘வெற்றி’, ‘பெற’, ‘பதற்றம்’, ‘இல்லாமல்’, ‘இருப்பது’, ‘வெற்றி’, ‘பெறுவதற்கான’, ‘சிறந்த’, ‘வழி’]
Explanation: The stop words ‘வேண்டும்’, ‘என்ற’ and ‘தான்’ are removed.
Like Article
Suggest improvement
Share your thoughts in the comments
Please Login to comment...