Cleaning data before BPE

[marianelamin]

• Create a set of data cleaning methods

  • Set to lowercase
  • Change á é í ó ú -> aeiou and ñ -> gn
  • Remove Emojis
  • Remove mentions
  • Remove hashtags
  • Remove links
  • Remove punctuation: . - : , ?
  • Remove extra spaces
  • Remove spaces before and after string content.
  • Stemming ?

• Create the Cleaning class. The idea is that each method above belongs to the cleaning class. This can be part of the c4v nlp cleaning library.

[dieko95]

@marianelamin Probably we should consider how we are going to treat emojis. As for the byte pair encoding, I'm not sure if it makes sense to include emojis.

Also, I would remove URLs.

Finally, I'm not 100% sure if we should use stemming. CC @Edilmo

[marianelamin]

When removing punctuation, our current method only removes . - : , ?. Should we also remove ! and " or '?
@Edilmo @dieko95