Python Unicode HowTo

>>> unicode('abcdef')

u'abcdef'

>>> s = unicode('abcdef')

>>> type(s)

<type 'unicode'>

>>> unicode('abcdef' + chr(255))    

Traceback (most recent call last):

...

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6:

ordinal not in range(128)

This is a classic python unicode pain point! Consider the following:

a = u'bats\u00E0'

print a

 => batsà

All good so far, but if we call str(a), let's see what happens:

str(a)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 4: ordinal not in range(128)

Oh dip, that's not gonna do anyone any good! To fix the error, encode the bytes explicitly with .encode and tell python what codec to use:

a.encode('utf-8')

 => 'bats\xc3\xa0'

print a.encode('utf-8')

 => batsà

The issue is that when you call str(), python uses the default character encoding to try and encode the bytes you gave it, which in your case are sometimes representations of unicode characters. To fix the problem, you have to tell python how to deal with the string you give it by using .encode('whatever_unicode'). Most of the time, you should be fine using utf-8.

I found elegant work around for me to remove symbols and continue to keep string as string in follows:

yourstring = yourstring.encode('ascii', 'ignore').decode('ascii')

It's important to notice that using the ignore option is dangerous because it silently drops any unicode(and internationalization) support from the code that uses it, as seen here (convert unicode):

>>> u'City: Malmö'.encode('ascii', 'ignore').decode('ascii')

'City: Malm'

For utf-8, it's sufficient to do: yourstring = yourstring.encode('utf-8', 'ignore').decode('utf-8') 

References:

https://docs.python.org/2.7/howto/unicode.html