Python Unicode Encode Error – Be on the Right Side of Change

Summary: The UnicodeEncodeError generally occurs while encoding a Unicode string into a certain coding. Only a limited number of Unicode characters are mapped to strings. Thus, any character that is not-represented / mapped will cause the encoding to fail and raise UnicodeEncodeError. To avoid this error use the encode(utf-8) and decode(utf-8) functions accordingly in your code.

You might be using handling an application code that needs to deal with multilingual data or web content that has plenty of emojis and special symbols. In such situations, you will possibly come across numerous problems relating to Unicode data. But python has well-defined options to deal with Unicode characters and we shall be discussing them in this article.

What is Unicode?

Unicode is a standard that facilitates character encoding using variable bit encoding. I am sure, you must have heard of ASCII if you are into the world of computer programming. ASCII represents 128 characters while Unicode defines 2²¹ characters. Thus, Unicode can be regarded as a superset of ASCII. If you are interested in having an in-depth look at Unicode, please follow this link.
Click on Unicode:- U+1F40D to find out what it represents! (Try it!!!?)

What is a UnicodeEncodeError?

The best way to grasp any concept is to visualize it with an example. So let us have a look at an example of the UnicodeEncodeError.

u = 'é'
print("Integer value for é: ", ord(u))
print("Converting the encoded value of é to Integer Equivalent: ", chr(233))
print("UNICODE Representation of é: ", u.encode('utf-8'))
print("ASCII Representation of é: ", u.encode('ascii'))

Output

Integer value for é:  233
Converting the encoded value of é to Integer Equivalent:  é
UNICODE Representation of é:  b'\xc3\xa9'
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    print("ASCII Representation of é: ",u.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)

In the above code, when we tried to encode the character é to its Unicode value we got an output but while trying to convert it to the ASCII equivalent we encountered an error. The error occurred because ASCII only allows 7-bit encoding and it cannot represent characters outside the range of [0..128].

You now have an essence of what the UnicodeEncodeError looks like. Before discussing how we can avoid such errors, I feel that there is a dire need to discuss the following concepts:

Encoding and Decoding

The process of converting human-readable data into a specified format, for the secured transmission of data is known as encoding. Decoding is the opposite of encoding that is to convert the encoded information to normal text (human-readable form).

In Python,

encode() is an inbuilt method used for encoding. Incase no encoding is specified, UTF-8 is used as default.
decode() is an inbuilt method used for decoding.

Example:

u = 'Πύθωνος'
print("UNICODE Representation of é: ", u.encode('utf-8'))

Output:

UNICODE Representation of é:  b'\xce\xa0\xcf\x8d\xce\xb8\xcf\x89\xce\xbd\xce\xbf\xcf\x82'

The following diagram should make things a little easier:

Codepoint

Unicode maps the codepoint to their respective characters. So, what do we mean by a codepoint?

Codepoints are numerical values or integers used to represent a character.
The Unicode code point for é is U+00E9 which is integer 233. When you encode a character and print it, you will generally get its hexadecimal representation as an output instead of its binary equivalent (as seen in the examples above).
The byte sequence of a code point is different in different encoding schemes. For eg: the byte sequence for é in UTF-8 is \xc3\xa9 while in UTF-16 is \xff\xfe\xe9\x00.

Please have a look at the following program to get a better grip on this concept:

u = 'é'
print("INTEGER value for é: ", ord(u))
print("ENCODED Representation of é in UTF-8: ", u.encode('utf-8'))
print("ENCODED Representation of é in UTF-16: ", u.encode('utf-16'))

Output

INTEGER value for é:  233
ENCODED Representation of é in UTF-8:  b'\xc3\xa9'
ENCODED Representation of é in UTF-16:  b'\xff\xfe\xe9\x00'

Now that we have an overview of Unicode and UnicodeEncodeError, let us discuss how we can deal with the error and avoid it in our program.

➥ Problem: Given a string/text to be written in a text File; how to avoid the UnicodeEncodeError and write given text in the text file.

Example:

f = open('demo.txt', 'w')
f.write('να έχεις μια όμορφη μέρα')
f.close()

Output:

Traceback (most recent call last):
  File "uniError.py", line 2, in <module>
    f.write('να έχεις μια όμορφη μέρα')
  File "C:\Users\Shubham-PC\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>

✨ Solution 1: Encode String Before Writing To File And Decode While Reading

You cannot write Unicode to a file directly. This will raise an UnicodeEncodeError. To avoid this you must encode the Unicode string using the encode() function and then write it to the file as shown in the program below:

text = u'να έχεις μια όμορφη μέρα'
# write in binary mode to avoid TypeError
f = open('demo.txt', 'wb')
f.write(text.encode('utf8'))
f.close()
f = open('demo.txt', 'rb')
print(f.read().decode('utf8'))

Output:

να έχεις μια όμορφη μέρα

✨ Solution 2: Open File In utf-8

If you are using Python 3 or higher, all you need to do is open the file in utf-8, as Unicode string handling is already standardized in Python 3.

text = 'να έχεις μια όμορφη μέρα'
f = open('demo2.txt', 'w', encoding="utf-8")
f.write(text)
f.close()

Output:

✨ Solution 3: Using The Codecs Module

Another approach to deal with the UnicodeEncodeError is using the codecs module.

Let us have a look at the following code to understand how we can use the codecs module:

import codecs

f = codecs.open("demo3.txt", "w", encoding='utf-8')
f.write("να έχεις μια όμορφη μέρα")
f.close()

Output:

✨ Solution 4: Using Python’s unicodecsv Module

If you are dealing with Unicode data and using a csv file for managing your data, then the unicodecsv module can be really helpful. It is an extended version of Python 2’s csv module and helps the user to handle Unicode data without any hassle.

Since the unicodecsv module is not a part of Python’s standard library, you have to install it before using it. Use the following command to install this module:

$ pip install unicodecsv

Let us have a look at the following example to get a better grip on the unicodecsv module:

import unicodecsv as csv

with open('example.csv', 'wb') as f:
    writer = csv.writer(f, encoding='utf-8')
    writer.writerow(('English', 'Japanese'))
    writer.writerow((u'Hello', u'こんにちは'))

Output:

Conclusion

In this article, we discussed some of the important concepts regarding Unicode character and then went on to learn about the UnicodeEncodeError and finally discussed the methods that we can use to avoid it. I hope by the end of this article you can handle Unicode characters in your python code with ease.

Please subscribe and stay tuned for more interesting articles!

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

🚀 If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!