Each year, volunteers from all over the world, work on improvements to the Python Language. The official Python 3.9 version was released on October 5, 2020. This version is an improvement made to Python 3.8. The official, detailed explanation of these improvements is available at the Python Website. This blog article attempts to explain these improvements to a Python newbie.
Let’s dive into the new features added to Python 3.9.
Note: All the Python examples in this article are Python 3.9.0b5 verified.
Feature 1: Union Operators to Simplify Dict Updates (PEP 584)
The dictionary is a basic data-structure built into the Python Language. It is a set of key-value pairs. It is a rule that the key be a hashable value and hence unique, within one dictionary.
The reader is likely familiar with several methods to merge dictionaries. Two commonly used methods are shown below.
$ python Python 3.9.0b5 (default, Oct 19 2020, 11:11:59) >>> >>> ## Company1 and Company2 are going to be merged. Payroll has to >>> ## merge the employee salary data. >>> company1 = { ... "Alice" : 200, ... "Bob" : 110, ... "Ryan" : 100, ... "Dup" : 60 ... } >>> company2 = { ... "Alex" : 80, ... "John" : 90, ... "Steve" : 102, ... "Dup" : 40 ... } >>> >>> ## Note, duplicate entries of company2 will override those of company1. >>> merged1 = {**company1, **company2} >>> merged1 {'Alice': 200, 'Bob': 110, 'Ryan': 100, 'Dup': 40, 'Alex': 80, 'John': 90, 'Steve': 102} >>> >>> ## Note, duplicate entries of company1 will override those of company2. >>> merged2 = {**company2, **company1} >>> merged2 {'Alex': 80, 'John': 90, 'Steve': 102, 'Dup': 60, 'Alice': 200, 'Bob': 110, 'Ryan': 100} >>> >>> ## Note, duplicate entries of company2 will override those of company1. >>> ## Here the merged3 dictionary is operated in-place. >>> merged3 = company1.copy() >>> for key, value in company2.items(): ... merged3[key] = value ... >>> merged3 {'Alice': 200, 'Bob': 110, 'Ryan': 100, 'Dup': 40, 'Alex': 80, 'John': 90, 'Steve': 102} >>> >>> ## Note, duplicate entries of company1 will override those of company2. >>> ## Here the merged4 dictionary is operated in-place. >>> merged4 = company2.copy() >>> for key, value in company1.items(): ... merged4[key] = value ... >>> merged4 {'Alex': 80, 'John': 90, 'Steve': 102, 'Dup': 60, 'Alice': 200, 'Bob': 110, 'Ryan': 100}
Python 3.9 (PEP 584) introduces two new elegant ways to merge dictionaries. The union (|
) operator merges two dictionaries. Whereas the in-place union (|=
) will update the dictionary in place.
Consider the following examples which use the same dictionaries as shown above.
$ python Python 3.9.0b5 (default, Oct 19 2020, 11:11:59) >>> >>> ## Company1 and Company2 are going to be merged. Payroll has to >>> ## merge the employee salary data. >>> company1 = { ... "Alice" : 200, ... "Bob" : 110, ... "Ryan" : 100, ... "Dup" : 60 ... } >>> company2 = { ... "Alex" : 80, ... "John" : 90, ... "Steve" : 102, ... "Dup" : 40 ... } >>> >>> ## Note, duplicate entries of company2 will override those of company1. >>> ## Note the use of the union(|) Dictionary operator. >>> merged1 = company1 | company2 >>> merged1 {'Alice': 200, 'Bob': 110, 'Ryan': 100, 'Dup': 40, 'Alex': 80, 'John': 90, 'Steve': 102} >>> >>> ## Note, duplicate entries of company1 will override those of company2. >>> merged2 = company2 | company1 >>> merged2 {'Alex': 80, 'John': 90, 'Steve': 102, 'Dup': 60, 'Alice': 200, 'Bob': 110, 'Ryan': 100} >>> >>> ## Note, duplicate entries of company2 will override those of company1. >>> ## Note the use of the in-place union(|=) Dictionary operator. Here the >>> ## merged3 dictionary is operated in-place. >>> merged3 = company1.copy() >>> merged3 |= company2 >>> merged3 {'Alice': 200, 'Bob': 110, 'Ryan': 100, 'Dup': 40, 'Alex': 80, 'John': 90, 'Steve': 102} >>> >>> ## Note, duplicate entries of company1 will override those of company2. >>> ## Here the merged4 dictionary is operated in-place. >>> merged4 = company2.copy() >>> merged4 |= company1 >>> merged4 {'Alex': 80, 'John': 90, 'Steve': 102, 'Dup': 60, 'Alice': 200, 'Bob': 110, 'Ryan': 100} >>>
Feature 2: Intuitive Type Hints for Generics (PEP 585)
In Pre-Python 3.9, it was a tad bit tedious to add type hints for generic types like lists or dictionaries. Such generic types needed a parallel hierarchy of type hints. One had to use the typing module as shown in the following examples. This was a roundabout way to provide type hints.
## typehints.py from typing import List, Dict ## numbers1 is appropriately a list of float values as is specified ## in the type hint 'List[float]' numbers1: List[float] = [2.9, 3.9, 4.9] ## dict1 is a dictionary with keys specified by strings and values ## specified by integers. This is specified by the type hint ## 'Dict[str, int] dict1: Dict[str, int] = {"one": 1, "two": 2} print(__annotations__)
$ python -V Python 3.8.5 $ python typehints.py {'numbers1': typing.List[float], 'dict1': typing.Dict[str, int]}
Python 3.9 (PEP 585) made it simpler and straightforward to specify type hints for generic types. The following example shows how!
## typehints.py ## numbers1 is appropriately a list of float values as is specified ## in the type hint 'list[float]' numbers1: list[float] = [2.9, 3.9, 4.9] ## dict1 is a dictionary with keys specified by strings and values ## specified by integers. This is specified by the type hint ## 'dict[str, int] dict1: dict[str, int] = {"one": 1, "two": 2} print(__annotations__)
Note the use of list[]
instead of List[]
or dict[]
instead of Dict[]
. One does not need to remember to include the typing module anymore. There is no parallel hierarchy anymore, either. The type hints themselves are more cleaner and intuitive.
$ python typehints.py {'numbers1': list[float], 'dict1': dict[str, int]}
Feature 3: Decorator Flexibility (PEP 614)
Pre-Python 3.9, required a decorator to be a named, callable object, such as function or class objects. PEP 614 loosens this grammar or syntax to allow a decorator to be any callable expression. The old decorator syntax was not generally considered limiting in the Python community. The main motivating use case for the PEP 614 enhancement are callbacks in the GUI Frameworks. The following example shows the limitation and the workaround for Pre-Python 3.9.
## deco.py from functools import wraps as func_wrap """ Imagine that the user wants to apply 3 different decorators to a specified function(such as decorate_this_func(), below). Each decorator is chosen based on the user's input. Pre-Python3.9, the user would have to do the following... """ ## This is the first decorator. It prints the strings "prefix1" and ## "suffix1", before and after the user provided string, respectively. ## This user provided string is returned by the decorated function i.e. ## decorate_this_func(). def decorator1(func): ## Note here that the decorator func_wrap() is used when one needs ## to use the arguments which have been originally provided to the ## decorated function (aka decorate_this_func()) @func_wrap(func) def decorator1_does_this(*args, **kwargs): ## The following 3 lines are turned into a one-liner, below. ## val1 = 'prefix {0} suffix' ## val2 = func(*args, **kwargs) ## val3 = val1.format(val2) ## return val3 return 'prefix1 {0} suffix1'.format(func(*args, **kwargs)) return decorator1_does_this ## This is the second decorator. It prints the strings "prefix2" and ## "suffix2", before and after the user provided string, respectively. ## This user provided string is returned by the decorated function i.e. ## decorate_this_func(). def decorator2(func): @func_wrap(func) def decorator2_does_this(*args, **kwargs): return 'prefix2 {0} suffix2'.format(func(*args, **kwargs)) return decorator2_does_this ## This is the third decorator. It prints the strings "prefix3" and ## "suffix3", before and after the user provided string, respectively. ## This user provided string is returned by the decorated function i.e. ## decorate_this_func(). def decorator3(func): @func_wrap(func) def decorator3_does_this(*args, **kwargs): return 'prefix3 {0} suffix3'.format(func(*args, **kwargs)) return decorator3_does_this ## The DECORATOR_DICT associates a string key with a decorator function. DECORATOR_DICT = {"dec1": decorator1, "dec2": decorator2, "dec3": decorator3} ## The user is asked for input. This allows them to choose the ## appropriate decorator. Note the user provides a string key. chosen_decorator_key = input(f"Choose your decorator key ({', '.join(DECORATOR_DICT)}): ") ## Pre-Python3.9, one could only use function or class objects to specify ## decorators for a function(i.e. the decoratee aka function to be ## decorated). To have the ability to choose multiple decorators, the ## user would have to apply the following workaround, to get a handle ## for the decorator function or class object. HANDLE_TO_CHOSEN_DECORATOR = DECORATOR_DICT[chosen_decorator_key] @HANDLE_TO_CHOSEN_DECORATOR def decorate_this_func(str_arg): return 'Use {0} to decorate this sentence'.format(str_arg) ## key_string is simply used to show how arguments can be passed thru ## decorators. key_string = chosen_decorator_key print(decorate_this_func(key_string)) """ The result is as follows... $ python -V Python 3.8.5 $ python deco.py Choose your decorator key (dec1, dec2, dec3): dec1 prefix1 Use dec1 to decorate this sentence suffix1 $ python deco.py Choose your decorator key (dec1, dec2, dec3): dec2 prefix2 Use dec2 to decorate this sentence suffix2 $ python deco.py Choose your decorator key (dec1, dec2, dec3): dec3 prefix3 Use dec3 to decorate this sentence suffix3 $ """
The workaround shown above is tedious but not really painful. The user has to indirectly extract and provide the decorator handle. However, the PEP 614 enhancement makes the solution very simple and elegant. The user now uses a handle to the decorator dictionary itself. The user provides a decorator key from the console, as input. This key extracts the specific decorator function handle, from the decorator dictionary. The following example shows the use of the enhancement.
## deco.py from functools import wraps as func_wrap """ Imagine again that the user wants to apply 3 different decorators to a specified function(such as decorate_this_func(), below). Each decorator is chosen based on the user's input. In Python3.9, the user does the following... """ ## This is the first decorator. It prints the strings "prefix1" and ## "suffix1", before and after the user provided string, respectively. ## This user provided string is returned by the decorated function i.e. ## decorate_this_func(). def decorator1(func): ## Note here that the decorator func_wrap() is used when one needs ## to use the arguments which have been originally provided to the ## decorated function (aka decorate_this_func()) @func_wrap(func) def decorator1_does_this(*args, **kwargs): return 'prefix1 {0} suffix1'.format(func(*args, **kwargs)) return decorator1_does_this ## This is the second decorator. It prints the strings "prefix2" and ## "suffix2", before and after the user provided string, respectively. ## This user provided string is returned by the decorated function i.e. ## decorate_this_func(). def decorator2(func): @func_wrap(func) def decorator2_does_this(*args, **kwargs): return 'prefix2 {0} suffix2'.format(func(*args, **kwargs)) return decorator2_does_this ## This is the third decorator. It prints the strings "prefix3" and ## "suffix3", before and after the user provided string, respectively. ## This user provided string is returned by the decorated function i.e. ## decorate_this_func(). def decorator3(func): @func_wrap(func) def decorator3_does_this(*args, **kwargs): return 'prefix3 {0} suffix3'.format(func(*args, **kwargs)) return decorator3_does_this ## The DECORATOR_DICT associates a string key with a decorator function. DECORATOR_DICT = {"dec1": decorator1, "dec2": decorator2, "dec3": decorator3} ## The user is asked for input. This allows them to choose the ## appropriate decorator. Note the user provides a string key. chosen_decorator_key = input(f"Choose your decorator key ({', '.join(DECORATOR_DICT)}): ") ## In Python3.9(PEP-614), the decorator syntax has been loosened up ## to mean any function or class objects or expression can be used ## to specify decorators for a function(i.e. the decoratee, ## aka function to be decorated). This is shown below. Note how the ## user can now use the result of the dictionary search directly, ## as a decorator. @DECORATOR_DICT[chosen_decorator_key] def decorate_this_func(str_arg): return 'Use {0} to decorate this sentence'.format(str_arg) ## key_string is simply used to show how arguments can be passed thru ## decorators. key_string = chosen_decorator_key print(decorate_this_func(key_string)) """ The result is as follows... $ python -V Python 3.9.0b5 $ python deco.py Choose your decorator key (dec1, dec2, dec3): dec1 prefix1 Use dec1 to decorate this sentence suffix1 $ python deco.py Choose your decorator key (dec1, dec2, dec3): dec2 prefix2 Use dec2 to decorate this sentence suffix2 $ python deco.py Choose your decorator key (dec1, dec2, dec3): dec3 prefix3 Use dec3 to decorate this sentence suffix3 $ “””
Feature 4: Prefix and Suffix Removal For String Objects (PEP-616)
PEP-616 added the removeprefix()
and removesuffix()
methods to the various string objects. This is an enhancement over the lstrip()
and rstrip()
methods. The removeprefix()
method removes the specified substring from the prefix of the string object. The removesuffix()
method removes the specified substring from the suffix of the string object. Consider the following examples:
$ python Python 3.9.0b5 (default, Oct 19 2020, 11:11:59) >>> >>> ## This example shows how to remove a Prefix from a string object. >>> the_bigger_string1 = "Remove this Prefix. Keep this side1." >>> print(the_bigger_string1) Remove this Prefix. Keep this side1. >>> >>> ## Now Remove the Prefix >>> remove_prefix = the_bigger_string1.removeprefix("Remove this Prefix. ") >>> print(remove_prefix) Keep this side1. >>> >>> >>> ## This example shows how to remove a Prefix from a string object. >>> the_bigger_string2 = "Keep this side2. Remove the Suffix." >>> print(the_bigger_string2) Keep this side2. Remove the Suffix. >>> >>> >>> ## Now Remove the Suffix >>> remove_suffix = the_bigger_string2.removesuffix(" Remove the Suffix.") >>> print(remove_suffix) Keep this side2. >>>
Feature 5: Annotated Type Hints For Functions And Variables (PEP-593)
The concept of function and variable annotations were introduced in Python 3.0. In the beginning, such annotations served as documentation and hints to the reader.
$ python Python 3.8.5 >>> >>> def heartrate(beats: "Total Heart-Beats as int", time: "Total time in seconds as int") -> "Beats per minute as int": ... """Calculate and return the Heart-Rate as beats-per-minute(bpm).""" ... mins = time/60 ... hr = beats/mins ... return hr ... >>> >>> heartrate.__annotations__ {'beats': 'Total Heart-Beats as int', 'time': 'Total time in seconds as int', 'return': 'Beats per minute as int'} >>> >>> heartrate(20,20) 60.0 >>> heartrate(140,120) 70.0 >>>
Note: The method heartrate()
returns a float number. The user intended to get an int. Python merrily went along and printed the results. A Type error was never flagged.
Then PEP-484 came along in Python 3.5 and proposed to use annotations for Type Hints. This allowed tools such as mypy to type check variables and function arguments. However, this enhancement served to be limiting. Users could either use general-string-documentation or type-hints and not both.
Here is a rendering of the above code with Type Hints instead of general-string-documentation.
>>> def heartrate(beats: int, time: int) -> int: ... """Calculate and return the Heart-Rate as beats-per-minute(bpm).""" ... mins = time/60 ... hr = beats/mins ... return hr ... >>> >>> heartrate.__annotations__ {'beats': <class 'int'>, 'time': <class 'int'>, 'return': <class 'int'>} >>> >>> heartrate(20,20) 60.0 >>> heartrate(140,120) 70.0 >>>
Note: Again, the method heartrate()
still returns a Float number. The user still intended to get an int. Python merrily went along and printed the results. A Type error was never flagged.
Here is what mypy has to say about the above code:
$ cat annotate.py # annotate.py def heartrate(beats: int, time: int) -> int: """Calculate and return the Heart-Rate as beats-per-minute(bpm).""" mins = time/60 hr = beats/mins return hr $ mypy annotate.py annotate.py:7: error: Incompatible return value type (got "float", expected "int") Found 1 error in 1 file (checked 1 source file)
Yes, mypy uses the Type Hints to flag an error for the return value.
Note: mypy will flag the general-string-documentation as errors. This is shown below.
$ cat annotate1.py # annotate1.py def heartrate(beats: "Total Heart-Beats", time: "Total time in seconds") -> "Beats per minute": """Calculate and return the Heart-Rate as beats-per-minute(bpm).""" mins = time/60 hr = beats/mins return hr $ mypy annotate1.py annotate1.py:3: error: Invalid type comment or annotation Found 1 error in 1 file (checked 1 source file)
So for Python 3.9, PEP-593 proposed the notion of annotated type hints(typing.Annotated). This allowed function and variable annotations to contain both, type-hints and general-string-documentation. In other words it combined the runtime(type-hints) and static (general-string-documentation) uses of annotations. A type checker such as mypy cares only about the first argument to Annotated. The interpretation of the rest of the metadata is left to the reader(user).
Here is the combined code from above—note that mypy is now happy with this. Of course all of this is now running in the Python 3.9 environment.
$ cat annotate.py ## annotate.py from typing import Annotated def heartrate(beats: Annotated[int, "Total Heart-Beats"], time: Annotated[int, "Total time in seconds"]) -> Annotated[int, "Beats per minute"]: """Calculate and return the Heart-Rate as beats-per-minute(bpm).""" mins = time/60 hr = beats/mins return int(hr) $ mypy annotate.py Success: no issues found in 1 source file
Note: The return value of the heartrate()
method has been fixed (i.e. return int(hr)
). The method properly returns an int now.
Here is how the code runs…
$ python Python 3.9.0b5 >>> >>> from typing import Annotated >>> >>> def heartrate(beats: Annotated[int, "Total Heart-Beats"], time: Annotated[int, "Total time in seconds"]) -> Annotated[int, "Beats per minute"]: ... """Calculate and return the Heart-Rate as beats-per-minute(bpm).""" ... mins = time/60 ... hr = beats/mins ... return int(hr) ... >>> >>> heartrate.__annotations__ {'beats': typing.Annotated[int, 'Total Heart-Beats'], 'time': typing.Annotated[int, 'Total time in seconds'], 'return': typing.Annotated[int, 'Beats per minute']} >>> >>> from typing import get_type_hints >>> >>> get_type_hints(heartrate) {'beats': <class 'int'>, 'time': <class 'int'>, 'return': <class 'int'>} >>> >>> get_type_hints(heartrate, include_extras=True) {'beats': typing.Annotated[int, 'Total Heart-Beats'], 'time': typing.Annotated[int, 'Total time in seconds'], 'return': typing.Annotated[int, 'Beats per minute']} >>> >>> heartrate(20,20) 60 >>> heartrate(140,120) 70 >>>
Feature 6: Powerful Python Parser (PEP-617)
Python 3.9 has reimplemented the python parser. The PEG (Parsing Expression Grammar) parser replaces the LL(1) parser. It is the coolest feature that the reader will not notice in their daily coding life. Guido Van Rossum, Python’s creator, found the PEG parser to be more powerful than the LL(1) parser. The PEG parsers did not need special hacks for hard cases. Guido’s research led to the implementation of the PEG parser (PEP-617) in Python 3.9.
Feature 7: Enhanced Time Zone Support (PEP-615)
The datetime module in Python’s standard library provides extensive support for working with dates and times. However, this module only supports the UTC time zone. Any other time zone has to be derived from and implemented on top of the abstract tzinfo base class. The third-party library dateutil is one such implementation of time zones. Python 3.9 adds a new zoneinfo standard library(PEP-615). This makes working with time zones very convenient. zoneinfo provides access to the Time Zone Database maintained by the Internet Assigned Numbers Authority (IANA). One can now get an object describing any time zone in the IANA database using zoneinfo.
Consider the following example which illustrates one way to use the zoneinfo object.
$ python Python 3.9.0b5 (default, Oct 19 2020, 11:11:59) >>> >>> ## One can now make timezone–aware timestamps by passing >>> ## the tzinfo or tz arguments to datetime functions: >>> ## Note that the datetime object includes the timezone >>> ## information. This is very useful to convert between timezones. >>> >>> from datetime import datetime >>> from zoneinfo import ZoneInfo >>> ## Imagine that there are four friends and they want to setup a >>> ## meeting at a mutually reasonable time. Each of these four >>> ## friends lives in different timezones. The US friend >>> ## takes up the task of setting up this meeting. The following >>> ## code illustrates how the ZoneInfo library is used to setup >>> ## such a meeting. >>> >>> ## The US friend uses the ZoneInfo library to find the >>> ## Current time in each of the timezones. This gives the US >>> ## friend a rough idea of the time differences in different >>> ## timezones. >>> >>> ## Current time in the Vancouver(Canada) timezone. >>> ca_friend = datetime.now(tz=ZoneInfo("America/Vancouver")) >>> ca_friend datetime.datetime(2020, 10, 21, 8, 10, 23, 212154, tzinfo=zoneinfo.ZoneInfo(key='America/Vancouver')) >>> print(ca_friend) 2020-10-21 08:10:23.212154-07:00 >>> >>> ## Current time in the New York(USA) timezone. >>> us_friend = datetime.now(tz=ZoneInfo("America/New_York")) >>> us_friend datetime.datetime(2020, 10, 21, 11, 10, 23, 215533, tzinfo=zoneinfo.ZoneInfo(key='America/New_York')) >>> print(us_friend) 2020-10-21 11:10:23.215533-04:00 >>> >>> ## Current time in the Berlin(Germany) timezone. >>> eu_friend = datetime.now(tz=ZoneInfo("Europe/Berlin")) >>> eu_friend datetime.datetime(2020, 10, 21, 17, 10, 23, 221999, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')) >>> print(eu_friend) 2020-10-21 17:10:23.221999+02:00 >>> >>> >>> ## The US friend wants to meet at a mutually reasonable >>> ## time for all the friends. So the US friend creates >>> ## a datetime object with a proposed time in the future. >>> ## They use their own US time as reference. This reference >>> ## is obtained from the ZoneInfo object. >>> lets_meet_at = datetime(2020, 10, 22, 10, 0, tzinfo=ZoneInfo("America/New_York")) >>> lets_meet_at datetime.datetime(2020, 10, 22, 10, 0, tzinfo=zoneinfo.ZoneInfo(key='America/New_York')) >>> print(lets_meet_at) 2020-10-22 10:00:00-04:00 >>> >>> ## This allows the US friend to find corresponding time >>> ## for the Canadian friend(specified as the Canadian local time). >>> ## Note, the Canadian friend is an early riser :). >>> lets_meet_at_ca = lets_meet_at.astimezone(ZoneInfo("America/Vancouver")) >>> lets_meet_at_ca datetime.datetime(2020, 10, 22, 7, 0, tzinfo=zoneinfo.ZoneInfo(key='America/Vancouver')) >>> print(lets_meet_at_ca) 2020-10-22 07:00:00-07:00 >>> >>> ## Similarly, the US friend finds the corresponding time >>> ## for the German friend(specified as the German local time). >>> ## The German friend works all day and is available after work :) >>> lets_meet_at_eu = lets_meet_at.astimezone(ZoneInfo("Europe/Berlin")) >>> lets_meet_at_eu datetime.datetime(2020, 10, 22, 16, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')) >>> print(lets_meet_at_eu) 2020-10-22 16:00:00+02:00 >>>
For the curious reader, the list of IANA time zones is found as follows…. Note that this creates a huge output.
>>> import zoneinfo >>> zoneinfo.available_timezones()
Feature 8: New Graphlib Module With Topological Sort Implementation
The new graphlib module now has an implementation of a topological sort of a graph. Consider the following example.
$ python Python 3.9.0b5 >>> >>> from graphlib import TopologicalSorter >>> >>> ## my_graph has two edges A->C->D and A->B. In other words >>> ## one could say A depends-on C, which depends-on D. As well >>> ## as A also depends-on B. >>> my_graph = {"A": {"B", "C"}, "C": {"D"}} >>> mg = TopologicalSorter(my_graph) >>> list(mg.static_order()) ['B', 'D', 'C', 'A'] >>>
Note that the suggested total order ['B', 'D', 'C', 'A']
is one interpretation of the order. It is not necessarily unique.
Finxter Academy
This blog was brought to you by Girish, a student of Finxter Academy. You can find his Upwork profile here.
Reference
All research for this blog article was done using Python Documents, the Google Search Engine and the shared knowledge-base of the Finxter Academy and the Stack Overflow Communities. Concepts and ideas were also researched from the following websites: