4 New Type Annotation Features in Python 3.11

A detailed tutorial — from installation to code examples

On Apr 25, 2022, Python published the latest alpha — 3.11.0a7, which represents the last release for the alpha phase for the 3.11 development. It’ll be moving to the beta phase early in May and the final official version of 3.11 is expected to be released in Oct 2022.

I know that it’s a big deal of work to adapt your applications to the latest Python, and most of the time, you don’t need to, because your applications are presumably working fine with an older version. However, if you’re the kind of person who likes to try new things, let’s get started with Python 3.11!

Note that you don’t want to modify your current applications because 3.11 is still in the alpha phase, but it doesn’t prevent us from trying it with a docker image — a separate virtual container that is independent of your applications.

Among the new features, I want to focus on type annotations in this article. Python is a dynamically typed language and the type annotations are used by many type checkers that can provide real-time code analysis. Many code bugs, such as type mismatches, can be identified while you’re still coding. Thus, the improvement of type annotations in Python 3.11 will help us write bug-free code.

Please note that we’re testing the alpha release, and these features are subject to change.

Prerequisites

Installing Python 3.11 with Docker

If you don’t have Docker installed on your computer, you can go to docker.com to find out the installation instruction. The idea of dockers is to create a reproducible container that streamlines the development of applications.

Once you install docker, you can run the following command on your command-line tool. In my case, I use the Terminal app on my Mac computer.

docker run -t -d python:3.11-rc-bullseye

This command pulls the image with the tag python:3.11-rc-bullseye. Just in case you may be wondering what this tag means. 3.11 represents the version of Python, rc means release candidate, and bullseye is the Debian release bullseye, this image is based on this Debian release.

If you’ve installed this image successfully, you should be able to see something as above, an image in the list of Containers/Apps. You can mouse on this image, and click the CLI icon, which launches the Terminal app.

You can simply type python, which enables the Python console in the Terminal, as shown below.

# python
Python 3.11.0a7 (main, Apr 20 2022, 17:55:51) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

From the prompt, we can see that it’s Python 3.11.0.a7 — the seventh alpha release for 3.11. We can also verify it by running the following code:

>>> import sys
>>> sys.version
'3.11.0a7 (main, Apr 20 2022, 17:55:51) [GCC 10.2.1 20210110]'

Using the Container in Visual Studio Code (VSC)

Using the command-line interface is fine to explore some basic features. However, it’s not the most pleasant in terms of writing code. Thus, we may want to use an IDE-like code editor, like Visual Studio Code (VSC). Although there are other alternative Python IDEs, for the purpose of this tutorial, we’ll stick to VSC.

Assuming you’ve installed the VSC on your computer, to get your VSC to work with the docker image, you need to install the Remote Containers extension. After the installation, you should be able to see a tab appearing on the sidebar, as shown in the figure below.

After clicking the tab, you can see that VSC populates the available containers — specifically the Python 3.11 container. Right-click the container, and you can attach VSC to the container. By attaching, a new window will pop up using the container.

Creating a Python file for evaluation

You can open a folder as our project. For simplicity, we’re just using the default root folder, and you can find the screenshot below.

In this folder, for evaluation purposes, we create a Python file called test11.py. You can run the file by calling python -m test11.py in your Terminal (you may have to use python3 -m test11.py if you may have python 3 installed).

4 New Features About Typing

1. Self — the Class Type

When we define methods in a class, we often need to return an instance of the same class. In many cases, the return value is the instance object that we call this method with. Consider the following example, to begin with.

class Box:
    def paint_color(self, color):
        self.color = color
        return self

The above code doesn’t use any type hints, and it may be unclear. So we can add more specific typing information. Some people may think of the following solution.

class Box:
    def paint_color(self, color: str) -> Box:
        self.color = color
        return self

However, you can’t get it to work, because, in the body of Box class, you can’t use Box, as it hasn’t been defined yet! Thus, an existing workaround is to use a type variant, which serves as a proxy for the to-be-defined type, as shown below:

from typing import TypeVarTBox = TypeVar("TBox", bound="Box")class Box:
    def paint_color(self, color: str) -> TBox:
        self.color = color
        return self

One inconvenience with this approach is that when you say want to change the name of the class, say we want to rename it to Container, we may have to make the following changes, the class name, the TBox definition, and the type for the return value in the paint_color method, as shown below:

from typing import TypeVarTContainer = TypeVar("TContainer", bound="Container")class Container:
    def paint_color(self, color: str) -> TContainer:
        self.color = color
        return self

These changes can be non-trivial if there are many usages like this in your project. With the introduction of the Self type in Python 3.11, things become more simplified.

from typing import Selfclass Box:
    def paint_color(self, color: str) -> Self:
        self.color = color
        return self

You should be familiar with the argument self in instance methods, which is the instance object. As a counterpart, Self represents the class where it’s used. In the example above, we use Self to indicate that the return value is an object in the type of “Self,” which is interpreted as the Box class.

Besides the benefits of eliminating the need to create type variants, there are also other benefits discussed in the official PEP 673, including usage in subclasses.

2. Arbitrary Literal String

The second feature also pertains to type annotation. Let’s review the status quo. When we define a function that accepts string literals, we can use the Literal type, as shown below:

from typing import Literaldef paint_color(color: Literal["red", "green", "blue", "yellow"]):
    pass

As you can see, when we want a function to take a string literal, you must specify the compatible string literals. However, because of the limited options, current code analysis thinks it’s an error (warning), as shown below:

To address this limitation, Python 3.11 introduces a new general type LiteralString, which allows the users to enter any string literals, like below:

from typing import LiteralString


def paint_color(color: LiteralString):
    pass


paint_color("cyan")
paint_color("blue")

The LiteralString type gives you the flexibility of using any string literals instead of specific string literals when you use the Literal type. For more specific use cases where LiteralString is applicable, such as constructing literal SQL query strings, you can refer to the official PEP 675.

3. Variadic Generics

We can use TypeVar to create generics with a single type, as we did previously for Box. When we do numerical computations, such as array-based operations in NumPy and TensorFlow, we use arrays that have varied dimensions and shapes.

When we provide type annotations to these varied shapes, it can be cumbersome to provide type information for each possible shape, which requires a separate definition of a class, as the exiting TypeVar can only handle a single type at a time.

from typing import Generic, TypeVarDim1 = TypeVar('Dim1')
Dim2 = TypeVar('Dim2')
Dim3 = TypeVar('Dim3')class Shape1(Generic[Dim1]):
    passclass Shape2(Generic[Dim1, Dim2]):
    passclass Shape3(Generic[Dim1, Dim2, Dim3]):
    pass

As shown above, for three dimensions, we’ll have to define three types and their respective classes, which isn’t clean and represents a high level of repetition that we should be cautious about. Python 3.11 is introducing the TypeVarTuple that allows you to create generics using multiple types. Using this feature, we can refactor our code in the previous snippet, and have something like the below:

from typing import Generic, TypeVarTupleDim = TypeVarTuple('Dim')class Shape(Generic[*Dim]):
    pass

Because it’s a tuple object, you can use a starred expression to unpack its contained objects, and in our case, it’s a variable number of types. The above Shape class can be of any shape, which has much-improved flexibility and eliminates the need of creating separate classes for different shapes.

For more detailed information about using TypeVarTuple, please refer to the official PEP 646.

4. TypedDict — Flexible Key Requirements

In Python, dictionaries are a powerful data type that saves data in the form of key-value pairs. The keys are arbitrary and you can use any applicable keys to store data. However, sometimes, you want to have a structured dictionary that has specific keys and the values of a specific type. In this case, you can use the TypedDict type.

from typing import TypedDictclass Name(TypedDict):
    first_name: str
    last_name: str

As shown above, we define Name which has first_name and last_name as the required keys and their values should be strings.

We know that some people may have middle names (say the corresponding key is middle_name), and some don’t. That is, Name class, as a TypedDict, should allow middle_class to be missing. There are no direct annotations to make a key optional, and the current workaround is creating a superclass that uses all the required keys, while the subclass includes the optional keys, as shown below:

from typing import TypedDictclass _Name(TypedDict):
    first_name: str
    last_name: strclass Name(_Name, total=False):
    middle_name: str

As shown above, the _Name class has first_name and last_name, while Name has middle_name. Notably, we need to specify total as False, meaning that the key middle_name can be omitted in the Name class.

Creating a superclass to address this business need is inconvenient, and we should have a better solution — Python 3.11 introduces NotRequired as a type qualifier to indicate that a key can be potentially missing for TypedDict. The usage is very straightforward, as shown below:

from typing import TypedDict, NotRequiredclass Name(TypedDict):
    first_name: str
    middle_name: NotRequired[str]
    last_name: str

As shown above, we no longer need a superclass/subclass structure. Instead, we just specify that middle_name is not required in this TypedDict — isn’t it much more concise? Please note that you may be seeing that NotRequired is not able to be imported, and I guess there might be an issue with the image.

If you have too many optional keys, you can specify those keys that are required using Required, instead of specifying those optional as not required. Thus, the alternative equivalent solution for the above issue is shown below:

from typing import TypedDict, Requiredclass Name(TypedDict, total=False):
    first_name: Required[str]
    middle_name: str
    last_name: Required[str]

Note in the code snippet we specify total as False, which makes all the keys optional. In the meantime, we mark these required keys as Required, which means that the other keys are potentially missing.

For more information about this feature, please refer to the official PEP 655.

Conclusions

In this article, we reviewed four new features in Python 3.11 about type annotations. These new features allow you to write more clean code by using more powerful type hints.

As a reminder, the final official release of Python 3.11 is Oct 2022. Are you ready for that?