Justin A. Ellis

A fun multi-lingual concurrency experiment

2023-03-04T00:00:00+00:00

The other day I was writing some code for a CI pipeline that needed to fetch release notes from several different git repos and collate them into one large set of release notes for all the different services. My first instinct was to write a little python helper function and use that in the CI pipeline. On the first iteration I just looped over the URLs for the different services and did a GET request, then I collected the results and did some formatting on them. Even though there were only a few GET requests I noticed this took a few seconds to run. Then I started thinking about concurrency and implemented something in Go for fun because I've been obsessed with Go recently. Then I implemented the same thing in Python and good ole Bash.

This post will mostly just go over these implementations, do a little explanation and mainly just point out that the overall code is nearly identical for these three languages. Then at the end we will see which one is the fastest.

In this example I am using a placeholder API to grab titles of TODO items. I add a 500ms delay to simulate a real endpoint that is a bit slow.

All the code here can be found on Github.

The overall structure is the same for all three.

Create function to fetch a single todo given its ID
Concurrently fetch all TODOs using this function
Process TODOs and output a single list of TODOs in markdown format

Go Version

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"os"
	"strconv"
	"sync"
	"time"
)

const (
	todoServer = "https://jsonplaceholder.typicode.com/todos"
)

func getTodo(index int) (string, error) {
	time.Sleep(500 * time.Millisecond)
	resp, err := http.Get(fmt.Sprintf("%s/%d", todoServer, index))
	if err != nil {
		return "", err
	}

	if resp.StatusCode != http.StatusOK {
		return "", fmt.Errorf("Error getting todo %d: %s", index, resp.Status)
	}
	defer resp.Body.Close()

	var todo struct {
		Title string `json:"title"`
	}

	if err := json.NewDecoder(resp.Body).Decode(&todo); err != nil {
		return "", err
	}

	return todo.Title, nil
}

func main() {
	numTodos, err := strconv.Atoi(os.Args[1])
	if err != nil {
		log.Fatalf("Error parsing number of todos: %v", err)
	}

	var wg sync.WaitGroup
	todos := make([]string, numTodos)

	for i := 0; i < numTodos; i++ {
		wg.Add(1)
		go func(j int) {
			defer wg.Done()
			todo, err := getTodo(j + 1)
			if err != nil {
				log.Printf("Error getting todo %d: %v", j+1, err)
			}
			todos[j] = todo
		}(i)
	}
	wg.Wait()

	fmt.Println("# Todos")
	for _, todo := range todos {
		if todo == "" {
			continue
		}
		fmt.Printf("* %s\n", todo)
	}
}

In the Go version I used waitgroups instead of channels since I want the output to be ordered and it is just easier with waitgroups. In the main function we loop over the TODO IDs and make an inline goroutine to get the todo and add it to the TODOs array. Notice that we make the array beforehand and are not appending to a slice, this would lead to race conditions. We then wait for all goroutines to finish by using wg.Wait(), lastly we loop over the TODOs that we have pulled down and make a markdown list with a title. Overall its very simple and concise. As usual, most of the lines are error checking.

Python Version

import asyncio
import logging
import sys
from typing import Optional

import httpx

TODO_SERVER = "https://jsonplaceholder.typicode.com/todos"

logger = logging.getLogger(__name__)


async def get_todos(client: httpx.AsyncClient, index: int) -> Optional[str]:
    try:
        await asyncio.sleep(0.5)
        response = await client.get(f"{TODO_SERVER}/{index}")
        response.raise_for_status()
    except httpx.HTTPStatusError as exc:
        logger.warning(f"Failed to get todo {index}: {exc}")
        return
    return response.json()["title"]


async def main(num_todos: int) -> None:
    async with httpx.AsyncClient() as client:
        tasks = [get_todos(client, index + 1) for index in range(num_todos)]
        results = await asyncio.gather(*tasks)

    print("# Todos")
    for todo in results:
        if todo is None:
            continue
        print(f"* {todo}")


if __name__ == "__main__":
    asyncio.run(main(int(sys.argv[1])))

In the Python version I used the asyncio library to make the concurrent requests using asynico.gather and used the httpx library to make the asynchronous HTTP requests. Like the Go version we have a separate get_todos function that takes in a client and an index and returns the title of the TODO. In the main function we create a client and then create a list of tasks that we will run concurrently. We then use asyncio.gather to run all of the tasks and wait for them to finish. They will be gathered in the order that the list comprehension was created with. Lastly we then loop over the results and print out the TODOs in markdown format. Overall this is slightly shorter than the Go version. There are many ways to do nearly the same thing in Python, but this is probably the most popular way nowadays since async/await was added to the language.

Bash Version

#!/bin/bash

TODO_SERVER="https://jsonplaceholder.typicode.com/todos"
NUM_TODOS=$1


get_todo(){
    sleep 0.5
    curl -s --fail $TODO_SERVER/$1 | jq -r '.title' > "/tmp/todo_$1.txt"
}

for ((i=1; i<=${NUM_TODOS}; i++));
do
    get_todo $i &
done

wait

echo "# Todos"
for ((i=1; i<=${NUM_TODOS}; i++));
do
    echo "* $(cat /tmp/todo_${i}.txt)" && rm "/tmp/todo_${i}.txt"
done

This one was the most fun to write. Even though I use bash everyday in CI/CD tasks, I usually don't write full scripts or try to do much concurrently. To mirror the structure of the other implementations I created a function to get a single TODO using curl and jq and then looped over the TODO IDs and called the function in the background. The big difference here is that since by running in the background we are using separate processes and it is hard to share data between them. To get around this I used a temporary file to store the TODO title with a filename that was the TODO ID.

I then used wait to wait for all the background processes to finish. Lastly I looped over the TODO IDs again and printed out, read in the content of the temporary files and output the TODOs in markdown format. This implementation is the shortest and most concise, but, as we will see in the next section, it is also the slowest.

Benchmarking

So I didn't do any fancy benchmarking, I ran each version with 200 TODOs and ran them 10 times and took the average. The results are below:

Language	Average Time (s)	Uncertainty (ms)
Go	0.71	26
Python	1.42	121
Bash	3.84	540

So as we can see the results are probably what you would expect. Go is fastest, Python is second and Bash is last; however, remember that all of these would have taken ~100 seconds if they were done sequentially. So even though Bash is the slowest, it is still 26x faster than the sequential version. Bash is slowest because it us spawning a new process for each TODO and doing extra IO with the temporary files. For Python vs Go, I'm sure there is a full reason why it is about 2x slower than the Go version but I think overall it is just because the Go version is compiled and the Python version is interpreted. Furthermore Go was specifically build with concurrency in mind and it is more of an add-on to Python.

Conclusion

So this was a strange one but I had fun playing around with it and hopefully you will dig into it a bit more. I'm planning on more posts using Go as I am really starting to like it and I think this post shows some of its power and simplicity. I'm also planning on doing some more posts on Python and Bash, so stay tuned. If you have any questions or comments feel free to reach out to me on Twitter or leave a comment below. Thanks for reading! (These last two sentences were completely generated with Copilot but it sounds right, so there it is.)

Production Dockerfile Tips and Tricks

2023-01-24T00:00:00+00:00

Container images are the backbone of a lot of modern workflows, especially in the age of Kubernetes and friends. Building images that are small, secure, and efficient is crucial for production level applications. In this post we will cover some quick tips and tricks for writing better Dockerfiles (you don't actually have to use Docker but the name Dockerfile is probably gonna stick around).

The TLDR version of this post is:

Use the smallest base image possible
Make builds deterministic
Run with a privileged user (never as root)
Separate dependencies from source code
Keep things that change more frequently closer to the bottom of the Dockerfile
The smaller the final image, the better

Instead starting with a bad Dockerfile and building up a better one, lets just start with a production level Dockerfile and use that as our example to explain some concepts.

FROM python:3.10.9-slim-bullseye

ENV USER=rta

WORKDIR /app

RUN useradd -mrU $USER && \
    chown -R $USER:$USER /app

COPY src/requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt && \
    rm -f requirements.txt

COPY --chown=$USER:$USER src/main.py /app

USER $USER
ENTRYPOINT ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Small base image and deterministic builds

FROM python:3.10.9-slim-bullseye

Ok, lets start at the top, notice we use a "slim" python image and we also specify the exact python version. Specifying the exact python version helps with making our build deterministic. And choosing the slim image helps keep the final image smaller. In general there are 3ish main flavors of python images:

python 3.10-alpine         8a3a8409a638   Less than a second ago   50.1MB
python 3.10-slim-bullseye  72fb1d206063   Less than a second ago   126MB
python 3.10                109421b4b8cd   Less than a second ago   921MB

Although the Alpine image is the smallest, it can be hard to work with sometimes. I would recommend always starting with the slim image and going from there. It is very unlikely that you will need a full debian or ubuntu OS in your Docker image.

So we have a small (mostly deterministic) base image. If we really want this truly deterministic then we could use the full image digest (docker images --digests | grep python) instead of the tag but this may be overkill and can make your Dockerfile less readable.

Privileged non-root user

ENV USER=rta

WORKDIR /app

RUN useradd -mrU $USER && \
    chown -R $USER:$USER /app

Next we set a USER environment variable that we will use throughout the Dockerfile. This next step is probably the one that is most often missed in example Dockerfiles that you find on the interwebs. After specifying the WORKDIR, which will create the directory, we create a privileged user and group with the useradd command. Using the -m flag we are creating a home director for this user. This is important since many python packages may put data files in the user's home directory by default so we need it to exist. In this same step (because we want to keep the number of layers to a minimum) we also give this user ownership over the WORKDIR. This is important so that we don't end up with any runtime errors if our code writes and files to this directory. Note we are just creating the user and group here, we are not setting the user yet. Its generally good to install packages (especially system level packages) using the default root user and to set the privileged user at the end of the Dockerfile.

Separating dependencies from source code

COPY src/requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt && \
    rm -f requirements.txt

In a lot of samples online you see some junk like COPY . .. Don't do this. First of all you will likely be copying a bunch of stuff you won't need, but even worse, you will be invalidating the cache any time you change any code. Since each layer is cached and since generally dependencies change way less often than the source code itself its best to separate dependencies from code as we do here. We just copy the requirements.txt file (remember those deterministic builds, use this requirements file as a real lock file using something like pip-tools, every dependency version should be fully specified). Keeping these layers separated like this will improve build times where you have a local cache and will also improve pull times on systems like Kubernetes.

In the next line we install our dependencies. Note that we are using the --no-cache-dir flag here, this will reduce the final docker image size since we are not creating a pip cache. We also clean up after ourselves by removing the requirements file since we don't need it to be in the final image.

Move things that change often close to bottom of Dockerfile

COPY --chown=$USER:$USER src/main.py /app

USER $USER
ENTRYPOINT ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Ok, the dependencies are built, now we need to actually bring in the source code. Note that we use the COPY --chown=user:group syntax. This is because by default the copy command will copy the file with root ownership but we want our user to own the source code.

Lastly, we set the user that (that only owns what is in this WORKDIR) and run our ENTRYPOINT script. There are some arguments as to whether you should use ENTRYPOINT or CMD for things like this but its a little harder to accidentally run the wrong thing with ENTRYPOINT than it is with CMD so I would recommend using ENTRYPOINT.

Multi-stage builds for smaller images

In some cases, especially if your code requires compilers or other system level libraries it is best to use a multi-stage build where you install any system dependencies along with your python dependencies in the first stage and copy just the needed python environment into the second stage. Lets assume that there a something in your code that requires a c-compiler when installing the python dependencies. Your Dockerfile may look something like

FROM python:3.10.9-slim-bullseye as build

RUN apt-get update && apt-get install -y --no-install recommends \
    build-essential gcc 

WORKDIR /app
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY src/requirements.txt .
RUN pip install -r requirements.txt

FROM python:3.10.9-slim-bullseye

WORKDIR /app

RUN useradd -mrU $USER && \
    chown -R $USER:$USER /app

COPY --from=build /opt/venv /opt/venv
COPY --chown=$USER:$USER src/main.py /app

USER $USER
ENV PATH="/opt/venv/bin:$PATH"
ENTRYPOINT ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

In the first stage we install our system level dependencies and our python dependencies. Note that we create a virtual environment here so we can copy over just our python dependencies into the second stage. The second stage creates our privileged user, copies over the virtual environment and sets the PATH variable to point there. As a rule of thumb, any time you need extra "stuff" to setup your app that you won't need at runtime you should use a two stage build and copy over the bare minimum of what will be needed.

Conclusion

In this short post we have seen some best practices for building production level dockerfiles. There are several great resources out there that go into a lot more detail than this post but hopefully this will get you on your way to writing great Dockerfiles. Aforementioned great resources

Some Advanced Typing Concepts in Python

2023-01-15T00:00:00+00:00

To start off, let me admit that yes, Python is a dynamically (gradually?) typed language. However, with modern Python (3.6+, and really with 3.10+) and static tooling (i.e. mypy, pyright/pylance) it is a pretty robust statically typed language. Most third party libraries either are typed or have types stubs distributed separately. From a developer perspective, this version of the language making use of full strict static typing is essentially a different language the Python of yesteryear.

I won't fully make the argument here but treating Python like a statically typed language greatly increases readability and expressiveness of the code, greatly reduces bugs and even the amount of tests you need to write, and makes it much easier to move on to a "real" statically typed language like Rust (do it, its great!) or Go (do it, its pretty good!) or even oldies like C/C++/Java.

While there is good material on more advanced typing concepts in Python (the mypy docs are actually quite good), most tutorials you find online are "hello world" level examples just introducing types to people who have never used Python or are from the dark days of Python pre-3.6. This post assumes that you have a decent grasp of Python and already know the basics of type annotations, or that you are coming from another statically typed language and just want to see if Python is up to snuff. Ok, rant over, lets get into some semi-advanced typing goodness.

All of the examples in this post are available here if you want to play around with them. I would suggest using vscode and the "strict" Pylance type checking mode.

Type annotations and the interface segregation principle

Remember the interface segregation principle from Uncle Bob's SOLID principles. Well if you don't, it basically means that users of your code should not be forced to rely on interfaces/types/classes that they don't need. We will see how Python type annotations can help with this.

Some general good practices

First lets look at how we can make our code more robust by defining loose types for input parameters and strict types for outputs. Lets say we are writing a function to filter out a certain word or phrase from some strings. You could write your function like this

from typing import List

def filter_things(things: List[str], filter_string: str) -> List[str]:
    return [t for t in things if filter_string not in t]

This is fine and would work like this

things = ["badger", "snake", "mushroom"]
filtered_things = filter_things(things, "mushroom")
# returns ["badger", "snake"]

But do we really need the inputs things to be a list, all we are doing is iterating over it. Lots of python types do that. We could guess and do something like

from typing import Any, Dict, List, Set, Tuple

def filter_things(things: List[str] | Tuple[str,...] | Set[str] | Dict[str, Any], filter_string: str) -> List[str]:
    return [t for t in things if filter_string not in t]

but that would be terrible and you have not just confused your user even more. Instead we can use the type that specifies the bare minimum constraint for what we are doing. Since we are just iterating over things we can do

from typing import Iterable, List

def filter_things(things: Iterable[str], filter_string: str) -> List[str]:
    return [t for t in things if filter_string not in t]

Using the Iterable type just states that things is a collection of strings that can be iterated over. Now our function can work with lots of different input types like tuples, sets, dicts, even generators since all of those types are iterable. Note that we do return a concrete type of List[str] here. It is always a good idea to hav the return type be as narrow as possible so the behavior is deterministic no matter what kind of input you give the function.

To hammer this home, lets look at one more example.

from typing import List

def get_fifth_element(things: List[str]) -> str:
    if len(things) < 5:
        raise ValueError(f"Length of input must be >= 5, found len={len(things)}")
    return things[4]

Again, we don't really need a list here but we aren't iterating. Instead we are checking the length and grabbing the value at a given index. So this means that we must be able to get the size of the input collection and we must be able to get the item in the collection at a given index. If we look at the docs we can see that the correct type to use here is Sequence since it implements the __len__ and __getitem__ methods which allow for us to call len on it and to index into it. So we have

from typing import Sequence 

def get_fifth_element(things: Sequence[str]) -> str:
    if len(things) < 5:
        raise ValueError(f"Length of input must be >= 5, found len={len(things)}")
    return things[4]

# good with a tuple
elements = ("earth", "wind", "water", "fire", "multipass")
fifth_element = get_fifth_element(elements)

# or a list
elements = ["earth", "wind", "water", "fire", "multipass"]
fifth_element = get_fifth_element(elements)

As we will see later, even this type is a bit too narrow still, but without defining our own type this will usually be fine.

Protocols

I had a previous post about Protocols but here we will go over them in the context of the interface segregation principle and just give a bit more detail as to why they are useful.

So lets start with the age old example of defining an Animal

import abc

class Animal(abc.ABC):
    @abc.abstractmethod
    def make_sound(self) -> None:
        pass

    @abc.abstractmethod
    def act(self) -> None:
        pass

    @property
    @abc.abstractmethod
    def num_legs(self) -> int:
        pass

Note, I used an ABC here. I could have used a protocol for this but I'm trying to fake a 3rd party library that may already have a lot of different implementation of a class using standard class hierarchy. Now lets implement some animals

class Dog(Animal):
    def make_sound(self) -> None:
        print("woof")

    def act(self) -> None:
        print("dog is walking")

    @property
    def num_legs(self) -> int:
        return 4


class Fish(Animal):
    def make_sound(self) -> None:
        print("blub")

    def act(self) -> None:
        print("fish is swimming")

    @property
    def num_legs(self) -> int:
        return 0

and lets make a function so we can hear some animal sounds

from typing import Iterable

def get_animal_sounds(animals: Iterable[Animal]) -> None:
    for animal in animals:
        animal.make_sound()

animals = [Dog(), Fish()]
get_animal_sounds(animals)
# woof
# blub

So this function takes in an Iterable (see we learned from the first part!) of Animals. If we run this function with these inputs then we will get woof and blub. Cool!, now we want to know what a fox says so we implement a fox

class Fox:
    def make_sound(self) -> None:
        print("redacted")

Sweet, lets plug this in

animals = [Dog(), Fish(), Fox()]
get_animal_sounds(animals)

Uh oh we get an error from our type checker like this

Argument of type "list[Dog | Fish | Fox]" cannot be assigned to parameter "animals" of type "Iterable[Animal]" in function "get_animal_sounds"
  "list[Dog | Fish | Fox]" is incompatible with "Iterable[Animal]"
    TypeVar "_T_co@Iterable" is covariant
      Type "Dog | Fish | Fox" cannot be assigned to type "Animal"
        "Fox" is incompatible with "Animal" PylancereportGeneralTypeIssues

Ah, yes because we just implemented make_sound for Fox and not the whole animal interface. Well thats dumb, I just want to know what the fox says, why do I need to implement the whole "interface". Protocols to the rescue

from typing import Iterable, Protocol

class CanMakeSound(Protocol):
    def make_sound(self) -> None:
        ...

def get_animal_sounds(animals: Iterable[CanMakeSound]) -> None:
    for animal in animals:
        animal.make_sound()

Now, we can find out what the fox says since all three animals now implement the CanMakeSound Protocol.

So of course this is a silly example but there are many real life situation where our functions, classes, etc only need to know about a few methods of the object and don't need to know about all of them. If we use Protocols to only specify what we need this means that a user can implement their own objects to just satisfy this protocol.

If you really are digging this, here is how we could make our get_fifth_element function even more flexible by using a custom protocol that only requires the bare minimum.

from typing import Any, Protocol, TypeVar

T = TypeVar("T", covariant=True)

class MySequence(Protocol[T]):
    def __getitem__(self, __idx: Any) -> T:
        ...

    def __len__(self) -> int:
        ...

def get_fifth_element(things: MySequence[str]) -> str:
    if len(things) < 5:
        raise ValueError(f"Length of input must be >= 5, found len={len(things)}")
    return things[4]

Generics with Protocol Constraints

In the examples above we used Protocols to make our functions very general (you could even say generic!); however, we only used them in the arguments and not in the return type. In general, it doesn't really make sense to return a Protocol. Sometimes you need to use generics with custom constraints. We can illustrate this by implementing the simple max function that returns the maximum of two inputs. First lets just do it for ints

def max(a: int, b: int) -> int:
    if a < b:
        return b
    return a

Ok, simple but the python builtin max function works on lots of other types, like float, str, and even lists and sequences. If we want a generic max function, look at the above function and figure out what the minimum requirement is. If we look, the only requirement is that a and b can be compared using the "less than" operator. In Python, that means that they both must implement the __lt__ magic method. So lets make a bounded generic and re-write our function

from typing import Protocol, TypeVar
from typing_extensions import Self

class HasLessThan(Protocol):
    def __lt__(self, __other: Self) -> bool:
        ...

T = TypeVar("T", bound=HasLessThan)

def max(a: T, b: T) -> T:
    if a < b:
        return b
    return a

Now if we use this function we will get the right return types and it will allow for many different comparisons

m = max(3, 4) # m is of type: int
m = max("hello", "world") # m is of type: string
m = max([4, 5, 6], [1, 2]) # m is of type: List[int]

As always, we could even define our own type. Lets define a line class where we want max to return the longer line object

from dataclasses import dataclass
from typing_extensions import Self

@dataclass
class Line:
    x_min: float
    x_max: float

    def __lt__(self, other: Self) -> bool:
        """Compare based on line length"""
        return (self.x_max - self.x_min) < (other.x_max - other.x_min)

m = max(Line(0, 3), Line(0, 5)) # returns Line(0, 5)

This is a very powerful method for defining generic functions/classes that only specify the bare minimum type constraints. If you are coming from Rust this is similar to generics with trait bounds.

Parameter Specification Variables

As Python is a gradually typed language, it cannot or could not always have static typing in all situations. Until Parameter Specification Variables or just "ParamSpec" we could not get good static typing on functions that are decorated. Once a function was decorated we lost the type information about the input and output parameters of that function. Lets see how we can fix that now with ParamSpec

import time
from typing import Callable, ParamSpec, TypeVar

T = TypeVar("T")
P = ParamSpec("P")

def timer(func: Callable[P, T]) -> Callable[P, T]:
    def inner(*args: P.args, **kwargs: P.kwargs) -> T:
        tstart = time.perf_counter()
        out = func(*args, **kwargs)
        print(f"function: {func.__name__} ran in {(time.perf_counter()-tstart)} seconds")
        return out

    return inner

@timer
def add(a: float, b: float) -> float:
    """
    Add two numbers
    """
    return a + b

out = add(3.4, 4.5) 
# function: add ran in 6.902000677655451e-06 seconds

Here we have created a timer decorator that simply prints the time it took to call the decorated function. When we add the decorator to the simple add function it will print out the runtime. This is a simple example but lets look at what is going on. Notice that the outer function of the decorator takes in the decorated function func and we use a type annotation of Callable[P, T]. T is just a generic variable stating that the function could return any type. The P variable is the ParamSpec. This is what allows to keep that type information in the decorated function. In the inner function we use *args and **kwargs as you always would for unknown inputs but now we can annotate them with P.args and P.kwargs, respectively. And this is the magic that respects the type information of the decorated function. Your code editor (I use vscode and it is amazing for this and really all things...) should show you the function definition when you hover over it and the type checker should respect the input and output types. I used this feature quite heavily in a previous post about implementing a fully typed lru-cache.

Overloads

Overloads let you specify different combinations of inputs and/or outputs for a single function. This can lead to some really bloated functions that do way too much but, like most things, everything in moderation.

Broadly speaking there are two cases where you would want to consider using overloads

Optional Second order decorators (i.e. decorators that may or may not take arguments)
Functions where there are several different combinations of input parameters which may or may not lead to different output types. Note, overloads should only be used here where the job can't be done with generics.

First lets look at the second-order decorator case by modifying our example above. What if we want the ability to format our message that prints the runtime, or what if we want to use a logger instead of just printing, but we still want to have the default behavior without passing in any arguments. We can modify it as follows

import time
from typing import Callable, Optional, ParamSpec, TypeVar, overload

T = TypeVar("T")
P = ParamSpec("P")

def _default_display_fn(function_name: str, call_time: float) -> None:
    print(f"function: {function_name} ran in {call_time} seconds")

@overload
def timer(__func: Callable[P, T]) -> Callable[P, T]:
    ...

@overload
def timer(*, display_fn: Callable[[str, float], None]) -> Callable[[Callable[P, T]], Callable[P, T]]:
    ...

def timer(
    __func: Optional[Callable[P, T]] = None,
    *,
    display_fn: Optional[Callable[[str, float], None]] = None,
) -> Callable[[Callable[P, T]], Callable[P, T]] | Callable[P, T]:

    display_fn = display_fn or _default_display_fn

    def decorator(func: Callable[P, T]) -> Callable[P, T]:
        def inner(*args: P.args, **kwargs: P.kwargs) -> T:
            tstart = time.perf_counter()
            out = func(*args, **kwargs)
            display_fn(func.__name__, time.perf_counter() - tstart)
            return out

        return inner

    if __func is not None:
        return decorator(__func)

    return decorator

There is a lot of detail here but the main point is the two overload decorators. These specify the two signatures of the timer decorator. The first is a simple decorator just like we had before where you can use it with no parentheses. The second case is our second-order decorator which we can pass in a function to display the run time. We could use it as follows

import logging

def logging_display_fn(function_name: str, call_time: float) -> None:
    logging.info(f"function: {function_name} ran in {call_time} seconds")

@timer(display_fn=logging_display_fn)
def add(a: float, b: float) -> float:
    """
    Add two numbers
    """
    return a + b

Now instead of using the default print we will use a logger. In both cases the type signature of the original add function are preserved. Just by looking at the complexity of the actual timer implementation we can start to see why overload can be dangerous since the function can get very complicated and it may be better to split into separate functions. Nonetheless using overload as a way to have both first- and second-order decorators is generally a decent idea.

To demonstrate the second case for overloads lets extend the functionality of our max function so that a user can pass in types that don't natively have support for <, but we allow them to pass in a key function to determine how to compare values of a and b:

from typing import Callable, Protocol, TypeVar, overload

from typing_extensions import Self

class HasLessThan(Protocol):
    def __lt__(self, __other: Self) -> bool:
        ...

T = TypeVar("T", bound=HasLessThan)
S = TypeVar("S")

def _default_key(__val):
    return __val

@overload
def max(a: T, b: T) -> T:
    ...

@overload
def max(a: S, b: S, *, key: Callable[[S], T]) -> S:
    ...

def max(a, b, *, key=None):
    key_func = key or _default_key
    if key_func(a) < key_func(b):
        return b
    return a

In the example above we have two overloads. The first overload is exactly what we had before where the inputs must support the < operator. The second overload now allows a user to pass in any type but they are required to pass in a key function that returns the value that will be used to compare the two inputs.

For example, lets return to our custom line type but this time, we won't implement the __lt__ method but instead use a key function in max

from dataclasses import dataclass

@dataclass
class Line:
    x_min: float
    x_max: float

m = max(Line(0, 3), Line(0, 5)) # this will fail the type check
m = max(Line(0, 3), Line(0, 5), key=lambda line: (line.x_max - line.x_min)) # returns Line(0, 5)

So here our overloads let us be more flexible in our types. Note, that in this case I did not include types in the actual implementation. It is up to you whether or not you do this since the actual implementation signature will not be exposed to the end user, only the overloads.

The max function in python actually has several more overloads if you want to check it out.

Wrapping Up

In this post we have covered some more advanced use cases for type annotations. The TLDR version is

Use general input types only specifying the bare minimum of what is needed
Use strict concrete output types
Use custom Protocols define the minimum interface needed to functions/methods/classes
Use custom Protocols as bounds on generic types to make code more general
Use Parameter Specification (i.e. ParamSpec) when making decorators
Use overloads sparingly when generics cannot do the job. Prefer overloads over usage of Union types, especially in return types

As I mentioned in the introduction, none of this is required for valid Python but making use of the type annotation system will make your code more readable and will make it much nicer to work with.

Early and late binding closures in Python

2022-11-23T00:00:00+00:00

So it happened again, late-binding closures bit me. I once again had to discover that this was a thing so I decided to make a little post about it just in case someone happens to run into this and by some miracle lands here.

The goal is to keep this short but I want to cover 3 main things

What are closures?
How can you shoot yourself in the foot with them?
How not so shoot aforementioned foot.

What are closures (the short version)

In many cases closures are taken to be synonymous with anonymous (or lambda, no not the AWS kind) functions. However, one can have an anonymous function that is not a closure and a closure that is not an anonymous function. In simple terms a closure is a function that captures at least one variable from the environment (i.e. outside of its lexical scope). Make sense? Lets look at some examples.

from typing import Callable

# closure with a nested function
def f(x: int) -> Callable[[int], int]:
    def g(y: int) -> int:
        return x + y

    return g

# closure with a lambda function
def h(x: int) -> Callable[[int], int]:
    return lambda y: x + y

CONSTANT: Final[int] = 4

# also a closure
l = lambda x: x + CONSTANT

# same...
def k(x: int) -> int:
    return x + CONSTANT

Both functions g and h return a closure and l and k are closures themselves. They are closures because they use variables outside of their lexical scope, g and h from the outer functions f and h scope, respectively, and l and k use the CONSTANT from the module scope.

So if we call some of these functions it would look like

# these return functions that take in an integer and return an integer
a = f(2) 
b = h(2)

print(a(4)) # returns 6 (2+4)
print(b(2)) # returns 4 (2+2)

So we can first generate our closure by calling g and h, respectively and then pass our second variable when calling that returned function. This may seem strange to newer programmers or maybe those from pure object oriented languages but using functions as first-class citizens in your code can be quite powerful.

The late-binding vs. early-binding problem

Ok, now that we have a decent understanding of what closures are, lets look at how we can shoot ourselves in the foot. Say we are all excited, because closures are awesome, and we want to make a list of functions to perform some action, say add one to a given number.

And since we love being "pythonic" we decide to be clever:

new_funcs = [lambda: ii+1 for ii in range(5)]
output = [nf() for nf in new_funcs]

We have created our closure and then created 5 functions, the first will do 0+1, the second 1+1 and if we evaluate these functions in order we will get [5,5,5,5,5]. Wait that doesn't seem right, what happened? Late-binding closures is what happened. This means that the value used in the closure, ii in this case, is looked up when the function is called and now when it was defined. Since the function is called after the loop if finished we will end up always using the last value in the loop.

How to avoid foot shooting

We can avoid the problem above in a few ways, one is to use something like our functions f or h above to bind the argument to the closure early by passing it as the argument of the outer function.

from typing import Callable

def add_one(x: int) -> Callable[[], int]:
    def g() -> int:
        return x + 1

    return g

# could use lambda here as well
def add_one(x: int) -> Callable[[], int]:
    return lambda: x + 1

new_funcs = [add_one(ii) for ii in range(5)]
outputs = [nf() for nf in new_funcs]

The output will now be my luggage combination [1,2,3,4,5]. Another option, which is much less verbose (although almost identical under the hood) is to use functools.partial

from functools import partial

def add_one(x: int) -> int:
    return x + 1

new_funcs = [partial(add_one, ii) for ii in range(5)]
outputs = [nf() for nf in new_funcs]

This will return the same output. Both of these approaches lead to early binding of the arguments and can be quite useful for making generic callback functions, among other things.

Well, thats it for this time and watch out for those early-binding closures.

Access Modifiers in Python

2022-01-15T00:00:00+00:00

If you have stumbled across this post then you are probably coming to Python from another object oriented language that has real support for access modifiers. Python has almost no support for access modifiers at runtime but it does now have pretty good support for access modifiers at "compile time" through linters and type checkers.

In this short post we will cover the main ways that one can mimic access modifiers in Python. We will cover private and protected attributes and methods, readonly attributes, and final methods and variables.

For this post, the code is checked with the great Pylance based language server in vscode. So other IDEs or static checkers/linters may not give the same warnings/errors. I also make use of the phrase "compile time", which just means that at the time that linters or type checkers are run.

Private vs Protected

I have been programming in Python for almost 15 years now and I only recently discovered that the "underscore" (i.e. naming a variable or method _name instead of name) convention actually represents a protected value and not a private value. Furthermore, I discovered that Python actually does have runtime support for private variables through a double underscore (i.e. __name). It will actually raise an error!

As a refresher, private means that the attribute/method can only be used inside the class where it is defined. Protected means that the attribute/method can only be used in the class where it is defined or its subclasses.

Ok, so how do we use this? What does it actually do? And why would we do it? Lets look at a convoluted, but illuminating example:

class Thing:
    def __init__(self, public: str, *, protected: str = "protected", private: str = "private"):
        self.public = public
        self._protected = protected
        self.__private = private

    def info(self) -> None:
        print(
            (
                f"This class has public attribute: {self.public}, "
                f"protected attribute: {self._protected}, "
                f"and private attribute: {self.__private}"
            )
        )

So here we have a simple Thing class with three attributes. One is public, one is protected, and one is private. We can see that we denote "protected" with a single underscore before the variable name and "private" with a double underscore before the variable name.

The first thing to note is that if you define a private attribute you should use it in the class since that is the only place it can be used. If you do not you will probably get a warning such as "__private is not accessed". Ok, lets use this thing!

>>> thing = Thing("public")

# this is fine because it is assessing the variables internally in the info method
>>> thing.info()
'This class has public attribute: public, protected attribute: protected, and private attribute: private'

# this is also fine because the public attribute is indeed public
>>> print(thing.public)
'public'

# this will run but will give an error when checked with pylance
>>> print(thing._protected)
'"_protected" is protected and used outside of the class in which it is declared'

# this will not actually run and will raise an AttributeError but it will also give an error when checked
>>> print(thing.__private)
'"__private" is private and used outside of the class in which it is declared'

So we see that we can use both the __private and _protected attributes inside the info method and we can use the public attribute anywhere. Furthermore, we cannot use the _protected or __private attributes outside of the class. So far in this example, there is really no difference between protected and private since we only have the one class. Lets make a subclass

class SomeThing(Thing):
    def more_info(self) -> None:
        print(f"This class has public attribute: {self.public}, protected attribute: {self._protected}")

>>> some_thing = SomeThing("public")

# still can use the info method which uses the private attribute internally
>>> some_thing.info()
'This class has public attribute: public, protected attribute: protected, and private attribute: private'

# can use the new more_info method that uses the public and protected attribute
>>> some_thing.more_info()
'This class has public attribute: public, protected attribute: protected'

We have made a subclass of Thing and we have a simple method that can indeed use the _protected attribute. However, if we try to add a new method that uses the private attribute then bad things will happen

class SomeThing(Thing):
    def more_info(self) -> None:
        print(f"This class has public attribute: {self.public}, protected attribute: {self._protected}")

    def use_private(self) -> None:
        print(f"Private attribute is {self.__private}")

>>> some_thing = SomeThing("public")

# this will raise an AttributeError and will also give an error when checked
>>> some_thing.use_private()
'"__private" is private and used outside of the class in which it is declared'

So we cannot use the __private attribute in the subclass, as expected.

Ok, we have shown a basic example of how to use protected and private attributes. The same rules work for protected and private methods. But when should you use them? There aren't any hard and fast rules for this but there are a few rules of thumb and things to be aware of

protected and private variables are part of a concept known as information hiding which deals with hiding implementation details from downstream users.
private attributes/methods should be used in cases where you don't want downstream users or developers to have access to that attribute or method. This is good for hiding implementation details which may be prone to change but will not affect downstream users.
protected attributes/methods should be used where developers can have access (through subclassing) but not outside users. This is useful for defining methods to be implemented by subclasses (through ABCs) which are then used in the parent class through a code re-use mechanism. Check out my previous post for more info on this use case.
If you use protected attributes/methods and allow for subclassing then these attributes/methods essentially become part of the public API since other developers can have access to them in their subclasses. This means that a change to the protected implementation in the parent class will affect all subclasses. We will see later how we can protect against unwanted subclassing with the @final decorator.

Readonly attributes

In the example above the public attribute was read/write. What if we want a readonly attribute? There are two ways to do this, one is enforced at runtime and the other is enforced through type checking at "compile time". Lets modify our example above to include a readonly attribute

class Thing:
    def __init__(self, readonly: str):
        self.__readonly = readonly

    @property
    def readonly(self) -> str:
        return self.__readonly


>>> thing = Thing("readonly")
>>> print(thing.readonly)
'readonly'

# this will raise an AttributeError but will also raise an error when checking
>>> thing.readonly = "Hello!"
'Cannot assign member "readonly" for type "Thing" Property "readonly" has no defined setter'

We have made use a @property which allows us to make readonly a readonly attribute. This will raise both a runtime error and a "compile time" error when trying to set this attribute. However, it does require the use of a private attribute (could also use protected, making sure to pay attention the caveats mentioned above) and the use of a property. There is a shorter way by using the Final type

from typing import Final


class Thing:
    def __init__(self, readonly: str = "readonly"):
        self.readonly: Final = readonly

>>> thing = Thing("readonly")
>>> print(thing.readonly)
'readonly'

# this will not raise a runtime error but will raise an error when checking
>>> thing.readonly = "Hello!"
'Cannot assign member "readonly" for type "Thing" "readonly" is declared as Final and cannot be reassigned'

In this case we only have to annotate the readonly attribute with the Final type to get the same behavior at "compile time", but we will not get the runtime error (i.e. we can set this variable). So depending on how you use type checking this method could more or less useful to you.

Lastly, the Final type does not only apply to attributes but can be used anywhere. It is very useful for defining module level constants that should not be modified.

Final classes and methods

In the last section, we saw how to use the Final type annotations to make variables readonly, at least in a static type checking sense. In this last section we will see how to make classes and methods more restricted in terms of what can be subclasses or overridden. Lets go back to our Thing class

from typing import final


@final
class Thing:
    def __init__(self, public: str, *, protected: str = "protected", private: str = "private"):
        self.public = public
        self._protected = protected
        self.__private = private

    def info(self) -> None:
        print(
            (
                f"This class has public attribute: {self.public}, "
                f"protected attribute: {self._protected}, "
                f"private attribute: {self.__private}, "
            )
        )

This is identical to our definition above but we have added the @final decorator. If we not try to subclass this class

class SomeThing(Thing):
    pass

then we will get an error (through static type checking, not at runtime) "Base class "Thing" is marked final and cannot be subclassed". We should also note that if we do mark a class a final then there is no need to distinguish between protected and private variables since both technically have the same meaning now since the class cannot be subclassed.

You can also use the @final decorator to control which methods are allowed to be overridden:

class Thing:
    def __init__(self, public: str, *, protected: str = "protected", private: str = "private"):
        self.public = public
        self._protected = protected
        self.__private = private

    @final
    def info(self) -> None:
        print(
            (
                f"This class has public attribute: {self.public}, "
                f"protected attribute: {self._protected}, "
                f"private attribute: {self.__private}, "
            )
        )

When we sublcass and try to override the info method

class SomeThing(Thing):
    def info(self) -> None:
        print("Overriding info method")

We would get the following error: "Method "info" cannot override final method defined in class "Thing"".

Both of these methods can be useful when you really want to control what downstream users/developers can do with your classes and methods.

Conclusion

In this post we have seen how to make attributes/methods both private and protected and the caveats associated with that. We have also seen how to use a @property decorator and Final annotation to make an attribute or variable readonly. Finally (get it?!) we have seen how to use the @final decorator to mark a class or a method as final, thus barring it from being subclassed or overridden.

Lastly, I'll offer some advice. In the world of open source software we need think differently about what is actually private, public, and protected. We don't want to restrict access too much so that it makes developer's lives harder when trying to extend your code, but we also don't want to create a large API surface with a lot of unneeded public methods and attributes that have to be kept backwards compatible. So now that you know how to use these features in Python, think carefully about how you use them and remember "With great power comes great responsibility".

Abstract Base Classes and Protocols: What Are They? When To Use Them?? Lets Find Out!

2022-01-11T00:00:00+00:00

In Python there are two similar, yet different, concepts for defining something akin to an interface, or a contract describing what methods and attributes a class will contain. These are Abstract Base Classes (ABCs) and Protocols.

Until the advent of type annotations, ABCs were the way to go if you wanted to have any kind of validation on class/instance methods or properties and isinstance checks. With type annotations, ABCs became more relevant as a way to define an "interface" for a given class and then use that as a type annotation.

However, to use ABCs as an interface we must rely on nominal subtyping and a strict class hierarchy (we will explain this later but, in short, we will have to subclass the ABCs in order to use it as an interface). With Protocols we can use structural subtyping or "Duck typing" (i.e. the class only has to have the same methods and attributes, no subclassing necessary).

So when do we use ABCs and when do we use Protocols? Before we dig into this, lets first get a basic understanding of how each works.

What are Abstract Base Classes

Here I will give a brief overview of ABCs, if you want a much more detailed explanation see this great video by one of the creators of ABCs.

In general there are two use cases for ABCs, as a pure ABC that defines an "interface" and as a tool for code re-use via the Framework Design Pattern or through Mixins.

Pure ABCs (ABC as Interface)

The simplest way to use an ABC is as a pure ABC, for example:

from abc import ABC, abstractmethod


class Animal(ABC):
    @abstractmethod
    def walk(self) -> None:
        pass

    @abstractmethod
    def speak(self) -> None:
        pass

Here we have defined an ABC Animal with two methods: walk and speak. Note that the way to do this is to subclass ABC and to decorate the methods that must be implemented (i.e. part of the "interface") with the @abstractmethod decorator.

Now we can implement this "interface" to create a Dog

class Dog(Animal):
    def walk(self) -> None:
        print("This is a dog walking")

    def speak(self) -> None:
        print("Woof!")

This will work fine but if we happened to forget to implement the speak method then we would get this error on creation

>>> dog = Dog()
TypeError: Can't instantiate abstract class Dog with abstract method speak

We can see that we get an error because we haven't implemented the abstract method speak. This ensures that all subclasses implement the correct "interface".

ABCs as a tool for code reuse

Another, and probably more common, use case for ABCs is for code reuse. Below is a slightly more realistic example of a base class for a statistical or Machine Learning regression model

from abc import ABC, abstractmethod
from typing import List, TypeVar

import numpy as np

T = TypeVar("T", bound="Model")


class Model(ABC):
    def __init__(self):
        self._is_fitted = False

    def fit(self: T, data: np.ndarray, target: np.ndarray) -> T:
        fitted_model = self._fit(data, target)
        self._is_fitted = True
        return fitted_model

    def predict(self, data: np.ndarray) -> List[float]:
        if not self._is_fitted:
            raise ValueError(f"{self.__class__.__name__} must be fit before calling predict")
        return self._predict(data)

    @property
    def is_fitted(self) -> bool:
        return self._is_fitted

    @abstractmethod
    def _fit(self: T, data: np.ndarray, target: np.ndarray) -> T:
        pass

    @abstractmethod
    def _predict(self, data: np.ndarray) -> List[float]:
        pass

Ok, lets unpack this first. There are two public methods, fit and predict. The fit method calls the private abstract method _fit and then sets the private attribute _is_fitted. The predict method checks if we have fit the model before trying to make predictions and then calls the private abstract method _predict. Lastly the base class defines a property is_fitted. We do this so that a user cannot set this value explicitly and it will only get set when calling fit.

A quick aside. Other than the ABC there are a few other important things going on. The first is the use of generic self in the definition of the fit method. Here we type self with a generic variable T that is bound to the model class. We do this to ensure that subclasses of Model will return the correct type when calling fit. Secondly we use a private attributed _is_fitted internally and expose this with the is_fitted property. This is not strictly necessary but it is good practice to keep this variable readonly since we really only want this to be True if we have actually successfully run fit.

Ok, back to ABCs. In the example above we have used this base class to implement some logic that will be inherited by all of its children, namely the fit and the predict methods. These methods delegate the actual work to the private abstract methods, _fit and _predict, that the children must implement. Of course we could make this a pure ABC and have all children implement fit and predict but it would be quite tedious for all children to have to re-implement the is_fitted validation in both of these methods. Furthermore, this is a simple example and in real situation there could be much more complicated shared code in the base class. Finally, we could have made the abstract methods public but in this case and probably most cases like the above example we should keep them private instead of polluting the class. A end consumer only needs to know that the class has a fit and a predict method and a is_fitted (i.e. readonly) property.

For a good, real life example of this kind of pattern take a look at Pytorch's Module. While this class does not actually use ABCs it uses the same pattern where there is a lot of reusable code in the base Module class and users only have to implement the forward method.

Ok, now that we have some understanding of how to use ABCs for code reuse. Lets implement a super simple model

class MeanRegressor(Model):
    def __init__(self):
        super().__init__()
        self._mean = None

    def _fit(self, data: np.ndarray, target: np.ndarray) -> "MeanRegressor":
        self._mean = target.mean()
        return self

    def _predict(self, data: np.ndarray) -> List[float]:
        return list(np.ones(data.shape[0]) * self._mean)

In this example we have implemented a very simple regression model to return the mean of the target for every sample. In this case the _fit method only sets a private state variable which is the mean of the target. The _predict method returns a list that is the length of the number of samples with the computed mean as the value. Lets see how this works.

>>> data = np.array([1.0, 2.0, 3.0, 4.0])
>>> target = np.array([2.0, 3.0, 5.0, 10.0])
>>> mean_regressor = MeanRegressor()

# try to predict without fitting first
>>>  mean_regressor.predict(data)
ValueError: MeanRegressor must be fit before calling predict

# Check if fitted
>>> mean_regressor.is_fitted
False

# Fit and predict
>>> preds = mean_regressor.fit(data, target).predict(data)
[5.0, 5.0, 5.0, 5.0]

# Check if fitted now
>>> mean_regressor.is_fitted
True

As we can see in the example above we are actually calling the methods and property from the ABC which under the hood is calling our concrete implementations of the private _fit and _predict methods. So we get the error checking and automatic setting of the _is_fitted attributes for free. This way, users do not need to worry about that when creating a new type of Model. Isn't that fun?

What are Protocols

Protocols were introduced in PEP-544 as a way to formally incorporate structural subtyping (or "duck" typing) into the python type annotation system.

There are two main, but related, use cases for Protocols. First, they can be used as an interface for classes and functions which can be used downstream in other classes or functions. Secondly, they can be used to set bounds on generic types.

Protocols as Interfaces

Protocols allow you to define an interface for a class or function that will be type checked on usage rather than on creation. For example, we can make our Animal ABC above a Protocol

from typing import Protocol

class Animal(Protocol):
    def walk(self) -> None:
        ...

    def speak(self) -> None:
        ...

Note that this looks pretty similar to our ABC based Animal class above. We inherit from typing.Protocol instead of abc.ABC and we don't need to add the @abstractmethod decorators since Protocols are not meant to be "implemented" but simply act as an interface in downstream tasks. Lastly, it is common practice to use ... in the body of methods in a Protocol instead of pass as we did in the ABC, although either will work in both places.

Ok, lets now implement a Dog

class Dog:
    def walk(self) -> None:
        print("This is a dog walking")

    def speak(self) -> None:
        print("Woof!")

Note here that we don't subclass animal, we simply have to implement the methods specified in the Animal Protocol. We can then use this in a downstream task like:

def make_animal_speak(animal: Animal) -> None:
    animal.speak()

>>> dog = Dog()
>>> make_animal_speak(dog)
'Woof!'

Here static type checkers would be happy because the dog instance does indeed implement the Animal Protocol because it has the same structure but is not itself a child class of Animal.

Lets see how Protocols enforce the interface. Lets say we forget to implement the speak method on our Dog class

>>> dog = Dog()
>>> make_animal_speak(dog)
Argument of type "Dog" cannot be assigned to parameter "animal" of type "Animal" in function "make_animal_speak"
  "Dog" is incompatible with protocol "Animal"
    "speak" is not present

In this case a static type checker (Pylance in the example above) would raise an error and tell the user that Dog does obey the Animal Protocol since it doesn't implement the speak method. Notice that this is different than an ABC that will raise an error when creating the Dog class, whereas Protocols will raise an error where they are used.

Ok, lets get a bit more tricky. What if we define the Dog class but change the signature of speak a bit

class Dog:
    def walk(self) -> None:
        print("This is a dog walking")

    def speak(self, name: str) -> None:
        print(f"Woof! My name is {name}")

When we try to use this in the function

>>> dog = Dog()
>>> make_animal_speak(dog)
Argument of type "Dog" cannot be assigned to parameter "animal" of type "Animal" in function "make_animal_speak"
  "Dog" is incompatible with protocol "Animal"
    "speak" is an incompatible type
      Type "(name: str) -> None" cannot be assigned to type "() -> None"
        Keyword parameter "name" is missing in destination

So, once again we get an error but this time it is because the speak method on Dog does not have the same signature. Pretty cool huh?

In the example function above, we don't actually even need the walk method since it won't be used in the function. We can narrow down the input type by defining a new Protocol

class SupportsSpeak(Protocol):
    def speak(self) -> None:
        ...

def make_animal_speak(animal: SupportsSpeak) -> None:
    animal.speak()

>>> dog = Dog()
>>> make_animal_speak(dog)
'Woof!'

So we still have the same Dog class with a walk and speak method but we define the new Protocol SupportsSpeak (the naming is somewhat standard if you are defining an interface with one method) that just defines the speak method. And everything still works and would indeed work with any class that has the same speak method signature. This is a simple example but this can be quite powerful in more complicated code.

One last thing to mention about using Protocols as interfaces is that it is possible to make them useable at runtime via an isinstance check with the runtime_checkable decorator.

from typing import Protocol, runtime_checkable

@runtime_checkable
class Animal(Protocol):
    def walk(self) -> None:
        ...

    def speak(self) -> None:
        ...

>>> dog = Dog()
>>> isinstance(dog, Animal)
True

Protocols as Generic Type Bounds

When defining a generic type variable in python we can give it a bound which means that the generic type must either be a child class of the bound if given a class bound or it must implement the protocol if given a Protocol. If you were paying attention we actually used a type bound in the Model ABC example above. In this case we typed the self variable with the generic type T that was bound by Model itself. This was done so that child classes of Model would return the correct type for fit which indeed returns self. If that is not clear take a look at the documentation. So that was an example of using a class bound.

However, in generic programming we usually want to make things as specific as possible. The max function is a great example of this. The builtin max function can take in several different types of input. We could use overload for every different kind of input but that can be tedious. Instead we notice that a specific implementation of max only requires that the input types define the __lt__ magic method (meaning we can do x < y). We can type (a somewhat simplified) max method as follows (for a much more detailed video, check out this)

from typing import TypeVar, Protocol


class SupportsLessThan(Protocol):
    def __lt__(self, __other: Any) -> bool:
        ...

S = TypeVar("S", bound=SupportsLessThan)

def my_max(x: S, y: S) -> S:
    if x < y:
        return y
    return x

Lets break this down. First, we implement a Protocol that defines the __lt__ method. We then create a generic type variable S that is bound by our SupportsLessThan Protocol. This means that S can be any type as long as it implements __lt__. We then define the max function which takes in two arguments x and y, which are both generic type S and returns the same type.

A lot of builtin types have a __lt__ method implemented so we can use this function with integers or strings, for example

>>> max_int = my_max(4, 5)
>>> max_str = my_max("hello", "world!")

In the example above the max_int will return an int and max_str will return a string. If we pass in an object that does not have __lt__ implemented then we will get an error.

So ABC or Protocol?

Yes. You should use both as they are good at different things and both have should their place in your toolbox. We have already seen above the main use cases for ABCs and Protocols and how they work. Given those examples here are some good overall suggestions and observations.

Abstract Base Classes
- Belong to their subclasses. An ABC is not usable by itself, it can only be used by implementing a child class. So because of this, ABCs inherently belong to their subclasses as part of a strict class hierarchy.
- ABCs are a good mechanism for code reuse, especially for boilerplate code or logic that will not change for any (or most) subclasses. The best strategy here is to have the ABC (i.e. parent class) do most of the work and have the children implement the specifics.
- Good for real time validation when creating an instance of a child class. As we saw above, ABCs will raise an error on creation if the child does not implement all abstract methods.
Protocols
- Belong where they are used. As we saw above, Protocols are not "implemented" but tell downstream code (i.e. other functions or classes) what the structure of the input object is expected to be. Also, we saw that we can define multiple protocols for the same kind of object depending on what is needed. This means that Protocols belong where they are used.
- Good for defining interfaces, especially for 3rd-party libraries when we don't want to tightly couple our code to a specific 3rd party library.
- Good (really the only way) for specifying flexible generic type bounds.
- This somewhat goes without saying but Protocols only are useful if using type annotations and cannot be used in any other way (except for runtime_checkable).

Ok, so we know that ABCs and Protocols are, we know what they are good at. So when should we use them? The answer to that is somewhat subjective and depends on your environment but here are some rules of thumb

Use ABCs if you want to reuse code. Inheritance is not always the best method of code reuse but it can be quite useful.
Use ABCs if you require strict class hierarchy (as in you need to use method resolution order or you need to check __subclasses__) in your application.
Use ABCs if you will need several implementations of a class with several methods.
Use Protocols for strict type annotations (i.e.only annotate the methods/attributes you need)
Use Protocols for generic bounds
Use Protocols for abstract interfaces for 3rd party libraries

Well, thats it for this time. Now go forth into our bold almost statically typed python future with confidence!

A Modern and Explicit approach to Python Exceptions

2021-12-13T00:00:00+00:00

Two of the most popular "modern" programming languages are Rust and Go. While these languages are quite different, both have a few things in common. First, they are loved by their users, are growing fast, and both forego exceptions as a way of handling errors, instead they treat errors like any other type/object.

Since Python is dynamically types or nowadays, "gradually typed", handling errors with exceptions makes sense. But in the modern Python era we can make use of type annotations to make error handling a bit more explicit. After all, "Explicit is better than implicit".

In this post we will be exploring some alternative ways of handling errors in Python based on the ways in which Rust and Go handle errors. But first, lets review how errors are handled in the three languages.

{: .notice--info} As I was writing this and working on the code I came across a much more fleshed out version of a python Rust result type so if you actually want to use this in your code I would recommend this package as it is much more fleshed out.

Traditional Error Handling in Python

Traditionally error handling in Python is handled through Exceptions. While there are many ways to handle these exceptions with differing levels of sophistication the basic structure looks like this:

def divide(x: float, y: float) -> float:
    if y == 0:
        raise ZeroDivisionError("Cannot divide by 0")
    return x / y

So here we check that the denominator is 0 (because we can't divide by 0) and if so, we will raise a special exception ZeroDivisionError with a custom message, otherwise we get on with our usual business of dividing x by y. When we call this function we would handle the error like:

try:
    value = divide(x, y)
except ZeroDivisionError:
    # handle here

Or if choose not to recover then we can just call the function and the Exception will cause the program to crash. This is all fine and is indeed the standard way of handling errors but it is a bit tricky to know that this function may raise this error. Even if is provided in the documentation (which it is best practice to document any exceptions that could be raised) there is nothing for any linters or static type checkers to pick up on that will let a user know that the divide function could raise an error. This is where Rust and Go do something different as we will now see.

Error Handling in Rust

In Rust errors are handles by using the Result type which is actually a rust enum containing two variants, an Ok representing success and an Err representing an error. We specify in the return type the values stored inOk and Err, respectively. Our divide function would look something like this in Rust:

fn divide(x: f64, y: f64) -> Result<f64, &'static str> {
    if y == 0.0 {
        return Err("Cannot divide by 0");
    }
    Ok(x / y)
}

If we call this function we can handle this in many different ways but a few common ways would be

// handle via pattern matching
match divide(x, y) {
    Ok(v) => println!("The result is {}", v),
    Err(e) => println!("Error in divide: {}", e),
}

// handle via a method on Result that will return the value is successful or
// panic if there was an error
let value = divide(x, y).unwrap();

There are many many other ways to handle this and don't worry about the syntax if you are unfamiliar with Rust, the main point is that value is not itself the value x/y but an enum of type Result with two variants, Ok(x/y) or Err("Cannot divide by 0"). This forces the caller to handle any potential errors explicitly. The second method by calling unwrap() is most similar to just calling the function in python and letting it raise an error if it indeed throws that error; however, calling unwrap is generally bad practice in Rust. Nevertheless, a caller is still aware that the function can fail just by the fact that the return type is a Result.

Error Handling in Go

In Go, errors are handled very explicitly. Instead of returning a single type like in Rust, Go returns both the value and an error. If there is no error then it will be set to nil. While this is more verbose than Rust it does follow the same principle of explicitly leaving error handling up to the caller by putting the error in the return of the function. Our division function would be written something like this:

import (
	"errors"
)

func Divide(x, y float64) (float64, error) {
	if y == 0.0 {
		return 0.0, errors.New("Cannot divide by 0")
	}
	return x / y, nil
}

Just like in the above cases we check for the error case of y=0 and then return a placeholder value and the error. If there is no error then we return the real value and nil. Then when calling this function we would do something like

import (
    "fmt"
)

value, err := Divide(x, y)

// it is always best practice to do this check
// the value should not be trusted if this check is not done
if err != nil {
    fmt.Println("An error has occured:", err)
    return
}
fmt.Println("Dividend is:", value)

Just like with Rust, the user will explicitly know that this function can fail, just by the fact that it returns a value an a possible error. This check of if err != nil is ubiquitous in Go code and is the main way to check for errors. One difference with Rust is that a user could still go ahead and use the value even if there is an error but that wouldn't be very smart and is not recommended.

A Go-like error handling pattern in Python

We have seen the basics of how errors are handled in Rust and Go. Lets see how we can make our Python error handling more Go-like.

The simplest way to do this is to re-write our function like:

from typing import Optional, Tuple


def divide(x: float, y: float) -> Tuple[float, Optional[ZeroDivisionError]]:
    if y == 0:
        return 0, ZeroDivisionError("Cannot divide by 0")
    return x / y, None

Here we return two values just like in our Go code, the actual value or a placeholder along with a ZeroDivisionError or None. In this case we still make use of the builtin exceptions because hey, we are using Python! For the typing, we have to specify the error as Optional since we can also return None. When using this function we can follow this pattern

value, err = divide(x, y)

if err is not None:
    raise err

print(f"Dividend is: {value}")

Looks almost just like the Go code! Just as in our example above we can still use the value without the check but at least we know that this can return an error by the return signature and types. In this case since we are still using Python exceptions we can raise the error if we don't have any way of avoiding it. This is similar to the try-except statement from above but now we explicitly use the error as a value and not just something that may happen when calling the function.

Just for fun, we can make this a bit cleaner using python generics!

T = TypeVar("T")
E = TypeVar("E", bound=Exception)

Result = Tuple[T, Optional[E]]


def divide(x: float, y: float) -> Result[float, ZeroDivisionError]:
    if y == 0:
        return 0, ZeroDivisionError("Cannot divide by 0")
    return x / y, None

In Rust-like fashion we now return a Result type that is just a type alias for a tuple of any type and an option error that is a subclass of Exception. While this does the same thing it is less verbose and could represent an idiomatic way of doing things.

If we don't want to specify the exact exception type we could make another type alias:

from typing import Optional, Tuple, TypeVar

T = TypeVar("T")

ResultWithErr = Tuple[T, Optional[Exception]]


def divide(x: float, y: float) -> ResultWithErr[float]:
    if y == 0:
        return 0, ZeroDivisionError("Cannot divide by 0")
    return x / y, None

This just tells us that we are returning a result and an error, where the error is an Exception of some kind.

These type aliases could be nice to cut down on boilerplate but possibly make things a bit less explicit. In any case, Python offers a lot of options in terms of typing and actually returning errors.

Doing error handling the Go way is pretty straightforward in Python and is definitely more explicit than the try-except methods.

A Rust-like error handling pattern in Python

While implementing errors Go-style was pretty straightforward, to get a Rust-like system will be a bit more work. So, in Rust-land an enum can hold values and have methods defined on it. In rust, the Result enum looks like:

enum Result<T, E> {
   Ok(T),
   Err(E),
}

where T and E are generic types. One could then implement methods on this enum. In Python things don't quite work that way, so to mimic the Result type we will instead define two generic classes Ok[T,E] and Err[T, E] and use a type alias of Union to make a Result[T, E]. In the end we will have a Rust-like error handling system with a small subset of the Rust functionality. Lets start with the Ok class

from typing import Any, Callable, Generic, TypeVar

T = TypeVar("T")
E = TypeVar("E", bound=BaseException)


class Ok(Generic[T, E]):
    _value: T
    __match_args__ = ("_value",)

    def __init__(self, value: T):
        self._value = value

    def __eq__(self, other: Any) -> bool:
        if isinstance(other, Ok):
            return self._value == other._value  # type: ignore
        return False

    def unwrap(self) -> T:
        return self._value

    def unwrap_or(self, default: T) -> T:
        return self.unwrap()

    def unwrap_or_else(self, op: Callable[[E], T]) -> T:
        return self.unwrap()

    def __repr__(self) -> str:
        return f"Ok({repr(self._value)})"

Ok (get it!) lets unpack this a bit. First this class is generic (since it inherits from Generic) over a general type variable T and a bound type variable E. In this case we place a bound on E so that it must be a subclass of BaseException. This is different than Rust where anything can be used as an error but Python already has exceptions so we may as well use them. Next we see that this class is initialized with a value of type T which can be any type. We define the magic method __eq__ to return True if the contained values match in another instance of Ok. This will be important later when performing pattern matching or any other equality tests. We have also defined a nice repr on this class so that it is easily inspected.

Now, we have some methods on this class which are meant to mirror the functionality of the Rust Result enum. Before we get into the methods, lets define the Err class since it will have the same methods.

class Err(Generic[T, E]):
    _err: E
    __match_args__ = ("_err",)

    def __init__(self, err: E):
        self._err = err

    def __eq__(self, other: Any) -> bool:
        if isinstance(other, Err):
            return self._err == other._err  # type: ignore
        return False

    def unwrap(self) -> NoReturn:
        raise self._err

    def unwrap_or(self, default: T) -> T:
        return default

    def unwrap_or_else(self, op: Callable[[E], T]) -> T:
        return op(self._err)

    def __repr__(self) -> str:
        return f"Err({repr(self._err)})"

Just as in the Ok case this class is generic over T and E. It is also initialized with an err of type E (must be a subclass of BaseException) And it defined __eq__ in an analogous way to Ok.

Now for the methods. Well, first lets define our result type:

Result = Union[Ok[T, E], Err[T, E]]

This is just a type-alias of Union. So if we do Result[float, ZeroDivisionError] this means that a function/method will return either Ok(float) or Err(ZeroDivisionError). Since we can return either we must have exactly the same methods on both the Ok and Err classes. This is as close as we can get to defining methods on a single enum as we can do in Rust. Without further ado lets look at the methods and how they would be used in our canonical example.

def divide(x: float, y: float) -> Result[float, ZeroDivisionError]:
    if y == 0:
        return Err(ZeroDivisionError("Cannot divide by 0"))
    return Ok(x / y)

where we return Err if y=0 and wrap our output in Ok if not just like our Rust example above. When we use this function we can handle the errors in multiple ways

# handle via pattern matching (very Rusty!, but only in python >= 3.10)
match divide(x, y):
    case Ok(v): print(f"The value is {v}")
    case Err(e): print(f"The error is {e}")

# unwrap the result and let an error be raised if the function failed
value = divide(x, y).unwrap()

The first method here uses Python's new structural pattern matching which has a syntax very similar to Rust. Note, the __match_args__ class attributes in Ok and Err lets us match on the values without providing the keyword. The second method here calls the unwrap method, which, as we can see from the code above, will just return the value if Ok and will raise the error if the function errored. This is nearly equivalent to just raising the error in the function itself (the standard python way) but now at least we know the function can error by the Result return type and the fact that we must call unwrap. However, there are more things we could do.

We can provide a default value to fall back on if the function errors

# this will return the Ok value if no error, otherwise it will return the default (0.0)
value = divide(x, y).unwrap_or(0.0)

We can also provide a callable to try to recover from the error

x, y = 1.0, 0.0
EPS = 1e-9
value = divide(x, y).unwrap_or_else(lambda e: x / (y + EPS))

In this case we supply a function that takes in the error (in this case we don't actually use the error value but we could) and returns the division again but with a small offset to the zero denominator. This may or not be a good idea in this case but it illustrates the point. This method unwrap_or_else will return the Ok value if there is no error, otherwise it will call the supplied function.

It is not shown directly here but all of this is fully typed and will work with type checkers in and intellisense!

So this was a bit more work but we have partially replicated some of the Rust functionality and we have made error handling explicit!

Summary

Python handles errors by raising exceptions which can then be caught in downstream callers. However, since Python is becoming gradually typed it makes sense to think of alternate ways to handle errors that are made explicit through the type system. In this post we have looked at how two modern languages, Rust and Go, handle errors and showed how we can do similar things in Python.

In summary:

Go handles errors by returning the value alongside the error. We can do this in Python by returning a tuple or the value and a possible error. We then check for this error explicitly in our downstream code and we know about it from the return type.
Rust handles errors by returning an enum with two variants Ok or Err. The caller then must pull out the value or error through various different methods. We can replicate this in python by returning a single value which is a type alias of Union. A downstream caller then will "unwrap" this in ways similar to Rust.

Lastly, I will mention that neither of these approaches is likely to become mainstream simply because of the massive code bases already written and the now idiomatic way of handling exceptions; however it may still be useful to use these methods on stand-alone projects or to just think about alternatives!

Building a fully typed LRU Cache in Python

2021-11-25T00:00:00+00:00

In this post we are going to build a fully typed LRU (least recently used) cache (almost) from scratch using Python. We will then create a function decorator that mirrors the builtin functools implementation.

This exercise will cover several advanced python concepts including generic types and other advanced typing concepts, decorators (including second order decorators), and magic methods.

{: .notice--info} This exercise is for learning and demonstration purposes and thus is not optimized for performance. All source code can be found here

What we will end up with

What we will have at the end of this exercise is a clone (at least in terms of the public interface) of the functools lru_cache decorator. We should get these results:

@lru_cache(maxsize=100)
def fib(n: int) -> int:
    if n < 2:
        return n
    return fib(n - 1) + fib(n - 2)

>>> [fib(n) for n in range(16)]
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610]

>>> fib.cache_info()
CacheInfo(hits=28, misses=16, maxsize=100, currsize=16)

In addition to this we also want to retain all type information from the decorated function. In older versions of functools.lru_cache this information is lost so static type checkers and intellisense could not function correctly. In our version we will retain this type information so that static type checkers will work correctly.

Building the cache class

First, we will build a standalone LruCache class to handle that actual heavy work. In most implementations of LRU cache, a hash map (i.e. dictionary) and a doubly linked list are used.

In this case, since the main point of this article is how to use some of the more advanced python features we will use one single built in data type, the OrderedDict

The full implementation is below:

from collections import OrderedDict
from typing import Generic, Hashable, Optional, TypeVar

T = TypeVar("T")


class LruCache(Generic[T]):
    def __init__(self, capacity: int):
        self.capacity = capacity
        self.__cache: OrderedDict[Hashable, T] = OrderedDict()

    def get(self, key: Hashable) -> Optional[T]:
        if key not in self.__cache:
            return None
        self.__cache.move_to_end(key)
        return self.__cache[key]

    def insert(self, key: Hashable, value: T) -> None:
        if len(self.__cache) == self.capacity:
            self.__cache.popitem(last=False)
        self.__cache[key] = value
        self.__cache.move_to_end(key)

    def __len__(self) -> int:
        return len(self.__cache)

    def clear(self) -> None:
        self.__cache.clear()

Ok, there is a lot to unpack here. First lets look at the basic functionality.

# build a new cache with capacity 2 (i.e. we can store at most two values at a time)
>>> cache = LruCache[str](capacity=2)

# put two values in to the cache
# cache keys will now be [1, 2]
>>> cache.insert(1, "hello")
>>> cache.insert(2, "hello again")

# length is 2 since that is the number of values in the cache
>>> print(len(cache))
2

# can retreive 1 since both 1 and 2 are in the cache
# cache keys are now [2, 1] since 2 was called more recently than 1
>>> print(cache.get(2))
hello again

# this will evict key 1 with value "hello"
# cache keys are now [3, 2]
>>> cache.insert(3, "goodbye")

# we now return None since 1 is no longer in the cache
>>> print(cache.get(1))
None

# but we can get the cached value for 3
# cache keys are still [3, 2] since 3 was called more recently than 2
>>> print(cache.get(3))
goodbye

# cache keys are now [2, 3]
>>> print(cache.get(2))
hello again

# this will drop key 3
# cache keys are now [1, 2]
>>> cache.insert(1, "I'm back!")

# get None since 3 was dropped
>>> print(cache.get(3))
None

# finally we can clear the cache
>>> cache.clear()
>>> print(len(cache))
0

We can see that the capacity determines how large the cache is and every call to cache.get or cache.insert puts that key at the top of the queue essentially keeping track of the most recently used keys and when we insert a value when the cache is full (i.e. the length is equal to the capacity) the least recently used value is dropped. This is what we want.

A few important things to notice about the code, first is that LruCache itself is generic over a type T which represents the value type of the cache. In the example above we specify LruCache[str] to indicate the type of value in the insert method and the return type in the get method is a string. We can pass in any type here and our IDE and static type checkers are able to infer and check the input and return types. Pretty awesome huh?

The next thing to notice is the Hashable type. Since we want this class to generic we need to give it the least amount of information possible about what kinds of keys will be uses. At the very least dictionary keys need to be hashable (i.e. implements the __hash__ method), and that is exactly what Hashable checks for. If we try to pass in a key that is not hashable like a list then we will get an error from a static type checker like pyright

Argument of type "list[int]" cannot be assigned to parameter "key" of type "Hashable" in function "get"
    "list[int]" is incompatible with protocol "Hashable"
      "__hash__" is an incompatible type

Another thing to notice here is the use of the magic method __len__. This is a fairly common one but by implementing this method it allows us to call len on an instance of LruCache.

One of ther bit of fanciness here is the use of the double leading underscore on __cache. We don't want to allow outside users to be able to modify the cache, and while there are no truly private variable in python by using the double underscore you will actually get an AttributeError if you try to access it outside of the class.

As for implementation, as mentioned above, we make use of the OrderedDict provided by the collections module. For the most part, OrderedDict is almost obsolete since regular python dictionaries now retain insertion order; however, it is very useful in this case because of the move_to_end method which lets us move any key-value pair to the end of the dictionary which is how we keep track of the lest/most recently used keys in our lru cache. Finally, the popitem method allows for last=False which drops the key-value pair in a FIFO (first-in-first-out) manner as oppsed to the default LIFO order (last-in-first-out) order. This lets us easliy pop off the least recently used key.

Building the function wrapper class

Now that we have a generic LruCache implementation we move on to building a class that will wrap our function that we wish to cache. What we want is something that has the exact same call signature as our original function but also has a cache_info and cache_clear method as well as a __wrapped__ attribute containing a refrerence to the wrapped function itself. This all mirrors the functools.lru_cache API.

from collections.abc import Callable
from typing import Final, Generic, NamedTuple, ParamSpec, TypeVar

T = TypeVar("T")
P = ParamSpec("P")


class CacheInfo(NamedTuple):
    hits: int
    misses: int
    maxsize: int
    currsize: int


class _LruCacheFunctionWrapper(Generic[P, T]):
    def __init__(self, func: Callable[P, T], maxsize: int):
        self.__wrapped__ = func
        self.__cache = LruCache[T](capacity=maxsize)
        self.__hits = 0
        self.__misses = 0
        self.__maxsize: Final = maxsize

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> T:

        call_args = args + tuple(kwargs.items())

        ret = self.__cache.get(call_args)

        if ret is None:
            self.__misses += 1
            ret = self.__wrapped__(*args, **kwargs)
            self.__cache.insert(call_args, ret)
        else:
            self.__hits += 1

        return ret

    def cache_info(self) -> CacheInfo:
        return CacheInfo(
            hits=self.__hits,
            misses=self.__misses,
            currsize=len(self.__cache),
            maxsize=self.__maxsize,
        )

    def cache_clear(self) -> None:
        self.__cache.clear()
        self.__hits = 0
        self.__misses = 0

Ok, lets walk through this. First, we define the CacheInfo named tuple to match the signature of the functools.lru_cache implementation. NamedTuples are good for things like this where you want an immutable but easy to read/use return type of some function or method. If you need to add methods or want to modify fields its probably better to use a dataclass.

The next thing to notice is that _LruCacheFunctionWrapper (leading underscore is used here to indicate that this is a "private" class) is generic of types P and T which as we see from the __init__ corresponds to the arguments and return type of the wrapped function, respectively. For the P generic we make use of the new (in python 3.10) ParamSpec which will make things much nicer as we will see. From python 3.8 up you can import this from typing_extensions but it is moved into the builtin typing module in python 3.10.

We also use private attributes (leading underscore) again here because we really don't want these attributes to be used outside of this class. Note that __wrapped__ is still accessible since it is a "dunder" attribute. We use the LruCache specifying the type using the generic variable T as our cache.

In this class we make use of another quite popular magic method __call__. The call magic method allows us to call an instance of this class like a function. Furthermore, we have used the power of generics and the new ParamSpec to actually indicate what the argument names and types are via the P.args and P.kwargs methods as well as the return type T. If we were to use this to wrap our function above then static type checkers and intellisense will know the correct types and will actually raise errors if the wrong type is passed in, just like it would on the original function.

>>> wrapped = _LruCacheFunctionWrapper(fib, maxsize=4)
>>> x = wrapped(3) # type signature (int) -> int

This call function itself is pretty simple since most of the work is handled in LruCache. We turn the arguments and keyword arguments into a tuple (i.e. something hashable) and use that as the key in our LRU cache. This does assume that the input arguments and keyword arguments in our original function are hashable. Unfortunately, as far as I know, there is no way to bound a ParamSpec so that it would check if the arguments are hashable. Maybe one day... Nonetheless, we create the call_args tuple and try to get the function return value from the LRU cache. If we get a non None return value then we increment the hits (hit the cache) counter and return the function value. If we don't have the value in the cache we call our wrapped function and insert the value into the cache and increment the misses counter (missed the cache).

The cache_info method simply returns a CacheInfo named tuple with the current state of the function cache. Lastly the clear_cache method just clears the LRU cache and resets the counters.

One last thing to note here before moving onto the final step is that this way of hashing the function arguments if far from optimal in that it assumes that positional arguments are not called as keyword arguments and that keyword arguments are called in the same order. It also does not take into account default keyword arguments. These things are all fixable through the use of something like a frozen set and using the inspect module. Maybe in a follow on post...

Building the decorator

Ok, we are almost there. We have our generic LruCache and a function wrapper that uses that cache. Now we just need our function decorator. In this case, since our decorator takes an input argument maxsize what we are really constructing is a decorator factory or second order decorator.

from collections.abc import Callable
from typing import ParamSpec, TypeVar

T = TypeVar("T")
P = ParamSpec("P")


def lru_cache(maxsize: int) -> Callable[[Callable[P, T]], _LruCacheFunctionWrapper[P, T]]:
    def decorator(func: Callable[P, T]) -> _LruCacheFunctionWrapper[P, T]:
        return _LruCacheFunctionWrapper(func, maxsize)
    return decorator

Without the type annotations this if just a function that returns another function which wraps our original function. However, through the use of the generic types, as we have done above we can retain and pass forward all of the function argument and return types.

The outer decorator lru_cache returns a callable that takes in a generic callable with arguments P and return type T and returns a generic instance of _LruCacheFunctionWrapper that we have defined above. The inner decorator takes in the actual function to be cached and returns an instance of _LruCacheFunctionWrapper.

This may look somewhat messy but the generic types are quite magical and really will help make your code more safe and "correct". But thats it, we now have created a decorator that will cache function calls with a maxsize of maxsize! Feel free to go back to the beginning and use this on our fibonacci sequence function and play around in an IDE to see the full power of types.

Summary

And there we have it, we have implemented a fully typed version of lru_cache through the heavy use of generics and OrderedDict. We have also made things pythonic by using magic methods like __init__ and __call__ and followed some best practices by separating out the caching implementation from the actual function wrapper.

A Practical Guide to MCMC Part 2: Further Resources

2021-01-16T00:00:00+00:00

If you are coming here Part 1 then I'm sorry to tell you that this is not the fully fleshed out post you were hoping for. If you are coming here directly, good for you going straight to the "Part 2", but alas you may be disappointed.

Part 1 was written when I was finishing up my postdoc career and was still fresh in my MCMC techniques. Since then I got a new job in Data Science and haven't had a lot of time to get back to blogging and have not done any MCMC since my postdoc days. However, I have gotten a fair number of requests for more information about advanced MCMC techniques and this post is my weak attempt to disseminate some further resources for you, though they may be a bit more academic than the last post.

MCMC Resources

Slides on some more advanced MCMC techniques: This talk and the code in the next point are part of a summer-school and also has some exercises if you want to try them out.
Simple Differential Evolution and Parallel Tempering Implementation: This contains some basic classes (in outdated python 2) to do differential evolution and parallel tempering. Can be useful to see how these work in practice instead of just through equations.
Brief yet technical intro to adaptive metropolis, custom jump proposals, and parallel tempering: This is from an academic paper which you are probably not interested in, but the appendix starting on page 17 has some useful information and more references if you are really interested.

Well thats all folks, hopefully this is some help. I will mention that there are some great packages out there like emcee and pymc3 that may do a lot of this for you although I haven't tried them out.

How to transition from Academia to Data Science

2021-01-03T00:00:00+00:00

Oh boy, another one of these blog posts about transitioning to Data Science from Academia. Well, in this post I'll try to add a slightly different take on the usual advice along with some more traditional advice. This is not a step-by-step guide because I don't think such a thing exists in this case. Instead, this post will be my opinionated views on what you can do to make a smooth transition from Academia to Data Science. Also, this post is not about "how to get a Data Science job". I won't talk about your resume, or prepping for interviews or any of that stuff. This post is about how to become a Data Scientist.

First of all, I should note that while a lot of the things I cover in this post could be generic and apply in many different situations, I will lay down a few assumptions first.

You are in a technical field already (Mathematics, Statistics, Physics, Chemistry, Biology, etc).
You have some programming knowledge (any language is fine).
Your intended Data Science career is applied and not research oriented.
All this will probably be more applicable if you are in a PhD program but could still be relevant for a B.S.

And with that out of the way, let's get into it.

What is Data Science anyway?

If you have stumbled across the blog, you probably have some idea of what Data Science is or maybe you think you know what it is. Data science could mean that you manage data bases and write SQL code all day. It could mean that you work with "big data" and do things with Hadoop (is that still a thing, probably...), Spark and other big data tools.¹ God forbid, it could mean that you do most of your work in Excel making linear regression models.

But in my experience when most people say Data Science, what they really mean is "Machine Learning Engineer"², i.e. someone who develops and applies ML models to real data and deploys them for use in real-world scenarios. Of course, there is more to it than that, but that is the gist and this the kind of Data Scientist that I will have in mind for the rest of this post.

How has Academia already prepared me for this role?

I know when I was a graduate student I thought that I was only prepared to do one thing with my life -- be a (astro)physics professor/researcher (after an extended post-doc career). In my time (2009-2014) in physics grad school, there was almost never any mention of any other career trajectory. It was assumed that this was the path that everyone would take, which basically meant there was one mode of operation: Write Papers.³

However, there are still a lot of things that you will learn in your academic career (both undergraduate and graduate) that will be very useful for your Data Science (ahem... Machine Learning Engineer) career.

Learning the struggle

There are some skills that can not be taught. They can only be learned through struggle. If you have done any kind of original research in your academic field then you know the struggle. You have to learn to figure things out for yourself because there is no "right answer" that you can look up in the back of the book. If you have ever written any significant code base then you also know the struggle. You have to figure out how to put everything together and debug the inevitable bugs that will appear.

  "https://jellis18.github.io/images/struggle.png"

alt="Struggle">

<figcaption>Learning the struggle. Sometimes completing

a ML project seems like a Sisyphean task.

In Data Science, most of your existence is struggle and you need to be able to feel comfortable not knowing what to do in a given moment. You need to be able to experiment and possibly fail. In many cases a stakeholder (fancy language for the person/entity that wants some ML product/functionality) does not know much about ML, all they know is what they want an end product to look like. Actually, in some cases they don't even know what they want the end product to look like. It is more common that they have a lot of data (perfectly clean of course) and they want “insights” from that data. You, as the Machine Learning Engineer need to be prepared to struggle through finding a solution to their problem.

Even though the struggles you will go through in a ML based problem may be different than those that you have had in academia, being comfortable with being uncertain will prepare you well for a career as a machine learning engineer.

In the next section, I will offer some advice to get more familiar with the ML-based struggle.

Learning how to learn

Very related to "the struggle" is learning how to learn. Everyone learns in different ways and throughout your academic career it is a good bet that you have discovered your own method. In many cases, definitely in mine, you realize that you don't really learn much from your classes. Sure you may do well on tests, but you forget it all in a week. You learn by doing. All the theory is approximately worthless without the practice. Would you trust a doctor that has spent years reading up on the latest surgical techniques but never actually performed a surgical procedure? Would you recruit a "guitarist" who knows all the musical theory but has never actually played a guitar?

Later I will offer some advice in this area, but knowing how to learn on your own is one of the most important skills in Data Science, or life for that matter.

"Soft Skills"

Overall, soft skills are a combination of people skills and communication skills. This one strongly depends on whether or now you have had to present your work to both technical and non-technical audiences. It also depends if you have been part of a large collaboration.

With that being said, being able to present technical information to non-technical audiences in a clear and concise way is a skill that will help you land a good Data Science job. As I mentioned above, I am focusing on an applied Data Science job, not a research data scientist position. This means that you will need to be able to communicate with the stakeholders as well as your teammates.

If you have been lucky enough to be part of a large collaboration with many different groups that need to work together, then you will have gained invaluable skills that will really help you in Data Science. Being able to translate between different groups is a very useful skill. For example, when I was a grad student I was part of a data analysis group and we used mostly Bayesian techniques. There was another group that used mostly frequentist techniques. On the surface these things may be very different, but there are some useful translations between the two and I was able to speak both languages, which made things move much more smoothly.

Again, in the next section I will give some advice on how to practice your communication skills.

Practical (and very opinionated) advice

As the title says, this advice is very opinionated, but these methods have worked well for me. Before I get into the actual advice let me explain what I look for in a Data Scientist candidate:

Someone who is easy going and has good personal/communication skills. This might not seem too important for a technical role, but to me it is critical. Most Data Scientists work in teams, so being able to get along and communicate is of paramount importance.
Someone who has shown initiative in their past and who is fairly self motivated. This does not need to be ML/Data Science related, it could apply to your academic field of study but it could also apply to your preparation for transitioning to data science.
Someone who has experience with "real" data. By real data, I mean messy data. I want to see that you have learned the struggle and have accepted it. Again, this can apply to your academic past or your data science preparation.
Someone who can converse about many different ML techniques. This does not mean that you need to know any in very deep detail, but you should have an idea of what techniques exist and the applications that they are used for.

With that, lets get to the advice

1. Learn Python

Python is the language of Machine Learning. It is very easy to use and it mostly wraps libraries that are written in more performant languages like C.

  "https://jellis18.github.io/images/python-ml.jpg"

alt="Python">

<figcaption>This is slightly outdated but Python

is growing very fast and has overtake other languages.

It is growing across all areas, not just ML but for this post I will focus on ML. Python is very powerful in ML because of several purpose-build ML libraries such as pandas, tensorflow, pytorch, scikit-learn, spacy and transformers just to name a few.

If you already know Python, that is great. If you don't, there are a plethora of resources out there to learn. I would not recommend using a book to learn Python when there are so many great resources out there already. I learned Python after switching from Matlab and have honed my skills over the years by looking at well-designed code bases like those mentioned above, and by watching various YouTube tutorials and other online sources. In fact I have never read a book on Python programming (of machine learning or deep learning or statistics for that matter, but I'll get into that a bit more later).

2. Learn Software Design Principles

Ok, so you've learned Python. Good to go, right? Well not really, I knew how to use Python for a pretty long time before I actually learned how to design software. I always use the analogy of building a house. You could be really good at using all of the tools, but if you don't have a blueprint, know the various codes and standards, or know where and how to get all of the materials, then you are out of luck. While knowing the Python language is very important, actually knowing software design principles (mostly language agnostic) will really let you stand out.

This section in itself could serve as several blog posts but I'll just mention the main highlights here:

Object Oriented Programming (OOP): While there is nothing wrong with functional programming and you will indeed use a mix of OOP and functional programming in your day-to-day existence. The ML ecosystem (i.e. the packages mentioned above, especially PyTorch and scikit-learn) is mostly in the OOP camp. Being able to make sense of OOP based packages and being able to write your own OOP-based code is super important.
Unit testing and continuous integration (CI): These two usually go hand-in-hand. Unit testing refers to writing specific tests of your individual code "units.” This serves a check that your code is right but also serves as a fail-safe against breaking your code when you make changes. CI basically involves automating this test process and can also involve other types of tests other than unit tests. Having a grasp on this will give you a huge advantage.
Design Patterns: These are re-usable patterns that solve specific problems. These patterns are mostly language agnostic. Now, you don't need to know all of the different patterns and you don't need to always use them; however, having a basic understanding of what they do, (and that they exist) will put you in a great position.
Documentation: In your academic career you probably wrote code that was just for you. You knew how to use it and it worked. As part of a data science team, other people will need to be able to use and understand your code. Code documentation is of paramount importance and being aware of the best practices is super important.

The list above only scratches the surface in terms of details. I provided a basic tutorial link for each, but there are a lot of other really good resources to learn more about proper coding practices and tools. Here are a few:

Making Pythonic code: Great talk on code re-factoring and making beautiful code.
Deep learning library from scratch: Good introduction to package design and learning how to think about building a package.
calmcode.io: Lots of useful videos on various tools and techniques
Traversy Media: Good introduction to the web-developnent side of Python.

I'll end this section by noting that, in my experience, most data scientists (especially of the machine learning engineer variety) only have limited knowledge of these things. So, this is not something that you will be lacking in terms of your competition. However, if you do know these things and take the time to learn them, then it will put you in a position to stand out above your competition. More importantly, it will make you a better data scientist.

Ok, here is where the standard advice comes in, but it is standard for a reason. If you are coming from an academic background and not a data science background, you have to do some projects to fill in for your lack of on-the-job experience. In some ways this is a bonus, especially if you are competing against people who have gone to school for data science or a similar field. Remember what I said about being self-motivated. For those in a data science or analytics program, most of their experience is through classes of which they "had" to do. For you, the self-taught data scientist, you show that you are self-motivated right away by presenting data science projects you have done.

Now there are lots of different kinds of projects you can do. I'll list three here in rough order of importance:

Data storytelling project: In my opinion, this kind of project is the most useful that you can do. Basically, choose an area that you are interested in, find data, analyze it and tell a story. You don't really even need to do any ML. These projects show off your skills with raw unfiltered data, (try not to use curated data sets) and can also give you some experience in the data science struggle. In my case I looked at police shootings of civilians and pulled in data from a few databases, census data, FBI crime statistics and local police dept data. In the process, I struggled big time to pull all of these sources together and learned a ton about pandas and various python mapping packages.
Full end-to-end ML application: This kind of project would be very impressive and will show that you have experience with actual ML deployment, which is missing in a lot of candidates and actual data scientists, (myself included).
Kaggle: You can learn a lot about ML just from doing some Kaggle competitions and looking at the public notebooks. These are on the list because they are important for you go get some experience applying ML to data, but they are last because this kind of work is now ubiquitous so it won't set you apart too much.

Once you have done some projects and gained some ML and data wrangling skills with a large dose of struggle, now you can sharpen up on your communication skills.

First off, definitely put whatever you do on GitHub which will make your work public and teach you about version control if you didn't already know about it. Make sure you document your project with a project README. This is just the lowest level of sharing your work.

Next, you may want to share this work on a personal blog or on Medium. This is another great way to hone your communication skills and to ensure that you actually understood your project. The best way to test your knowledge is to try to explain your project to someone else. However, writing a blog post is still somewhat passive and these skills are not the main communication skills you will need to be a successful Data Scientist.

Lastly, you can present your work by giving a talk. There are probably a lot of forums for this kind of thing. In my case I looked on Meetup for any local Data Science related groups. It’s probably a good idea to attend a few meetups first and introduce yourself to the organizers. They are usually on the lookout for anyone who wants to give a talk. If you get a chance, offer to give a presentation on one of your projects.

So, this sounds like a lot and your future company may not even look at all of this work that you have done, but even if they don't look at it, the fact that you have done all of this and developed and honed all of these skills will help you immensely in your overall career as a Data Scientist.

4. Learn the lingo (fake it till you make it)

Back when I was an undergraduate, a physics professor of mine gave me some great advice. He told us that one of the most important things in terms of your career is that you "learn the lingo,” that is, you learn the language of the field that you are in or want to be in. For example, if someone says to you that they "used a transformer model for sentiment analysis,” what do you need to take away from that statement? You definitely don't need to know in any detail what a transformer model is or how to implement it, but you should know that sentiment analysis refers to measuring how "positive" or "negative" a segment of text is and that a "transformer" model is a kind of deep learning model that is very popular in natural language processing nowadays.

  "https://jellis18.github.io/images/fake-until-make.jpeg"

alt="">

<figcaption>Knowing the lingo of your field can make or break your

career advancement.

In other words, it is far better to have a wide, but shallow understanding of many concepts instead of a very deep understanding of a few concepts. This is almost the opposite of academic research in many cases. In academia, one pretty much needs to have a very deep understanding of their area at the expense of not knowing much outside of that area.⁴ As a data scientist the exact opposite is true, it is far better to have a base understanding of many concepts and their uses rather than knowing the exact algorithmic details.

This gets back to my earlier point about learning how to learn. If you have a shallow understanding of many things then that will allow you to assess a given problem quickly and come up with a potential solution; however, to actually implement that solution you will probably need to learn more that you know at the moment. Therefore, you need to be able to learn on your feet. As an academic, it may seem like you always need to come up with something new, but in data science no one really cares if you come up with something new, they only care if you solve the problem. This means that there is no shame is scouring the internet for solutions to a problem similar to yours and adapting it to solve your problem.

All of this may seem wrong, but it is some of the best advice that I ever received and I think it has helped me immensely. Now, this all sounds nice, but how do you do it? Well, this is the hard part. I think the best way to develop this skill is to allow yourself to explore many areas, but also control yourself from digging too deep, at least in the beginning when you are still learning. Take introductory online course, follow tutorials, do some simple projects, but don't think that you need to know all of the details.

5. Learn online

Ok, now on to perhaps another controversial piece of advice. Don't read books to learn your Data Science/ML skills. At least, don't use books as a main source of information. The only thing I would recommend books for is to learn basic programming language fundamentals, (although those are probably better online as well).

For some complete anecdotal evidence of this, I have never read a book on programming, machine learning, deep learning, statistics or communication, yet in my academic career I published several papers and led large working groups. Outside of academia, I have advanced fairly quickly and have led projects in several ML areas and I achieved all of that strictly from learning online.

Now, learning online does not mean that you passively sit back and watch some YouTube videos. It means that you watch some YouTube videos or online courses or read blog posts or arXiv papers and then try the things out yourself. You struggle. You watch and read some more. Struggle some more. Apply the techniques in your work or in your projects. Struggle some more. By the end of this struggle session, you will realize that you have managed to learn a whole lot, not by some ground up foundational approach but by a more stochastic process of trial and error which eventually leads to a much deeper understanding of the material.

This may all sound crazy and it may not work for you, but it has worked for me and it has worked for a lot of other great Data Scientists and Machine Learning Engineers. I have a very shotgun approach to learning in which I dabble in many different things, but I can offer a few good resources here in addition to the resources that I have already sprinkled throughout this post.

Coursera: There are a lot of great courses here that can give you a good introduction to many topics especially machine learning and deep learning. Don't expect to be an expert on anything after taking a Coursera course, but they can give you a good base to start with.
fast.ai: I would say this is the best resource for learning cutting edge Deep Learning techniques. Unfortunately, it’s backed by a software library that, in my opinion, combines all of the worst coding practices and puts them all in one place. Even so, there are two great Deep Learning courses and a more unknown Machine Learning (github here) and Linear Algebra course.
YouTube: You can pretty much find everything you want about Data Science/ML/Programming on YouTube. The problem is that the quality varies a lot. Instead of recommending specific channels or videos, (some of which I've already mentioned above) I'll just say that, for me, the most useful videos in many cases are real-time walkthroughs where they are coding in real time (this is one of the reasons fast.ai is great). However, sometimes you are just interested in a concept, so the coding doesn't matter as much. Either way, YouTube is an amazing resource for everyone to use.
Blog posts: towards data science can have some really good posts, but the mileage varies. This is mainly to say that looking at random blog posts to learn things may not be a bad idea. Even if the post itself is not very good, usually there are other references that can lead you to what you want.
Code Documentation and Source Code: Lots of large code bases have really extensive documentation and tutorials. Going through these tutorials can be a great learning experience. In some other cases looking at the source code itself can show you some good programming practices, (and in some cases can surprise you at how bad the source code for various projects is). For well-organized and excellently documented code, I would recommend PyTorch and scikit-learn for starters.
Twitter: Can be a dumpster fire, but can also be a great tool for learning new things and making new connections. I actually found my current job through a twitter friend. There are too many to list here, but search for data science or machine learning and follow some more popular accounts. Eventually, you will grow this network and use it as a kind of filter for information.

Wrap up

If you made it this far, congratulations! This post turned out to be much longer than I originally had planned. I have covered a lot of things here and it may seem daunting. When I first started on this journey, I kept thinking that there was no way I could learn all of this stuff and maybe it would just be better to stay in academia. If you are passionate about your academic field then by all means don't give up, but if you are just staying in because you think there is no other option then that is a completely wrong assumption.

All of the things I have mentioned in this post are things I have learned over the previous 3.5 years as a Data Scientist. I did not do all of these things in the very beginning and have learned a lot since then.

Lastly, if you are in this transition and you are feeling stressed, remember once you gain these skills you will be a hot commodity and will be able to get a fulfilling job almost anywhere.

If you are reading this and you think "This guy has no idea what he is talking about,” you would be right. I know almost nothing about big data, which is the point I make in the next paragraph. Hang in there... ↩
While this term is gaining steam, I think it should be much more prevalent as it is a fairly precise statement of the job role. ↩
I should note that this is getting better now with students being encouraged to think about alternative career roles and even having guest speakers from outside of academia. ↩
That is not to say that this is the desired state, but more that the incentives in academia lead to this outcome. The point my professor was making was that we should not do this, which is relatively easy in a field like data science, but much harder in academic physics. ↩

Justin A. Ellis

A fun multi-lingual concurrency experiment

Go Version

Python Version

Bash Version

Benchmarking

Conclusion

Production Dockerfile Tips and Tricks

Small base image and deterministic builds

Privileged non-root user

Separating dependencies from source code

Move things that change often close to bottom of Dockerfile

Multi-stage builds for smaller images

Conclusion

Some Advanced Typing Concepts in Python

Type annotations and the interface segregation principle

Some general good practices

Protocols

Generics with Protocol Constraints

Parameter Specification Variables

Overloads

Wrapping Up

Early and late binding closures in Python

What are closures (the short version)

The late-binding vs. early-binding problem

How to avoid foot shooting

Access Modifiers in Python

Private vs Protected

Readonly attributes

Final classes and methods

Conclusion

Abstract Base Classes and Protocols: What Are They? When To Use Them?? Lets Find Out!

What are Abstract Base Classes

Pure ABCs (ABC as Interface)

ABCs as a tool for code reuse

What are Protocols

Protocols as Interfaces

Protocols as Generic Type Bounds

So ABC or Protocol?

A Modern and Explicit approach to Python Exceptions

Traditional Error Handling in Python

Error Handling in Rust

Error Handling in Go

A Go-like error handling pattern in Python

A Rust-like error handling pattern in Python

Summary

Building a fully typed LRU Cache in Python

What we will end up with

Building the cache class

Building the function wrapper class

Building the decorator

Summary

A Practical Guide to MCMC Part 2: Further Resources

MCMC Resources

How to transition from Academia to Data Science

What is Data Science anyway?

How has Academia already prepared me for this role?

Learning the struggle

Learning how to learn

"Soft Skills"

Practical (and very opinionated) advice

1. Learn Python

2. Learn Software Design Principles

3. Do some projects (and share them)

4. Learn the lingo (fake it till you make it)

5. Learn online

Wrap up