The Versatility of __call__: A Python Developer’s Secret Weapon
While reviewing the code in my current chatbot project, I discovered asyncio.to_thread accepts function parameters, removing the need to wrap the function with partial() as shown in the previous post. As a quick recap, wrapping a function with partial() allows a partial invocation of a function, by defining a subset of the arguments. The resulting callable can be called later with the remaining arguments (or further wrapped by additional partial() calls).
Refactoring with __call__: From Personal Code to FastAPI
Consider the following fictional example,
def foo(arg_a, arg_b, arg_c):
...Suppose we have both arg_b and arg_c defined, but we do not know about arg_a. If both arg_b and arg_c do not affect the behaviour of later computations, it would be nice to not having to keep passing the values around. Fortunately, Python allows the definition of higher-order functions. Higher-order functions are functions accepting functions as arguments. A form of encapsulation can then be achieved, by partially invoking function foo and pass the resulting callable objects to other functions.
from functools import partial
new_foo = partial(foo, arg_b="bar", arg_c="baz")
...
later_calculation(new_foo)I take a lot of inspiration from functional programming paradigm in code. I certainly see some resemblance between partial() and a concept called currying. In languages that allow currying, if we invoke a function without all the arguments, it returns a callable. The invocation can be completed later with the callable with the remaining arguments. My favourite library toolz, which offers useful tools inspired by functional programming, has a function decorator that allows Python to mimic this behaviour.
Back to the refactoring exercise mentioned earlier. I mostly use Visual Studio Code for my work, and the powerful search function is useful to find all the lines calling the partial function. While removing the extra partial invocations for asyncio.to_thread calls, I realized I also use that a lot for other things. Not realizing it as a problem at first, I turned my attention towards the FastAPI application, seeking a way to reduce definition of global variables.
We are often reminded on why we should not define globals, but when used cautiously, it is a useful and convenient tool. For instance, we can setup a logger, and use it throughout the module to log useful information about program execution.
import logging
from fastapi import FastAPI
app = FastAPI()
logger = logging.getLogger(__name__)
@app.get('/')
def index() -> str:
logger.info("Received web request")
return "Hello world"Suppose I want to reduce the use of globals, and only include logger in the request handler functions whenever needed. FastAPI offers dependency injection through the use of Depends function. The Depends function requires a callable, and FastAPI calls it whenever needed, but where do I pass the __name__ argument? Thankfully, this is addressed in the documentation in the advanced usage section.
Hint hint, it involves __call__. Adapting this to our example,
import logging
from typing import Any
from dataclass import dataclass
from fastapi import FastAPI, Depends
app = FastAPI()
@dataclass(frozen=True)
class get_logger:
name: str
def __call__(self) -> Any:
return logging.getLogger(self.name)
@app.get('/')
def index(logger: Any = Depends(get_logger(__name__))) -> str:
logger.info("Received web request")
return "Hello world" It is indeed possible to write this with partial, as we discussed earlier. According to the discussion in this issue on github, all it takes is to rewrite get_logger as an async generator, as shown below.
from collections.abc import AsyncGenerator
async get_logger(name: str) -> AsyncGenerator[Any, None]:
yield logging.getLogger(name)
@app.get('/')
def index(logger: Any = Depends(partial(get_logger, __name__))) -> str:
logger.info("Received web request")
return "Hello world" Now let us dial back a little bit and explain what __call__ is. It is a magic or dunder (short for double underscore) method in Python that makes objects instantiated from the class callable, as if they are a function. In our simplified example, the class is implemented as a dataclass to have the constructor generated automatically (frozen=True to allow FastAPI to cache the result). The callable object can also expect function parameters, as shown below,
@dataclass(frozen=True)
class get_logger:
name: str
def __call__(self, arg_a):
logger = logging.getLogger(self.name)
logger.info("Logger is initiated with arg_a:%s", arg_a)
return logger
logger_getter = get_logger(__name__)
logger = logger_getter("bar") While I cannot demonstrate it in our simplified example, defining a callable as a class also brings benefits such as state management, seen in a normal class. It is also especially useful in cases where complex setup is needed.
Remember the refactoring exercise I mentioned earlier, where I found many uses of partial function calls in my code? The use of __call__ in my FastAPI application inspired me to look into other parts of the chatbot. Specifically, this section, which was also featured in the article I posted last week.
import signal
from functools import partial
def shutdown_handler(
_signum, _frame, exit_event: threading.Event
) -> None:
exit_event.set()
manager = multiprocessing.Manager()
exit_event = manager.Event()
for s in (signal.SIGHUP, signal.SIGTERM, signal.SIGINT):
signal.signal(
s, partial(shutdown_handler, exit_event=exit_event)
)
with ProcessPoolExecutor() as executor:
executor.submit(process_run, tg_run, exit_event)
executor.submit(process_run, web_run, exit_event)In the snippet, shutdown_handler is a function I pass to signal.signal. According to the documentation, the handler function expects a signal number, as well as a stack frame. I needed it to send exit signals to processes whenever any of those signal is captured, hence I hacked exit_event into the handler with use of partial.
After the refactoring exercise done to the FastAPI application, I started to think if this is the best use of partial. In my production code, I also pass in some other arguments. Eventually, it becomes harder to recognize that the function is meant to be a handler as the list of arguments grows (increasing argument list sometimes also means that the function’s scope needs revision, but that’s a topic for another day).
I am breaking the naming convention on purpose, both to preserve the existing name and emphasize its callable nature. Putting naming issues aside for now, if we refactor the code to use __call__ it becomes
from types import FrameType
from dataclasses import dataclass
@dataclass
class shutdown_handler:
exit_event: Event
def __call__(self, signal_num: int | None, frame: FrameType | None) -> None:
self.logger.info(
"MAIN: Sending exit event to all tasks in pool",
signal=signal_num,
frame=frame,
)
self.exit_event.set()
...
with ProcessPoolExecutor(max_workers=5) as executor, suppress(queue.Empty):
for signal_num in (signal.SIGHUP, signal.SIGTERM, signal.SIGINT):
signal.signal(signal_num, shutdown_handler(exit_event))Compared to the previous version, I find this significantly more readable and understandable. By reading the __call__ method, it becomes apparent that the callable object is written to act as a signal handler.
I briefly mentioned that defining a callable as a class implementing __call__ allow storing of states previously. Alternatively, I want to explore another approach to doing this, through genruler, the DSL library I recently reimplemented.
__call__ vs. Closures: A Genruler Exploration
Genruler is a DSL library, where it defines and parses a lisp-like domain-specifc language. Like other programming languages, it defines functions. As an example for our next discussion, we take a look at string.concat. It is mapped to a function written in Python, which is simplified below,
def concat(link: str, *arguments: str) -> Callable[[dict[Any, Any]], str]:
def inner(context: dict[Any, Any]) -> str:
return link.join(arguments)
return inner The function is written as a closure, according to MDN Web Docs (while the context is JavaScript, it applies to Python too),
A closure is the combination of a function bundled together (enclosed) with references to its surrounding state (the lexical environment). In other words, a closure gives a function access to its outer scope. In JavaScript, closures are created every time a function is created, at function creation time.
Genruler works by first parsing the code written in the DSL. Then it is only evaluated when a context is given (think of it as a source of data), but for our discussion it is inessential. We just need to know that evaluation only happens once a context is given.
Therefore, in our simplified example, it also follows the two stage procedure. When it is called with the link and a list of string (parsed and extracted from user’s code), it returns the inner function. When it comes to the time to evaluate, the inner function is called with the context. As mentioned, it is a string concatenation function, which is really just the string .join() method.
In this case, the function concat fits the definition of a closure, as its inner function does have access to the outer scope (link and arguments). This, is how state can be managed, albeit in a more limited capacity, in a function closure.
Genruler is an experimental project, as this reimplementation isn’t made to be used in a real project, unlike the previous incarnation. I tested some ideas while working on the library, and one of it was figuring out if it is possible to translate the closure back to the lisp-like syntax. It was inconclusive, but in one of my attempts, I rewrote a module to use __call__ dunder.
The rewrite is fairly simple, for example, the concat function we saw earlier, can be rewritten as,
class concat:
def __init__(self, link: str, *arguments: str):
self.link = link
self.arguments = arguments
def __call__(self, context: dict[Any, Any]) -> str:
return self.link.join(self.arguments) Both code snippet achieve similar goals, my only nitpick is that for the function closure version, I cannot really rewrite inner as a lambda function. It would work for the example, but lambda in Python has a limitation where only one statement is allowed. Should the function contains more than one statement, it would fail.
On readability, I find both acceptable, but people who are not as familiar with functional programming may prefer the class-based syntax implementing __call__ dunder. In the end, it is a very subjective matter, depending on the environment where the code is deployed.
I still have not picked a favourite between the two type of representation. Functionally, they work the same. The compiled bytecode may show some difference in performance, but it is still too early to delve into the topic. The point of this article, is more on showcasing how __call__ can be an alternative.
The Power of __call__ and the Zen of Python
It is amazing how a seemingly irrelevant discovery eventually leads to the writing of this article. The refactoring exercise shows a pattern I find worthy of further observation. While Python supports certain functional programming construct, it is not designed for that (would be nice to have immutable dictionary, for instance). Therefore, in some cases, we could really consider writing callback functions, with __call__ dunder rather than partial.
The __call__ dunder can also be a viable alternative for function closures, as shown above. Combining the properties of a class, it allows for more flexibility, and can still be callable.
There is this saying, quoted from the Zen of Python, by Tim Peters
There should be one — and preferably only one — obvious way to do it.
Programming isn’t really just an exercise of expressing ideas in a way computers understand. More often than not, it is a tool that allow us to communicate our thought process. The fun thing (or otherwise) about having individual thought is that we may approach the same problem differently.
In one of my recent job interview, I was asked if I classify myself as a neurotypical. According to the interviewer, a person is considered neurotypical if his or her thoughts and views follows the consensus. It gets harder to classify myself thesedays, as the advance of technology makes it easier for us to express our thoughts. In fact, it takes more effort keeping things to oneself.
In such settings, it is hard to define what majority means in a world that is so diverse. I do find my opinion follows the crowd in one topic, but not some others. On the other hand, isn’t this is where innovation comes? I haven’t heard from my interviewer since, but I still find the conversation enlightening.
Back to the quote, there may be one true way to solve any given problem, but it doesn’t mean we shouldn’t explore. We now live in a world, where everyone is seeking for the quickest solution to problem. In the context of our discussion today, I hope I have done my job to show some of the possibilities __call__ dunder can bring.
That’s all I have for this week, thanks for reading, and I shall write again, next week.
This article received editorial assistance from a large language model to improve readability and to ensure clarity, while all content and code remain my own. I’m still open to project collaborations and job opportunities as the enlightening conversation didn’t lead to anything. Reach me via Medium or LinkedIn for more information.
