Fredrik Håård's Blaag

@fhaard
I'm a programmer, consultant, developer, occasional teacher and speaker. Among my least disliked programming languages are Python, and a majority of these posts are related to Python in one way or another.
RSS Feed

Python Closures and Decorators (Pt. 2)

Edit: got complaints that code was hard to read, trying out Pygments.

In part 1, we looked at sending functions as arguments to other functions, at nesting functinons, and finally we wrapped a function in another function. We'll begin this part by giving an example implementation on the exercise I gave in part 1:

>>> def print_call(fn):
...   def fn_wrap(*args, **kwargs):
...     print("Calling %s with arguments: \n\targs: %s\n\tkwargs:%s" % (
...            fn.__name__, args, kwargs))
...     retval = fn(*args, **kwargs)
...     print("%s returning '%s'" % (fn.func_name, retval))
...     return retval
...   fn_wrap.func_name = fn.func_name
...   return fn_wrap
...
>>> def greeter(greeting, what='world'):
...     return "%s %s!" % (greeting, what)
...
>>> greeter = print_call(greeter)
>>> greeter("Hi")
Calling greeter with arguments:
     args: ('Hi',)
     kwargs:{}
greeter returning 'Hi world!'
'Hi world!'
>>> greeter("Hi", what="Python")
Calling greeter with arguments:
     args: ('Hi',)
     kwargs:{'what': 'Python'}
greeter returning 'Hi Python!'
'Hi Python!'
>>>

So, this is at least mildly useful, but it'll get better! You may or may not have heard of closures, and you may have heard any of a large number of defenitions of what a closure is - I won't go into nitpicking, but just say that a closure is a block of code (for example a function) that captures (or closes over) non-local (free) variables. If this is all gibberish to you, you're probably in need of a CS refresher, but fear not - I'll show by example, and the concept is easy enough to understand: a function can reference variables that are defined in the function's enclosing scope.

For example, take a look at this code:

>>> a = 0
>>> def get_a():
...   return a
...
>>> get_a()
0
>>> a = 3
>>> get_a()
3

As you can see, the function get_a can get the value of a, and will be able to read the updated value. However, there is a limitation - a captured variable cannot be written to:

>>> def set_a(val):
...   a = val
...
>>> set_a(4)
>>> a
3

What happened here? Since a closure cannot write to any captured variables, a = val actually writes to a local variable a that shadows the module-level a that we wanted to write to. To get around this limitation (which may or may not be a good idea), we can use a container type:

>>> class A(object): pass
...
>>> a = A()
>>> a.value = 1
>>> def set_a(val):
...   a.value = val
...
>>> a.value
1
>>> set_a(5)
>>> a.value
5

So, with the knowledge that a function captures variables from it's enclosing scope, we're finally approaching something interesting, and we'll start by implementing a partial. A partial is an instance of a function where you have already filled in some or all of the arguments; let's say, for example that you have a session with username and password stored, and a function that queries some backend layer which takes different arguments but always require credentials. Instead of passing the credentials manually every time, we can use a partial to pre-fill those values:

>>> #Our 'backend' function
... def get_stuff(user, pw, stuff_id):
...   """Here we would presumably fetch data using the supplied
...   credentials and id"""
...   print("get_stuff called with user: %s, pw: %s, stuff_id: %s" % (
...         user, pw, stuff_id))
>>> def partial(fn, *args, **kwargs):
...   def fn_part(*fn_args, **fn_kwargs):
...     kwargs.update(fn_kwargs)
...     return fn(*args + fn_args, **kwargs)
...   return fn_part
...
>>> my_stuff = partial(get_stuff, 'myuser', 'mypwd')
>>> my_stuff(3)
get_stuff called with user: myuser, pw: mypwd, stuff_id: 3
>>> my_stuff(67)
get_stuff called with user: myuser, pw: mypwd, stuff_id: 67

Partials can be used in numerous places to remove code duplication where a function is called in different places with the same, or almost the same, arguments. Of course, you don't have to implement it yourself; just do from functools import partial.

Finally, we'll take a look at function decorators (there may be a post on class decorators in the future). A function decorator is (can be implemented as) a function that takes a function as parameter and returns a new function. Sounds familiar? It should, because we've already implemented a working decorator: our print_call function is ready to be used as-is:

>>> @print_call
... def will_be_logged(arg):
...   return arg*5
...
>>> will_be_logged("!")
Calling will_be_logged with arguments:
     args: ('!',)
     kwargs:{}
will_be_logged returning '!!!!!'
'!!!!!'

Using the @-notation is simply a convenient shorthand to doing:

>>> def will_be_logged(arg):
...   return arg*5
...
>>> will_be_logged = print_call(will_be_logged)

But what if we want to be able to parameterize the decorator? In this case, the function used as a decorator will received the arguments, and will be expected to return a function that wraps the decorated function:

>>> def require(role):
...   def wrapper(fn):
...     def new_fn(*args, **kwargs):
...       if not role in kwargs.get('roles', []):
...         print("%s not in %s" % (role, kwargs.get('roles', [])))
...         raise Exception("Unauthorized")
...       return fn(*args, **kwargs)
...     return new_fn
...   return wrapper
...
>>> @require('admin')
... def get_users(**kwargs):
...   return ('Alice', 'Bob')
...
>>> get_users()
admin not in []
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 7, in new_fn
Exception: Unauthorized
>>> get_users(roles=['user', 'editor'])
admin not in ['user', 'editor']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 7, in new_fn
Exception: Unauthorized
>>> get_users(roles=['user', 'admin'])
('Alice', 'Bob')

...and there you have it. You are now ready to write decorators, and perhaps use them to write aspect-oriented Python; adding @cache, @trace, @throttle are all trivial (and before you add @cache, do check functools once more if you're using Python 3!).

Blaag created 120301 19:04
blog comments powered by Disqus


Page created using blaag and abusing docutils. RSS Feed generated using PyRSS2Gen.