Python for Snakecharmers

Presenter Notes

Presenter Notes

The Zen of Python (PEP20)

Beautiful is better than ugly.

Explicit is better than implicit.

Simple is better than complex.

Complex is better than complicated.

Flat is better than nested.

Sparse is better than dense.

Readability counts.

Special cases aren't special enough to break the rules.

Although practicality beats purity.

Presenter Notes

The Zen of Python Continued

Errors should never pass silently.

Unless explicitly silenced.

In the face of ambiguity, refuse the temptation to guess.

There should be one-- and preferably only one --obvious way to do it.

Although that way may not be obvious at first unless you're Dutch.

Now is better than never.

Although never is often better than right now.

If the implementation is hard to explain, it's a bad idea.

If the implementation is easy to explain, it may be a good idea.

Namespaces are one honking great idea -- let's do more of those!

Presenter Notes

PEP: Python Enhancement Proposals

python.org/dev/peps

Presenter Notes

PEP8

A Style Guide for Python

1     # yes -- lined up with opening delimiter
2     foo = long_function_name(var_one, var_two,
3                             var_three, var_four)
4     print var_one
5 
6     # no
7     foo = long_function_name(var_one, var_two,
8         var_three, var_four)
9     print var_one
  • Use 4 spaces per indentation level
  • Never mix tabs and spaces

Presenter Notes

PEP8 Continued

Whitespace

  • Separate classes and top-level functions with 2 blank lines
  • Separate methods with single blank lines

Imports

1     # yes
2     import os
3     import sys
4     from subprocess import Popen, PIPE
5 
6     # no
7     import os, sys

Presenter Notes

More PEP8

Block Style

 1 # yes
 2 if foo == 'blah':
 3     one()
 4     two()
 5     three()
 6 
 7 # no
 8 if foo == 'blah': do_blah_thing()
 9 for x in lst: total += x
10 
11 # no...just...no
12 if foo == 'blah': one(); two(); three = \
13 four()

Presenter Notes

File I/O — Simple

 1 >>> fh = file("myfile.txt")
 2     <open file 'seminars.txt', mode 'r' at 0x15938a0>
 3 >>> print fh.read()
 4     Sometimes it is best
 5     To forget sysadmins
 6     Who write all in Perl
 7 >>> fh.seek(0)
 8 >>> print fh.readline()
 9     Sometimes it is best
10 >>> print fh.readline()
11     To forget sysadmins
12 >>> fh.close()

Presenter Notes

File I/O — Advanced

1 >>> fh = file("myfile", 'w')
2 >>> fh.read()
3     IOError: File not open for reading
4 >>> fh.close()
5 >>> import tempfile
6 >>> temp = tempfile.TemporaryFile(mode='wb')
7 >>> temp
8     <open file '<fdopen>', mode 'wb' at 0x1593930>
9 >>> temp.close()

tempfile.TemporaryFile is destroyed after being closed

Files can be opened in several modes (or combinations thereof)

  • r — read
  • w — write
  • a — append
  • b — binary
  • + — simultaneous read and write

Presenter Notes

Other Handy Stuff

Only need a file for a little bit?

 1 from tempfile import TemporaryFile
 2 from subprocess import Popen
 3 out = TemporaryFile(mode="r+w")
 4 p = Popen('ls', stdout=out)
 5 # displays where you are in a file
 6 out.tell()
 7     "4L"
 8 # go to a specific point
 9 out.seek(0)
10 print out.read()
11     Documents
12     Pictures
13     Music
14     Desktop

Need to ensure a file opened?

1 with file('myfile', 'r+w') as fh:
2     data = fh.read()
3     #do stuff

Presenter Notes

Lambdas

They won't remember your name in the morning.

1 >>> lambda x: x==4
2 <function <lambda> at 0x7f944ed0e938>

Just like a one-night stand, lambdas are great if you just need a quickie

Pass them to other functions for non-namespace-cluttering Pythonic justice

1 >>> filter(lambda x: x%4==0, [2,4,6,8,10,12])
2 [4, 8, 12]
3 help(filter)
4 filter(...)
5     filter(function or None, sequence) -> list, tuple, or string
6 
7     Return those items of sequence for which function(item) is true.  If
8     function is None, return the items that are true.  If sequence is a tuple
9     or string, return the same type, else return a list.

Awesome.

Presenter Notes

Builtins

Imported anytime you run Python, has tons of handy stuff

  • errors ( ImportError, AttributeError, ZeroDivisionError )
  • types we love ( dict, set, int, str )
  • handy constants ( True, False, None )

Also some handy functions

 1 >>> __builtin__.globals()
 2 # returns a dict of all global variables
 3 >>> y = "whoami"
 4 >>> __builtin__.id(y) # id function returns an object's memory address
 5 14637984
 6 >>> __builtin__.coerce(9.4, 0x023) # coerces any two numbers to same notation
 7 # if coercion is not possible, raise TypeError
 8 (9.4, 35.0)
 9 >>> __builtin__.coerce(9.4, 'eleventy-one')
10 TypeError: number coercion failed

Presenter Notes

Builtins Continued

Other builtin stuff

  • next
  • filter
  • map
  • reduce
  • pow
  • ord
  • hasattr
  • cmp

Presenter Notes

Do You Believe in Magic?

Creation

  • __init__
  • __new__

Equality

  • __eq__ #equal to
  • __lt__ #less than
  • __le__ #less than or equal to
  • __gt__ #greater than
  • __ge__ #greater than or equal to

Presenter Notes

Magic Methods Continued

Class/Object

  • __module__ #the module an object is a part of
  • __class__ #the fully qualified class of an object
  • __base__ #base class of the object
  • __bases__ #a list of base classes
  • __subclasscheck__ #pass in an object to see if it is a subclass of this one
  • __subclasses__ #find everything that is a subclass of this one
  • __instancecheck__ #pass in an object to see if it is an of this class
  • __call__ #shorthand of this is ()
  • __dict__ #returns a dict of (by default) all the attributes of an instance

Presenter Notes

Magic Methods, Still

Attributes

  • __getattribute__
  • __setattr__
  • __delattr__

Coercion

  • __repr__ #return a representation of an object <object at 0x7f4e3b92c270>
  • __str__ #stringify the object
  • __format__ #default object formatter
  • __name__ #string of the name of the object

Presenter Notes

Type Coercion

Coercion vs. Casting

1 2.0 + float(1) #cast
2 1.0 + 2  #coerce

Python tries to do what you mean

Uses functions like object.str(), or object.int() depending on operator

 1 #str + int
 2 >>> "hello" + 8
 3 TypeError: cannot concatenate 'str' and 'int' objects
 4 #str + str
 5 >>> "hi" + " " + "there"
 6 'hi there'
 7 #str + int
 8 >>> "hi" * 2
 9 "hi"
10 #float + int
11 >>> 9.9 + 9
12 18.9
13 #tuple + tuple
14 (9,10,11) + (12, )
15 (9, 10, 11, 12)

Presenter Notes

Iterators and Iterables

Tons of things are iterable!

1 >>> for word in ["hello", "there", "snakecharmer"]:
2 >>>     #iterate over words in a list
3 >>>     print word
4 hello
5 there
6 snakecharmer

Iterables must:

  • Make it possible to traverse a list
  • Guarantee (eventual) reachability of all values
  • Allow the use the object.next() operator to reach the next value

Presenter Notes

Itertools

 1 a = [1, 2, 3, 4]
 2 b = ['a', 'b', 'c', 'd']
 3 # roughly equivalent to a.extend(b), but does not store the result in a
 4 itertoolsmodule.chain(a, b) 
 5 # all combinations of length 3 of all items in a list.
 6 itertoolsmodule.combinations(a, 3)
 7 # return elements from a until b is exhausted, then repeat infinitely
 8 itertoolsmodule.cycle(b)
 9 # just like the builtin "filter" function
10 # there's also ifilterfalse which is the inverse, obviously
11 itertoolsmodule.ifilter(lambda x: x % 2 == 0)
12 # returns (key, subset) grouped by the value of second_arg(first arg)
13 itertoolsmodule.groupby(a, lambda x: x % 2 == 0)
14 # returns three independent iterables of the contents of a
15 itertools.tee(a, 3)

Presenter Notes

Generators

A very special iterable

Only creates results when they are asked for

All thanks to the "yield" keyword

Preseve their state and will calculate the next result when the .next() method is called

Use "yield" where you might normally use "return"

Presenter Notes

Generator Examples

Primes

1 def fib_gen();
2     a = 1
3     b = 1
4     while True;
5         yield a
6         (a, b) = (b, a + b)

Usage

 1 >>> fibs = fib_gen()
 2 >>> for i in range(10): print fibs.next()
 3 1
 4 1
 5 2
 6 3
 7 5
 8 8
 9 13
10 21
11 34
12 55

Presenter Notes

Generator Examples Continued

Walk a Directory

1 def filename_generator(dirname):
2     subdirlist = [dirname]
3     while subdirlist:
4         dirname = subdirlist.pop()
5         for item in os.listdir(dirname):
6             if os.path.isfile(os.path.join(dirname, item)):
7                 yield os.path.join(dirname, item)
8             else:
9                 subdirlist.append(os.path.join(dirname, item))

Usage

1 >>> files = filename_generator('/home')
2 >>> print files.next()
3 "afile.txt"
4 >>> while True: files.next() # Eventually generators can be exhausted
5 StopIteration

Presenter Notes

Controlling Generators

1 def counter (maximum):
2     i = 0
3     while i < maximum:
4         val = (yield i)
5         # If value provided, change counter
6         if val is not None: i = val
7         else: i += 1

You can tell a generator to continue as if the "val" in gen.send(val) was its last yield

 1 >>> it = counter(10)
 2 >>> print it.next(), " | ", it.next()
 3 0 | 1
 4 >>> print it.send(8)
 5 8
 6 >>> it.next()
 7 9
 8 >>> print it.next()
 9 Traceback (most recent call last):
10 File "t.py", line 15
11     print it.next()
12 StopIteration

Presenter Notes

New-Style Classes

New-style classes are intended to unify classes and types

(also enable metaclasses, but let's not go there, yet)

1 class OldStyleClass():
2     pass
3 class NewStyleClass(object):
4     # new-style classes all inherit from object, directly...
5     pass
6 class ChildNewStyleClass(NewStyleClass):
7     # ...or indirectly
8     pass

Method resolution order

  • multiple inheritance
  • methods are resolved in order of inheritance declaration

    1 class A(object): pass
    2 class B(object): pass
    3 class C(A, B): pass
    

Presenter Notes

New-Style Classes Continued

the super() method

  • less unnecessary verbosity
  • pretty standard OOP concept

    1 # assume that A, B, and Call have a method "m"
    2 # and inherit like this: A -> B -> C
    3 class D(C):
    4     def m(self): self._dothings; A.m(), B.m(), C.m()
    5     #but what if the heirarchy changes?
    6 
    7 #if we used super
    8 class D(C):
    9     def m(self): self._dothings; super(C, self)
    

Presenter Notes

New-Style Classes Continued

the new initialization step

  • runs before init
  • used by a lot of the builtin types like str and int
  • needed if you want to override the constructor of an immutable type
  • first argument must be a class

Presenter Notes

Decorators

Decorators wrap a function to add functionality

Think like you are wrapping a function up in another function

Did I mention that functions act like classes? They can be decorated too!

Python comes with some default decorators

1 @staticmethod
2 @classmethod
3 @property

@staticmethod

  • Has no access to the class it is a part of
  • Used for dependency injection and not much else
  • Unlike Java, in Python modules are the preferred method of code organization

Presenter Notes

Default Decorators Continued

@classmethod

  • Is passed the class it is called on
  • Can be called on the class itself, not just on an instance of it
  • Handy for writing ORMs, for example

@property

  • Is a function that can be called like a normal attribute
  • can have @<propertyname>.setter and .getter if you like
  • Or just getter, if the property shouldn't be changed by the user

Now it's time to roll our own!

Presenter Notes

Decorator Enforced Singletons

 1 def singleton(cls):
 2     """
 3     A decorator to make a class into a singleton.
 4 
 5     When a class is wrapped in this decorator, this decorator will always make
 6     any calls on that class (or an instance of it) goes to the same instance of
 7     a given class.
 8 
 9     See PEP 318 for clarification.
10     http://www.python.org/dev/peps/pep-0318/#examples
11     """
12     instances = {}
13     def getinstance():
14         if cls not in instances:
15             instances[cls] = cls()
16         return instances[cls]
17     return getinstance
18 
19 @singleton
20 class SingletonDict(dict):
21     pass

Presenter Notes

Decorator for Memoizing Functions

 1 class memoized(object):
 2     """Decorator that caches a function's return value each time it is called.
 3     If called later with the same arguments, the cached value is returned, and
 4     not re-evaluated.
 5     """
 6     def __init__(self, func):
 7         self.func = func
 8         self.cache = {}
 9     def __call__(self, *args):
10         try:
11             return self.cache[args]
12         except KeyError:
13             value = self.func(*args)
14             print "caching new value"
15             self.cache[args] = value
16             return value
17         except TypeError:
18             # uncachable -- for instance, passing a list as an argument.
19             # Better to not cache than to blow up entirely.
20             return self.func(*args)
21     def __repr__(self):
22         """Return the function's docstring."""
23         return self.func.__doc__
24     def __get__(self, obj, objtype):
25         """Support instance methods."""
26         return functools.partial(self.__call__, obj)

Presenter Notes

Memoized Functions Continued

1     @memoized
2     def fibonacci(n):
3         "Return the nth fibonacci number."
4         if n in (0, 1):
5             return n
6         return fibonacci(n-1) + fibonacci(n-2)
7 
8     print fibonacci(12)

Presenter Notes

Metaclasses

Classes writing classes

1 >>> class MyObject(object):
2 >>>     pass
3 >>> print MyObject
4 <class '__main__.MyObject'>

Remember the type keyword? Well it's not just for discovering the type of an object

In Python, easy ways to do things (using the class keyword to make classes) can be done manually if you want to

1 >>> Foo = type('Foo', (object,), {'myattr':'bar'})
2 >>> print Foo
3 <class '__main__.Foo'>
4 >>> print Foo.myattr()
5 bar

That has the same result as using the class keyword

Did you catch that we used a string to name the class?

How about that we created attributes using a dictionary?

Presenter Notes

So What's a Metaclass Do?

1 # A class is an instantiation of a metaclass (kind of)
2 MyClass = MetaClass()
3 # An object is an instantiation of a class
4 MyObject = MyClass()

Surprise: type is actually a metaclass, which is why we were just able to make classes by calling it

Don't forget, everything is an object in Python, which means it was created from a class. Doesn't that mean primitive like int, as well as other types like str, dict, and friends are all classes?

Yes.

So what class is type?

1 >>> dict.__class__
2 type
3 >>> dict.__class__.__class__
4 type
5 >>> dict.__class__.__class__.__class__
6 type

Presenter Notes

How Do I Use a Metaclass?

The __metaclass__ attribute

1 class Foo(object):
2     __metaclass__ = somemetaclass
3     #python will now use somemetaclass to create Foo

What can you put in 'somemetaclass'?

Why, something that creates a class, of course

What creates a class?

type, or anything that subclasses it

__metaclass__ does NOT need to be a class

Presenter Notes

Metaclasses in Action

 1 def upper_attr(future_class_name, future_class_parents, future_class_attr):
 2     """
 3     Return a class object, with the list of its attribute turned
 4     into uppercase.
 5     """
 6 
 7     # pick up any attribute that doesn't start with '__'
 8     attrs = ((name, value) for name, value in future_class_attr.items() if not
 9                 name.startswith('__'))
10 
11     # turn them into uppercase
12     uppercase_attr = dict((name.upper(), value) for name, value in attrs)
13 
14     # let `type` do the class creation
15     return type(future_class_name, future_class_parents, uppercase_attr)
16 
17 class MyClass(object):
18     __metaclass__ = upper_attr  # this will affect all classes in the module
19     z = 'hello'
20 
21 y = MyClass()
22 print y
23 print y.Z
24 # y.z will give an AttributeError, since y.z was turned into y.Z
25 # by our metaclass

Presenter Notes

But Why?

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why). -- Tim Peters, Python Guru

ORM's like Django use metaclasses all the time

But odds are you don't need them

99% of the time, you don't need class alteration

99% of the time you DO need class alteration, you can use decorators or monkey patching

But for that 0.01% of the time, metaclasses are awesome

Presenter Notes

Unit Testing: Standard

unittest

Included in the standard library

1 import unittest
2 
3 def fun(x);
4     return x + 1
5 
6 class MyTest(unittest.TestCase);
7     def test(self);
8         self.assertEqual(fun(3), 4)

Presenter Notes

Unit Testing: Simpler

pytest

Not in standard library :( pip install pytest

1 # content of test_sample.py
2 def func(x);
3     return x + 1
4 
5 def test_answer();
6     assert func(3) == 5

Then run "py.test"

Presenter Notes

Unit Testing: Awesome

nose

Extends unittest for testing bliss

Fancy automatic test discovery

Tons of plugins for stuff like:

  • xUnit-compatible test output
  • coverage metrics
  • test selection (e.g. only running a subset of tests)

Presenter Notes

Nose Continued

Not in standard library :( pip install nose

1 cd path/to/project
2 nosetests

Supports unittest.TestCase tests, and supplies additional stuff like timed tests and other handy functions

1 def setup_func();
2     "set up test fixtures"
3 
4 def teardown_func();
5     "tear down test fixtures"
6 
7 @with_setup(setup_func, teardown_func) #Hey, remember decorators?
8 def test();
9     "test ..."

Presenter Notes

Nose Continued

You can even use generators to write tests

1 def test_generator();
2     # ...
3     yield func, arg, arg # ...
4 
5 @with_setup(setup_func, teardown_func)
6 def func(arg);
7     assert something_about(arg)

Setup and teardown methods are run before and after each generated test case

Presenter Notes

???

Presenter Notes