for
and while
loops in Python.for
loops.list
, dictionary
, or set
using comprehension.try
/except
statement.DRY principle
to write modular code.This topic material is based on the Python Programming for Data Science book and other sources. Adapted for our purposes in the course.
for
Loops¶For loops allow us to execute code a specific number of times.
for n in [2, 7, -1, 5]:
print(f"The number is {n} and its square is {n**2}")
print("I'm outside the loop!")
The main points to notice:
for
begins the loop. Colon :
ends the first line of the loop.n
has taken all the values in the listlist
, tuple
, range
, set
, string
.word = "Python"
for letter in word:
print("Gimme a " + letter + "!")
print(f"What's that spell?!! {word}!")
A very common pattern is to use for
with the range()
. range()
gives you a sequence of integers up to some value (non-inclusive of the end-value) and is typically used for looping.
range(10)
list(range(10))
for i in range(10):
print(i)
We can also specify a start value and a skip-by value with range
:
for i in range(1, 101, 10):
print(i)
We can write a loop inside another loop to iterate over multiple dimensions of data:
for x in [1, 2, 3]:
for y in ["a", "b", "c"]:
print((x, y))
list_1 = [0, 1, 2]
list_2 = ["a", "b", "c"]
for i in range(3):
print(list_1[i], list_2[i])
There are many clever ways of doing these kinds of things in Python. When looping over objects, I tend to use zip()
and enumerate()
quite a lot in my work. zip()
returns a zip object which is an iterable of tuples.
for i in zip(list_1, list_2):
print(i)
We can even "unpack" these tuples directly in the for
loop:
for i, j in zip(list_1, list_2):
print(i, j)
enumerate()
adds a counter to an iterable which we can use within the loop.
for i in enumerate(list_2):
print(i)
for n, i in enumerate(list_2):
print(f"index {n}, value {i}")
We can loop through key-value pairs of a dictionary using .items()
. The general syntax is for key, value in dictionary.items()
.
courses = {521 : "awesome",
551 : "riveting",
511 : "naptime!"}
for course_num, description in courses.items():
print(f"DSCI {course_num}, is {description}")
We can even use enumerate()
to do more complex un-packing:
for n, (course_num, description) in enumerate(courses.items()):
print(f"Item {n}: DSCI {course_num}, is {description}")
for
loops¶Just as we can nest conditional statements, we can also nest loops.
A nested loop is a
for
orwhile
loop contained within anotherfor
orwhile
loop.
As with nested if
statements, it's very important to be careful about your indentation.
professors = ['Moreira', 'McAlister', 'Soni']
classes = ['QTM 350', 'QTM 340']
for cl in classes:
for prof in professors:
print("Is {cl} taught by {prof}?".format(cl = cl, prof = prof))
for i in range(1, 6):
for j in range(1, i + 1):
print("*", end=" ")
print(" ")
for
loops can take a very long to execute if:list
s are very long. That was a (hopefully somewhat gentle) introduction to for
loops. If you're feeling like you want more practice:
while
loops¶We can also use a while
loop to excute a block of code several times. But beware! If the conditional expression is always True
, then you've got an infintite loop!
n = 10
while n > 0:
print(n)
n -= 1
print("Blast off!")
Let's read the while
statement above as if it were in English. It means, “While n
is greater than 0, display the value of n
and then decrement n
by 1. When you get to 0, display the word Blast off!”
For some loops, it's hard to tell when, or if, they will stop! Take a look at the Collatz conjecture. The conjecture states that no matter what positive integer n
we start with, the sequence will always eventually reach 1 - we just don't know how many iterations it will take.
n = 11
while n != 1:
print(int(n))
if n % 2 == 0: # n is even
n = n / 2
else: # n is odd
n = n * 3 + 1
print(int(n))
Hence, in some cases, you may want to force a while
loop to stop based on some criteria, using the break
keyword.
n = 123
i = 0
while n != 1:
print(int(n))
if n % 2 == 0: # n is even
n = n / 2
else: # n is odd
n = n * 3 + 1
i += 1
if i == 10:
print(f"Ugh, too many iterations!")
break
The continue
keyword is similar to break
but won't stop the loop. Instead, it just restarts the loop from the top.
n = 10
while n > 0:
if n % 2 != 0: # n is odd
n = n - 1
continue
break # this line is never executed because continue restarts the loop from the top
print(n)
n = n - 1
print("Blast off!")
Comprehensions allow us to build lists/tuples/sets/dictionaries in one convenient, compact line of code. I use these quite a bit! Below is a standard for
loop you might use to iterate over an iterable and create a list:
subliminal = ['Davi', 'ingests', 'many', 'eggs', 'to', 'outrun', 'large', 'eagles', 'after', 'running', 'near', '!']
first_letters = []
for word in subliminal:
first_letters.append(word[0])
print(first_letters)
List comprehension allows us to do this in one compact line:
letters = [word[0] for word in subliminal] # list comprehension
letters
We can make things more complicated by doing multiple iteration or conditional iteration:
[(i, j) for i in range(3) for j in range(4)]
[i for i in range(11) if i % 2 == 0] # condition the iterator, select only even numbers
[-i if i % 2 else i for i in range(11)] # condition the value, -ve odd and +ve even numbers
There is also set comprehension:
words = ['hello', 'goodbye', 'the', 'antidisestablishmentarianism']
y = {word[-1] for word in words} # set comprehension
y # only has 3 elements because a set contains only unique items and there would have been two e's
Dictionary comprehension:
word_lengths = {word:len(word) for word in words} # dictionary comprehension
word_lengths
Tuple comprehension doesn't work as you might expect... We get a "generator" instead (more on that later).
y = (word[-1] for word in words) # this is NOT a tuple comprehension - more on generators later
print(y)
try
/ except
¶If something goes wrong, we don't want our code to crash - we want it to fail gracefully. In Python, this can be accomplished using try
/except
. Here is a basic example:
this_variable_does_not_exist
print("Another line") # code fails before getting to this line
try:
this_variable_does_not_exist
except:
pass # do nothing
print("You did something bad! But I won't raise an error.") # print something
print("Another line")
Python tries to execute the code in the try
block. If an error is encountered, we "catch" this in the except
block (also called try
/catch
in other languages). There are many different error types, or exceptions - we saw NameError
above.
5/0 # ZeroDivisionError
my_list = [1,2,3]
my_list[5] # IndexError
my_tuple = (1,2,3)
my_tuple[0] = 0 # TypeError
Ok, so there are apparently a bunch of different errors one could run into. With try
/except
you can also catch the exception itself:
try:
this_variable_does_not_exist
except Exception as ex:
print("You did something bad!")
print(ex)
print(type(ex))
In the above, we caught the exception and assigned it to the variable ex
so that we could print it out. This is useful because you can see what the error message would have been, without crashing your program. You can also catch specific exceptions types. This is typically the recommended way to catch errors, you want to be specific in catching your error so you know exactly where and why your code failed.
try:
this_variable_does_not_exist # name error
# (1, 2, 3)[0] = 1 # type error
# 5/0 # ZeroDivisionError
except TypeError:
print("You made a type error!")
except NameError:
print("You made a name error!")
except:
print("You made some other sort of error")
The final except
would trigger if the error is none of the above types, so this sort of has an if
/elif
/else
feel to it. There is also an optional else
and finally
keyword (which I almost never used), but you can read more about here.
try:
this_variable_does_not_exist
except:
print("The variable does not exist!")
finally:
print("I'm printing anyway!")
We can also write code that raises an exception on purpose, using raise
:
def add_one(x): # we'll get to functions in the next section
return x + 1
add_one("blah")
def add_one(x):
if not isinstance(x, float) and not isinstance(x, int):
raise TypeError(f"Sorry, x must be numeric, you entered a {type(x)}.")
return x + 1
add_one("blah")
This is useful when your function is complicated and would fail in a complicated way, with a weird error message. You can make the cause of the error much clearer to the user of the function. If you do this, you should ideally describe these exceptions in the function documentation, so a user knows what to expect if they call your function.
Finally, we can even define our own exception types. We do this by inheriting from the Exception
class - we'll explore classes and inheritance more in the next chapter!
class CustomAdditionError(Exception):
pass
def add_one(x):
if not isinstance(x, float) and not isinstance(x, int):
raise CustomAdditionError("Sorry, x must be numeric")
return x + 1
add_one("blah")
Key takeaways:
for
loop iterates through a sequence and does the same thing for each item in that sequence. while
loop runs the code code as long as some condition is met. while
loops sometimes get "stuck", if you're not careful about ensuring this condition will eventually evaluate to False
. A function is a reusable piece of code that can accept input parameters, also known as "arguments". For example, let's define a function called square
which takes one input parameter n
and returns the square n**2
:
def square(n):
n_squared = n**2
return n_squared
square(2)
square(100)
square(12345)
Functions begin with the def
keyword, then the function name, arguments in parentheses, and then a colon (:
). The code executed by the function is defined by indentation. The output or "return" value of the function is specified using the return
keyword.
When you create a variable inside a function, it is local, which means that it only exists inside the function. For example:
def cat_string(str1, str2):
string = str1 + str2
return string
cat_string('My name is ', 'Davi')
string
If a function changes the variables passed into it, then it is said to have side effects. For example:
def silly_sum(my_list):
my_list.append(0)
return sum(my_list)
l = [1, 2, 3, 4]
out = silly_sum(l)
out
The above looks like what we wanted? But wait... it changed our l
object...
l
If your function has side effects like this, you must mention it in the documentation (which we'll touch on later in this chapter).
If you do not specify a return value, the function returns None
when it terminates:
def f(x):
x + 1 # no return!
if x == 999:
return
print(f(0))
Sometimes it is convenient to have default values for some arguments in a function. Because they have default values, these arguments are optional, and are hence called "optional arguments". For example:
def repeat_string(s, n=2):
return s*n
repeat_string("mds", 2)
repeat_string("mds", 5)
repeat_string("mds") # do not specify `n`; it is optional
Ideally, the default value for optional arguments should be carefully chosen. In the function above, the idea of "repeating" something makes me think of having 2 copies, so n=2
feels like a reasonable default.
You can have any number of required arguments and any number of optional arguments. All the optional arguments must come after the required arguments. The required arguments are mapped by the order they appear. The optional arguments can be specified out of order when using the function.
def example(a, b, c="DEFAULT", d="DEFAULT"):
print(a, b, c, d)
example(1, 2, 3, 4)
Using the defaults for c
and d
:
example(1, 2)
Specifying c
and d
as keyword arguments (i.e. by name):
example(1, 2, c=3, d=4)
Specifying only one of the optional arguments, by keyword:
example(1, 2, c=3)
Specifying all the arguments as keyword arguments, even though only c
and d
are optional:
example(a=1, b=2, c=3, d=4)
Specifying c
by the fact that it comes 3rd (I do not recommend this because I find it is confusing):
example(1, 2, 3)
Specifying the optional arguments by keyword, but in the wrong order (this can also be confusing, but not so terrible - I am fine with it):
example(1, 2, d=4, c=3)
Specifying the non-optional arguments by keyword (I am fine with this):
example(a=1, b=2)
Specifying the non-optional arguments by keyword, but in the wrong order (not recommended, I find it confusing):
example(b=2, a=1)
Specifying keyword arguments before non-keyword arguments (this throws an error):
example(a=2, 1)
So far, we've assumed that we know how many arguments will be passed into a function at any given time. But this isn't always the case.
Fortunately, Python gives us two ways to handle an arbitrary number of arguments:
*args
: allows a function
to receive an arbitrary number of (positional) arguments, which can be "unpacked" as needed. The function treats them as a tuple
. **kwargs
: allows a function
to receive a dictionary
of (keyword) arguments, which can be "unpacked" as needed. *args
in practice¶The *args
syntax allows you to input an arbitrary number of arguments into a function.
def my_function(*fruits):
print("The last fruit is " + fruits[-1] + ".")
my_function("strawberry")
my_function("strawberry", "apple")
**kwargs
in practice¶The *kwargs
is similar to *args
, but allows for an arbitrary number of keyword arguments.
dict
by the function.def my_function(**fruits):
print(fruits)
### Keyword and value are automatically placed into dictionary
my_function(name = "apple", amount = 5)
### The specific keyword can be altered as needed
my_function(name = "banana", cost = 10)
In general, **kwargs
is useful when you want flexibility.
For example, suppose you have a website, in which people can (optionally) fill out the following information:
Name
. Email
. Phone number
.Location
.But because not everyone fills out every field, the function you use to store this information needs to be flexible about how many arguments it receives.m
def store_user(**info):
## For now, this is just a placeholder to demonstrate
for item in info.items():
print(item)
store_user(Name = "Davi", Location = "Atlanta")
In many programming languages, functions can only return one object. That is technically true in Python too, but there is a "workaround", which is to return a tuple.
def sum_and_product(x, y):
return (x + y, x * y)
sum_and_product(5, 6)
The parentheses can be omitted (and often are), and a tuple
is implicitly returned as defined by the use of the comma:
def sum_and_product(x, y):
return x + y, x * y
sum_and_product(5, 6)
It is common to immediately unpack a returned tuple into separate variables, so it really feels like the function is returning multiple values:
s, p = sum_and_product(5, 6)
s
p
As an aside, it is conventional in Python to use _
for values you don't want:
s, _ = sum_and_product(5, 6)
s
_
You can also call/define functions that accept an arbitrary number of positional or keyword arguments using *args
and **kwargs
.
def add(*args):
print(args)
return sum(args)
add(1, 2, 3, 4, 5, 6)
def add(**kwargs):
print(kwargs)
return sum(kwargs.values())
add(a=3, b=4, c=5)
In Python, functions are actually a data type:
def do_nothing(x):
return x
type(do_nothing)
print(do_nothing)
This means you can pass functions as arguments into other functions.
def square(y):
return y**2
def evaluate_function_on_x_plus_1(fun, x):
return fun(x+1)
evaluate_function_on_x_plus_1(square, 5)
So what happened above?
fun(x+1)
becomes square(5+1)
square(6)
becomes 36
A namespace is the "space" where a given set of variable names have been declared.
Python has several types of namespaces:
So far, we've mostly been working with variables defined in the global namespace.
## define global variable
my_var = 2
## reference global variable
print(my_var)
If you declare a variable within a function definition, that variable does not persist outside the scope of that function.
In the function below, we declare a new variable called answer
, which is eventually return
ed.
def exponentiate(num, exp):
### "answer" is a new variable
answer = num ** exp
return answer
### This will throw an error
print(answer)
If you've defined a variable in the global namespace, you can reference it inside a function.
## define global variable
my_var = 2
## define function
def add_two(x):
## references my_var
return x + my_var
add_two(2)
whos
¶Remember that you can check which variables are defined using whos
.
whos
test_var = 2
def test_func(x):
test_var = x ** 2
return test_var
new_var = test_func(5)
There are two ways to define functions in Python. The way we've beenusing up until now:
def add_one(x):
return x+1
add_one(7.2)
Or by using the lambda
keyword:
add_one = lambda x: x+1
type(add_one)
add_one(7.2)
The two approaches above are identical. The one with lambda
is called an anonymous function. Anonymous functions can only take up one line of code, so they aren't appropriate in most cases, but can be useful for smaller things.
evaluate_function_on_x_plus_1(lambda x: x ** 2, 5)
Above:
lambda x: x**2
evaluates to a value of type function
(otice that this function is never given a name - hence "anonymous functions").5
are passed into evaluate_function_on_x_plus_1
5+1
, and we get 36
.What we've learned so far:
return
some output. if
statements, for
loops, etc.).DRY stands for Don't Repeat Yourself. See the relevant Wikipedia article for more about this principle.
As an example, consider the task of turning each element of a list into a palindrome.
names = ["gabriel", "davi", "juliana"]
name = "davi"
name[::-1] # creates a slice that starts at the end and moves backwards, syntax is [begin:end:step]
names_backwards = list()
names_backwards.append(names[0] + names[0][::-1])
names_backwards.append(names[1] + names[1][::-1])
names_backwards.append(names[2] + names[2][::-1])
names_backwards
The code above is gross, terrible, yucky code for several reasons:
names
;Let's try this a different way:
names_backwards = list()
for name in names:
names_backwards.append(name + name[::-1])
names_backwards
The above is slightly better and we have solved problems (1) and (3). But let's create a function to make our life easier:
def make_palindromes(names):
names_backwards = list()
for name in names:
names_backwards.append(name + name[::-1])
return names_backwards
make_palindromes(names)
Okay, this is even better. We have now also solved problem (2), because you can call the function with any list, not just names
. For example, what if we had multiple lists:
names1 = ["juliana", "davi", "gabriel"]
names2 = ["apple", "orange", "banana"]
make_palindromes(names1)
make_palindromes(names2)
How far you go and how you choose to apply the DRY principle is up to you and the programming context. These decisions are often ambiguous. Should make_palindromes()
be a function if I'm only ever doing it once? Twice? Should the loop be inside the function, or outside? Should there be TWO functions, one that loops over the other?
In my personal opinion, make_palindromes()
does a bit too much to be understandable. I prefer this:
def make_palindrome(name):
return name + name[::-1]
make_palindrome("milad")
From here, if we want to "apply make_palindrome
to every element of a list" we could use list comprehension:
[make_palindrome(name) for name in names]
There is also the in-built map()
function which does exactly this, applies a function to every element of a sequence:
list(map(make_palindrome, names))
Recall list comprehension from earlier in the chapter:
[n for n in range(10)]
Comprehensions evaluate the entire expression at once, and then returns the full data product. Sometimes, we want to work with just one part of our data at a time, for example, when we can't fit all of our data in memory. For this, we can use generators.
(n for n in range(10))
Notice that we just created a generator object
. Generator objects are like a "recipe" for generating values. They don't actually do any computation until they are asked to. We can get values from a generator in three main ways:
next()
list()
gen = (n for n in range(10))
next(gen)
next(gen)
Once the generator is exhausted, it will no longer return values:
gen = (n for n in range(10))
for i in range(11):
print(next(gen))
We can see all the values of a generator using list()
but this defeats the purpose of using a generator in the first place:
gen = (n for n in range(10))
list(gen)
Finally, we can loop over generator objects too:
gen = (n for n in range(10))
for i in gen:
print(i)
Above, we saw how to create a generator object using comprehension syntax but with parentheses. We can also create a generator using functions and the yield
keyword (instead of the return
keyword):
def gen():
for n in range(10):
yield (n, n ** 2)
g = gen()
print(next(g))
print(next(g))
print(next(g))
Below is some real-world motivation of a case where a generator might be useful. Say we want to create a list of dictionaries containing information about houses in Canada.
import random # we'll learn about imports in a later chapter
import time
import memory_profiler
city = ['Vancouver', 'Toronto', 'Ottawa', 'Montreal']
def house_list(n):
houses = []
for i in range(n):
house = {
'id': i,
'city': random.choice(city),
'bedrooms': random.randint(1, 5),
'bathrooms': random.randint(1, 3),
'price ($1000s)': random.randint(300, 1000)
}
houses.append(house)
return houses
house_list(2)
What happens if we want to create a list of 1,000,000 houses? How much time/memory will it take?
start = time.time()
mem = memory_profiler.memory_usage()
print(f"Memory usage before: {mem[0]:.0f} mb")
people = house_list(500000)
print(f"Memory usage after: {memory_profiler.memory_usage()[0]:.0f} mb")
print(f"Time taken: {time.time() - start:.2f}s")
def house_generator(n):
for i in range(n):
house = {
'id': i,
'city': random.choice(city),
'bedrooms': random.randint(1, 5),
'bathrooms': random.randint(1, 3),
'price ($1000s)': random.randint(300, 1000)
}
yield house
start = time.time()
print(f"Memory usage before: {mem[0]:.0f} mb")
people = house_generator(500000)
print(f"Memory usage after: {memory_profiler.memory_usage()[0]:.0f} mb")
print(f"Time taken: {time.time() - start:.2f}s")
Although, if we used list()
to extract all of the genertator values, we'd lose our memory savings:
print(f"Memory usage before: {mem[0]:.0f} mb")
people = list(house_generator(500000))
print(f"Memory usage after: {memory_profiler.memory_usage()[0]:.0f} mb")
One problem we never really solved when talking about writing good functions was: "4. It is hard to understand what it does just by looking at it". This brings up the idea of function documentation, called "docstrings". The docstring goes right after the def
line and is wrapped in triple quotes """
.
def make_palindrome(string):
"""Turns the string into a palindrome by concatenating itself with a reversed version of itself."""
return string + string[::-1]
In Python we can use the help()
function to view another function's documentation. In IPython/Jupyter, we can use ?
to view the documentation string of any function in our environment.
make_palindrome?
But, even easier than that, if your cursor is in the function parentheses, you can use the shortcut shift + tab
to open the docstring at will.
# make_palindrome('uncomment this line and try pressing shift+tab here.')
General docstring convention in Python is described in PEP 257 - Docstring Conventions. There are many different docstring style conventions used in Python. The exact style you use can be important for helping you to render your documentation, or for helping your IDE parse your documentation. Common styles include:
The NumPy style:
def function_name(param1, param2, param3):
"""First line is a short description of the function.
A paragraph describing in a bit more detail what the
function does and what algorithms it uses and common
use cases.
Parameters
----------
param1 : datatype
A description of param1.
param2 : datatype
A description of param2.
param3 : datatype
A longer description because maybe this requires
more explanation and we can use several lines.
Returns
-------
datatype
A description of the output, datatypes and behaviours.
Describe special cases and anything the user needs to
know to use the function.
Examples
--------
>>> function_name(3,8,-5)
2.0
"""
def make_palindrome(string):
"""Turns the string into a palindrome by concatenating
itself with a reversed version of itself.
Parameters
----------
string : str
The string to turn into a palindrome.
Returns
-------
str
string concatenated with a reversed version of string
Examples
--------
>>> make_palindrome('davi')
'daviivad'
"""
return string + string[::-1]
make_palindrome?
When specifying function arguments, we specify the defaults for optional arguments:
# scipy style
def repeat_string(s, n=2):
"""
Repeat the string s, n times.
Parameters
----------
s : str
the string
n : int, optional
the number of times, by default = 2
Returns
-------
str
the repeated string
Examples
--------
>>> repeat_string("Blah", 3)
"BlahBlahBlah"
"""
return s * n
Type hinting is exactly what it sounds like, it hints at the data type of function arguments. You can indicate the type of an argument in a function using the syntax argument : dtype
, and the type of the return value using def func() -> dtype
. Let's see an example:
# NumPy style
def repeat_string(s: str, n: int = 2) -> str: # <---- note the type hinting here
"""
Repeat the string s, n times.
Parameters
----------
s : str
the string
n : int, optional (default = 2)
the number of times
Returns
-------
str
the repeated string
Examples
--------
>>> repeat_string("Blah", 3)
"BlahBlahBlah"
"""
return s * n
repeat_string?
Type hinting just helps your users and IDE identify dtypes and identify bugs. It's just another level of documentation. They do not force users to use that date type, for example, I can still pass an dict
to repeat_string
if I want to:
repeat_string({'key_1': 1, 'key_2': 2})
Most IDE's are clever enough to even read your type hinting and warn you if you're using a different dtype in the function.
!jupyter nbconvert _02-py-loops-functions.ipynb --to html --template classic --output 02-py-loops-functions.html